In 2018 I rented my first real server. An Ubuntu VPS, one root password, no further thought. Two weeks later fail2ban showed me the bill: 2,847 failed SSH logins in 14 days, most of them from IPs in China and Russia. Not a single one would have gotten through: the password was strong and long. The number kept me awake anyway. Passwords aren't the weakest link because they're too short; they're the weakest link because we reuse them everywhere. The same password that protected my server back then had been sitting in some forum years earlier. Today every service I run myself has a second factor. This article is about what TOTP actually does, where it hurts, and how to roll it out without locking yourself out.

TOTP stands for Time-based One-Time Password, and the core idea fits in one sentence: the server and your authenticator app share a secret, both compute the same six-digit code from the same time value, and anyone who presents the code proves they know the secret. The trick is time. A code is valid for 30 seconds, then both sides compute a new one. An intercepted code is worthless before you finish typing it in.

Under the hood: HMAC-SHA1, truncation, and the time window

The standard is called RFC 6238 and builds on HMAC, a construction that computes a check value from a key and a message. In TOTP the message is a counter derived from the clock: the Unix time in seconds, divided by 30, rounded down. Just now my counter was somewhere around 1,723,456,789; the exact number doesn't matter. What matters is that it changes every 30 seconds, and that it changes for server and app at exactly the same moment.

The secret consists of 20 random bytes, 160 bits of entropy. In practice you see it as a 32-character Base32 string, something like JBSWY3DPEHPK3PXP. The app stores it, the server stores it, and neither ever sends it over the wire again. I generate secrets with openssl rand -base64 20; if you'd rather click than type, the bitcalc Password Generator does the same job, 32 characters from the Base32 alphabet are exactly a 160-bit secret. When a secret ends up in a file, I compute its SHA-256 with the bitcalc Hash Generator and store the hash separately. That way I notice any tampering before the file goes anywhere.

HMAC-SHA1 turns secret and counter into a 20-byte hash. SHA1 sounds like 1995 and is still the right tool here: the hash is never transmitted in full, and the famous weaknesses of SHA1 are about collisions, not about the kind of predictability that would help an attacker. Six digits come out of that hash at the end. The algorithm reads the last four bits and uses them as an offset: a number between 0 and 19 that picks the position where four bytes get extracted. Those four bytes form a 31-bit number, and the code is the remainder after dividing by 10^6. Six digits aren't arbitrary, they're the most convenient slice of a million possible codes.

That's where the security lives. An attacker who intercepted a code has 30 seconds to use it. An attacker who guesses gets a few tries per window before the service locks. At three tries per window, covering all one million combinations takes about 116 days. The 30-second window is the compromise between those two worlds: short enough to make codes useless, long enough to type them in calmly.

2FA for SSH: pam_oath and the alternatives

The classic approach for SSH is pam_oath, a PAM module that puts TOTP straight into the login flow. Setup takes ten minutes and looks like this:

# 1. Install the module
apt install libpam-oath

# 2. Generate a secret (hex!) and register it
openssl rand -hex 20
echo "HOTP/T30 admin 5c1f2a9b8e7d6c4f3a2b1e0d9c8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f" \
     >> /etc/security/users.oath

# 3. Enable PAM, file: /etc/pam.d/sshd
auth required pam_oath.so usersfile=/etc/security/users.oath window=30

# 4. Require both factors, file: /etc/ssh/sshd_config
AuthenticationMethods publickey,keyboard-interactive

Two details decide whether this works in the long run. First, the syntax: publickey,keyboard-interactive with a comma means either factor is enough. If you want both, separate the methods with a space, then every session needs the SSH key and a TOTP code. I run the comma version so I can rescue myself with the key alone when the OATH service misbehaves. Second, the window parameter: it accepts codes from the previous and the next window, so small clock offsets and hasty typing don't cause failures. Two steps of tolerance are plenty, more just widens the door for stolen codes.

The alternatives to pam_oath come in two flavors: YubiKeys via pam_u2f, and OpenSSH's ed25519-SK. Both even make the login phishing-resistant, because the key includes the hostname in the signature. A YubiKey is for people who won't type a code. My mother will never manage an authenticator string, but she will plug a stick into a USB port. My Pi and my VPS still run pam_oath, because it needs no hardware and takes five minutes. One thing people forget: a second factor protects the login, not the session. If someone steals your SSH key, they don't need a TOTP code. Keys belong on hardware, or at least password-protected inside the agent.

Web apps and mail: TOTP where it belongs

For web apps, TOTP is the default because it works the same everywhere. Nextcloud, Grafana, Proxmox, Cockpit, all of them support it, most out of the box. If you run many services, put a central login like Authelia or authentik in front and set up 2FA exactly once. I run Authelia in front of Grafana, Portainer, and two client dashboards. One endpoint is deliberately excluded: Prometheus. It gets polled by machines, and a six-digit code that changes every 30 seconds would only break what used to work. Machine access doesn't belong in TOTP, it belongs in API keys with scoped permissions.

Mail is the trickiest case, because SMTP has no concept of TOTP. Your mail server has to authenticate to foreign servers with a username and password, otherwise everything lands in spam. What you can secure is access to the mailbox: webmail with TOTP, IMAP with app passwords instead of the plaintext password, and a master password that lives only on the server. My mail client uses an app password I generated once and buried in the password manager. Nobody knows the actual mailbox password anymore, it only exists inside the webmailer's TOTP setup. Even a stolen laptop can't get straight to the mail.

Backup codes matter more than the app on your phone

Here's the story that explains this section. In spring 2024 I moved apartments. New phone, new SIM, the usual. That evening I wanted to log into my Nextcloud admin, and the login demanded a TOTP code. My old phone with the authenticator was in a moving box I couldn't find that evening, and the new one wasn't set up. Without backup codes I would have been locked out of my own cloud, and the only way back would have been an admin reset that kills foreign sessions and loses logs. What saved me was a sheet of ten codes I had printed two years earlier and filed away. Ten minutes later I was in, and I had printed fresh codes.

That's why my order is: backup codes first, app second. The app is replaceable, a phone breaks, gets stolen, or ends up in a moving box. The codes are the one artifact that carries you through exactly that situation. Ten codes per service is the standard and it's enough, you burn maybe one per year. Each code works exactly once, then you cross it off.

Where do they go? Two places. A sheet in an envelope in the filing cabinet or safe, and a copy in your password manager. What I never do: store the codes in the cloud of the very service they protect. A break-in into your Nextcloud account must not deliver the codes for that same Nextcloud. If you keep the list digitally, hash it first with the bitcalc Hash Generator and store the hash separately, then you'll notice any tampering.

And the last rule: replace used codes promptly. I print a fresh list once ten codes are gone. The day you use your last code is guaranteed to be the day you lose your phone.

Storing backup codes safely: Ten codes per service, in two places: a printed sheet in the filing cabinet or safe, one copy in the password manager. Never in the cloud of the service the codes protect. Cross off used codes and replace them before the list runs dry.

The time problem: drift, wrong clocks, and the server without NTP

TOTP stands and falls with the clock. Server and app compute the same code from the same second value, and if one side is five minutes off, ten windows have passed and not a single code matches. The symptom is always the same: the code gets rejected even though you just typed it from the app.

Most authenticator apps tolerate one window forward and one back, some two. Beyond that, only the clock helps. On servers the answer is NTP, concretely systemd-timesyncd or chrony. I once had a KVM server whose BIOS clock ran two minutes late after every reboot because the RTC battery was dead. For two months I believed my TOTP setup was broken, until I ran hwclock -s and the next code worked immediately. The real mistake: the server never fetched network time after booting. Since then I check timedatectl after every reboot and want to see System clock synchronized: yes.

On the client side the clock is rarely the problem, but two cases exist: dual-boot machines where Windows sets the hardware clock to local time while Linux uses UTC, and containers with a frozen clock. Docker containers inherit the host clock, so fixing the host is enough. And when the clock is right, the code is right, and it still fails, it's almost always the secret: a Base32 string mistyped by one character. Check it, re-import it, done.

Rolling it out in five steps

This is how I migrated the last service without leaving anyone standing outside:

  1. Check the clock: timedatectl shows System clock synchronized: yes, otherwise set up chrony or timesyncd.
  2. Generate the secret: openssl rand -base64 20, double-check the Base32 string with the Password Generator if you have to type it.
  3. Set up the second factor: app or hardware key, scan the QR code, test one code.
  4. Print and store backup codes: ten codes, two places, see the box above.
  5. Enforce the second factor: only after a few days of trial operation, so nobody locks themselves out.

The order is the point. Start with step 5 and you'll lock yourself out, guaranteed. Skip step 4 and the first dead phone cuts off your access. Skip step 1 and you'll spend two months chasing a time problem that never was one. I know all of this from personal experience.

Bottom line

TOTP is the best ratio of security to effort I know for self-hosted services. No certificate sprawl, no second server, no recurring cost. One secret, one clock, six digits. The math is simple enough to understand and robust enough to have worked unchanged since 2011. RFC 6238 is one of the few standards I can explain from memory without looking it up.

What I want you to take away: start small. One service, one secret, ten backup codes. SSH with pam_oath, then Nextcloud, then the rest. The 116 days of brute-force resistance are a nice feeling, but the real win shows up in the fail2ban log: no hits at all anymore, because a password alone no longer gets anyone in. And if you ever stand in front of a login while the app sits on your old phone in a moving box, those ten printed codes will be your best friend.