Debian 13.5: FIDO2 SSH Authentication

Extend the hardware token from local PAM login to SSH authentication. Resident sk- keys, credential TTL, ControlMaster, and real-world token removal behavior — on a single VM over loopback.

Revision: v1.0. Prerequisite: the FIDO2 lab — token PIN must already be set.

Debian 13.5 / trixie OpenSSH 10.3p1 FIDO2 resident sk- keys Loopback server + client
Contents
  1. Background and philosophy
  2. Lab topology: loopback
  3. Prerequisites and safety
  4. Generate a resident FIDO2 SSH key
  5. Non-resident key comparison
  6. Copy the public key to the server
  7. ssh-agent, credential TTL, and ControlMaster
  8. Test the connection
  9. Scenario lab: verify-required, TTL, and touch-required
  10. Failure tests and token removal behavior
  11. Server-side hardening
  12. Backup token enrollment
  13. Resident key portability
  14. Git hosting services — context note
  15. Connection to the blue/yellow model
  16. Rollback
  17. Troubleshooting
  18. References

1. Background and philosophy

This lab extends the FIDO2 hardware token from local PAM login — covered in the FIDO2 lab — to SSH authentication. The same physical token, the same PIN, the same touch gesture. A different enrollment path and a different trust domain.

How FIDO2 SSH differs from what you already built

PropertyPAM login (Module 1)FIDO2 SSH (this lab)
Enrollment toolpamu2fcfgssh-keygen
Credential stored in/etc/security/u2f_keysauthorized_keys on server
Authentication pathPAM stack → pam_u2f.soSSH key exchange → libfido2
Private key locationToken secure elementToken secure element
PIN enforcementpinverification=1 in PAM-O verify-required at keygen + authorized_keys
RollbackRemove PAM include, restore backup filesRemove public key from authorized_keys

What the network adds — and what it introduces

This project teaches isolated-system security as a foundation, not a limitation. A system secured without assuming a network connection is often more defensible and lower-maintenance than one that depends on the network for its security properties. SSH is covered because it is a natural and important extension of hardware-backed authentication into networked access — not as a replacement for thinking carefully about what the network introduces.

The network adds reach: one token can authenticate to many hosts. It introduces attack surface: credential forwarding risks, lateral movement, and the possibility that an established session outlives the intended access window. Understanding both sides is the goal of this lab.

Auth methodPrivate key locationCopied without token?Requires PIN?Requires touch?
Password SSHServer (hashed)N/AYes (the password)No
File-based key SSH~/.ssh/id_ed25519 on diskYes — triviallyOptional (passphrase)No
FIDO2 SSH (this lab)Token secure elementNo — physically impossibleYes (verify-required)Yes

2. Lab topology: loopback as server and client

This lab uses a single VM as both the SSH server and the SSH client, connecting over loopback (ssh localhost). This removes the second-machine setup burden without obscuring the conceptual model.

The security property being demonstrated does not require a real network: the token, the key material, and the authentication flow are identical whether the server is on loopback or across the internet. If the lab works on loopback, it works on a real network.

Role labeling

Throughout this lab, every command block is labeled with the role it represents — even though both roles are on the same machine.

Server-side — actions the server administrator takes: editing /etc/ssh/sshd_config, managing service state. Requires sudo. Client-side — actions the end user takes: key generation, agent management, making connections.

Enable openssh-server

Server-side — sudo
sudo apt install openssh-server
sudo systemctl enable --now ssh
sudo systemctl status ssh

Confirm password login works before touching FIDO2

Verify that basic SSH works over loopback with a password before any FIDO2 configuration. This confirms the server is running and is your fallback throughout the lab.

Client-side
ssh localhost

You should be prompted for your Unix password and land in a shell. Type exit to close the test session. Keep this terminal open — it is your breakglass connection throughout the lab.

Do not close this terminal until the full lab is complete. It is your recovery path. If FIDO2 configuration goes wrong, this password-authenticated session lets you fix it without locking yourself out.

3. Prerequisites and safety

Confirm OpenSSH version

Client-side
ssh -V

Expected output on the lab VM:

OpenSSH_10.3p1 Debian-1, OpenSSL 3.6.2 7 Apr 2026

sk- key type support requires OpenSSH 8.2 or later. Version 10.3p1 is confirmed on Debian 13.5 trixie.

Confirm token is visible

Client-side
fido2-token -L

Expected: a line such as /dev/hidraw1: KEY-ID FIDO+U2F. If nothing is returned, confirm the token is inserted and USB passthrough is configured in VirtualBox. hidraw1 is illustrative — the actual number depends on other HID devices already present, so always use the path this command reports rather than assuming hidraw1.

Packages

Server-side — sudo
sudo apt update
sudo apt install openssh-client openssh-server libfido2-1

libfido2-1 is typically already installed if you completed the PAM lab. openssh-client and openssh-server are usually present on the lab VM.

Replace these placeholders before running commands

PlaceholderMeaning
FIDO_USERThe local user account that will use FIDO2 SSH.
Keep a password-authenticated SSH session open to localhost throughout this lab. All PAM and sshd configuration changes carry the risk of locking you out of SSH. The open session is your recovery path — do not close it until every test in section 10 passes.

4. Generate a resident FIDO2 SSH key

Generate the key

Insert the FIDO2 token. Run the following as FIDO_USER:

Client-side
ssh-keygen -t ed25519-sk -O resident -O verify-required -C "fido2-ssh-lab"

What each flag means:

FlagEffect
-t ed25519-skEd25519 curve with security key. Preferred over ecdsa-sk on current OpenSSH. Use ecdsa-sk only if connecting to a server running OpenSSH older than 8.2.
-O residentStore the credential as a discoverable (resident) credential on the token itself. The token holds the full credential — no private key material is written to disk.
-O verify-requiredBake PIN (user verification) into the credential at enrollment. The token will require the PIN on every signing operation regardless of what the client or server requests.
-C "fido2-ssh-lab"Comment. Appears in authorized_keys and ssh-add -l output. Use something meaningful — token model, host, or purpose.

During generation you will be prompted for the FIDO2 PIN, then asked to touch the token when it flashes. Accept the default file location (~/.ssh/id_ed25519_sk) and leave the passphrase blank when prompted — the token PIN is the protection.

Output files

Client-side
ls -l ~/.ssh/id_ed25519_sk ~/.ssh/id_ed25519_sk.pub
cat ~/.ssh/id_ed25519_sk.pub

The public key line will look like:

sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab

The stub file — what it is and is not

The file id_ed25519_sk is called a stub file. Inspect it:

Client-side
file ~/.ssh/id_ed25519_sk
head -1 ~/.ssh/id_ed25519_sk
/home/FIDO_USER/.ssh/id_ed25519_sk: OpenSSH private key
-----BEGIN OPENSSH PRIVATE KEY-----

Despite the header and the filename, this is not an exportable private key. The private key material is stored in the token's secure element and never leaves it. What the stub file contains is a credential handle — an identifier that tells the SSH agent which resident credential on the token to use when signing. The stub file alone cannot authenticate anything: it requires the physical token to complete any signing operation.

You can verify this: copy the stub file to a different machine, or try to use it with a different FIDO2 token of the same model. Authentication will fail in both cases. The credential is bound to the specific token that enrolled it.

Keep vs. delete — a usability argument

The stub file's only practical value is convenience: when it is present in ~/.ssh/ with a standard name, the SSH client discovers and offers it automatically — exactly like a traditional key file. Without it, you must run ssh-add -K explicitly at the start of each session to load the credential from the token into the agent.

That extra step is small, but it matters. Security controls that create friction get worked around. A user who finds the manual load step annoying enough will start looking for ways to skip the token entirely — that is the real risk, not the stub file existing on disk. The stub file alone is useless without the physical token: it contains a credential handle, not key material. An attacker who reads your home directory cannot authenticate with it.

The security difference between keeping and deleting the stub is therefore minimal. The usability difference is real. Keeping the stub is a reasonable choice on a personal, single-user machine. Deleting it is appropriate on a shared or sensitive machine where no handle should exist on disk, and the explicit ssh-add -K ceremony is the deliberate policy.

If you delete the stub and want to reduce the friction of the manual load step, add an alias or a small function to your shell profile:

Client-side — add to ~/.bashrc or ~/.zshrc
# Load FIDO2 SSH credential from token into agent (with 2-hour TTL)
alias fido-load='ssh-add -K -t 2h'

Then source ~/.bashrc (or open a new terminal) and run fido-load once at the start of a session. One short command, PIN + touch, and the credential is loaded for two hours — functionally identical to the automatic stub discovery, without anything on disk.

The credential on the token is unaffected by either choice. Deleting the stub does not delete the credential. Recreating the stub later is always possible: ssh-keygen -K downloads the resident credential handle from the token and writes a new stub to the current directory.

Exercise: delete the stub and verify auth still works

Remove the key from the agent first, then delete the stub:

Client-side
ssh-add -d ~/.ssh/id_ed25519_sk
rm ~/.ssh/id_ed25519_sk
ls ~/.ssh/

Only id_ed25519_sk.pub should remain. Now reload the credential directly from the token:

Client-side
ssh-add -K
ssh-add -l

The key should appear in the agent output. Authentication will work in section 8. Decide whether to keep the stub absent (and use the alias above) or recreate it for automatic discovery — document your reasoning.

5. Non-resident key comparison

Generate a second key without -O resident to see the difference:

Client-side
ssh-keygen -t ed25519-sk -O verify-required -C "fido2-ssh-lab-nonresident" -f ~/.ssh/id_ed25519_sk_nr

Inspect both stub files:

Client-side
ls -l ~/.ssh/id_ed25519_sk* 2>/dev/null
wc -c ~/.ssh/id_ed25519_sk_nr ~/.ssh/id_ed25519_sk.pub
PropertyResident (-O resident)Non-resident (default)
Credential stored onToken only (discoverable)Token + stub file required
PortabilityToken alone; ssh-keygen -K re-derives stub anywhereMust carry stub file to each client machine
If stub is lostRe-derive from token with ssh-keygen -KCredential is inaccessible — cannot recover the handle
Token storage slots usedOne slot per credential (tokens have limited capacity, typically 25)No on-token storage slot used
When to preferCarry one token, use on multiple machinesFixed single client; low token slot availability; file on encrypted volume
This lab uses resident keys as the primary path — they correctly represent "credential lives in hardware." Clean up the non-resident key when finished with this comparison:
Client-side
rm -f ~/.ssh/id_ed25519_sk_nr ~/.ssh/id_ed25519_sk_nr.pub

6. Copy the public key to the server

Append to authorized_keys

On loopback, the client and server are the same machine and the same user. Use ssh-copy-id:

Client-side
ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub localhost

You will be prompted for your Unix password (the existing SSH session is password-authenticated). This appends the public key to ~/.ssh/authorized_keys.

Verify:

Client-side
cat ~/.ssh/authorized_keys

The line beginning with sk-ssh-ed25519@openssh.com is your FIDO2 public key.

Add server-side verify-required enforcement

The -O verify-required flag at keygen time bakes PIN enforcement into the credential itself — the token requires the PIN regardless of server policy. The server-side verify-required option in authorized_keys is an additional, independent layer: the server will reject a signature that was produced without user verification, even if the token did not require it.

Use both. Edit authorized_keys and prepend verify-required to the sk- key line:

Server-side — edit as FIDO_USER
nano ~/.ssh/authorized_keys

Change the sk- line from:

sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab

To:

verify-required sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab
Do not modify any other lines in authorized_keys. If your password-based SSH public key is also in that file, leave it untouched — it is your current session's key.

Set correct permissions

Client-side
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

7. ssh-agent, credential TTL, and ControlMaster

Confirm the agent is running

Client-side
echo $SSH_AUTH_SOCK

Expected: a path like /run/user/1000/keyring/ssh or /tmp/ssh-XXXX/agent.XXXX. If the variable is empty, start an agent: eval $(ssh-agent -s).

Load the resident credential

Client-side
ssh-add -K

Enter the FIDO2 PIN when prompted, then touch the token when it flashes. The credential is now loaded into the agent.

macOS note (for students who also use a Mac): On Apple's fork of OpenSSH, ssh-add -K means "store in the macOS Keychain" — a completely unrelated operation. The FIDO2 resident key load command on macOS is different. This lab runs on Debian 13.5 where ssh-add -K correctly loads FIDO2 resident keys. Do not copy SSH commands from macOS-specific documentation without checking the flag meanings.

Confirm the key is loaded

Client-side
ssh-add -l

Expected: a line containing SK and your comment (fido2-ssh-lab).

Remove a key manually before TTL expires

Client-side
ssh-add -d ~/.ssh/id_ed25519_sk.pub

Or remove all keys: ssh-add -D. Run ssh-add -K to reload.

Credential TTL — loading with a time limit

Instead of loading the credential indefinitely, load it with a lifetime:

Client-side
ssh-add -K -t 2h

The agent will automatically drop the credential after two hours. The next SSH connection attempt after expiry will fail with Permission denied (publickey) — run ssh-add -l to confirm the agent is empty, then ssh-add -K to reload.

The TTL bounds the window during which an unattended machine with the token still inserted could be misused. Token removal closes that window immediately regardless of TTL.

What the TTL caches — and what it does not. The agent retains the credential handle, not a signed output. For sk- keys, the signing operation occurs on the token hardware at each new connection attempt. The token must be physically present. The TTL controls how long the agent keeps the handle; it does not cache signatures. This is testable: load with a TTL, remove the token, attempt a connection — it fails. Reinsert the token within the TTL window — it succeeds with touch only (no PIN re-entry, because the handle is still valid).

ControlMaster — one auth ceremony, many connections

A complementary pattern to the TTL is ControlMaster: a persistent master SSH connection that multiplexes subsequent connections without re-invoking the agent. One PIN + touch authentication covers many git push, scp, and ssh commands in sequence.

Add to ~/.ssh/config (create the file if it does not exist):

Client-side
nano ~/.ssh/config
Host localhost 127.0.0.1
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 2h

With this in place, the first ssh localhost authenticates with PIN + touch and establishes the master. Subsequent ssh localhost commands in the same or other terminals reuse the master channel silently — no agent, no token, no prompt. After two hours of inactivity the master closes and the next connection starts a new ceremony.

In practice, scope ControlMaster to specific hosts you connect to frequently — not Host * — to avoid unintentional multiplexing to sensitive targets.

8. Test the connection

With the credential loaded in the agent and the public key in authorized_keys:

Client-side
ssh-add -K
ssh localhost

Expected behavior:

  1. The agent presents the sk- key to the server.
  2. The server issues an authentication challenge.
  3. The agent sends the challenge to the token; the token prompts for PIN (because verify-required was set at enrollment).
  4. After PIN entry, the token flashes — touch it.
  5. Authentication succeeds; you receive a shell prompt.
whoami
exit

Subsequent connections within the same agent session

Client-side
ssh localhost whoami
ssh localhost date

Each connection requires a fresh token interaction (PIN + touch) because verify-required is set. This is correct behavior for Scenario A. Sections 7 and 9 cover how to calibrate this with a TTL or ControlMaster when per-connection friction is too high for your workflow.

If a connection fails here, ssh -vvv shows the client's side — but since sshd is a systemd unit, sudo journalctl -u ssh -f run alongside it shows the server's reasoning too. Section 17 covers reading both together.

9. Scenario lab: verify-required, TTL, and touch-required

You have now used verify-required with per-connection PIN + touch. This section works through three scenarios that represent different operational contexts and tradeoffs. The goal is not to identify the "most secure" option — it is to understand what each control proves, in what context, for what user, at what moment in their session.

The options

OptionPIN required?Touch required?When appropriate
NeitherNoNoNever for FIDO2 — defeats the purpose of hardware authentication
touch-required (default for sk-)NoYesWithin an active session where per-connection PIN friction is undesirable and presence confirmation alone is sufficient; not for automation — FIDO2 tokens are human credentials
verify-requiredYesYesInteractive user login; high-value or infrequent access; contexts where proving identity (not just presence) matters at each connection

Scenario A — High-value or infrequent access

Context: You are connecting to a production server, a privileged account, or any target where each connection is a meaningful event. Full PIN + touch on every connection is appropriate. The friction cost is accepted because each connection carries real risk.

This is what you have already configured and tested. No changes needed. PIN + touch on every ssh invocation.

Scenario B — Active working session with calibrated TTL

Context: The same developer, same token, connecting to a development host repeatedly during an afternoon of work. Full PIN + touch once to load the agent; subsequent connections within a two-hour window are transparent.

Client-side
ssh-add -D
ssh-add -K -t 2h
ssh-add -l

The -t 2h flag loads the credential with a two-hour lifetime. Now connect several times:

Client-side
ssh localhost date
ssh localhost whoami
ssh localhost uptime

First connection: PIN + touch. Subsequent connections within the TTL: touch only (the PIN was proven at load time; the TTL certifies the handle has not been refreshed since then, bounding the window).

This is stronger than a certificate file on disk: re-loading after TTL expiry requires the physical token and PIN — a file on disk does not decay and does not require re-proof of presence.

Scenario C — Calibration exercise

What threat does the TTL actually address? Think through each before reading the answer:

  1. You walk away from your desk and leave the token plugged in. A colleague sits down.
  2. You leave a session open overnight.
  3. You set the TTL to zero (disable caching). What changes?
  4. You set the TTL to 8 hours. What changes?

Answers:

  • 1: A short TTL (e.g., 30 minutes) limits how long the plugged-in token can be used to make new connections without you. Token removal closes this window immediately.
  • 2: An overnight TTL means the credential is gone by morning. No action from you required. The session itself (if you left one open) is unaffected — see section 10.
  • 3: Every connection requires ssh-add -K: PIN + touch, full ceremony. Maximum assurance, maximum friction. On a workflow with dozens of git push operations per hour this creates enough friction to drive workarounds.
  • 4: Matches a typical working day. At the end of the day the credential expires on its own. You need to make a conscious decision about the right value for your actual session length and risk.

Student deliverable: Write one paragraph justifying the TTL value you would choose for your own working context. What threat does it address? What friction does it accept? What does it not address?

Note on automation and service accounts

FIDO2 hardware tokens are human authentication devices. An automated service account that needs SSH access should use a machine identity: an SSH certificate issued by a CA, a short-lived credential from a secrets manager, or a traditional key pair with tightly scoped authorization. Putting a hardware token in an automation pipeline and then disabling the touch requirement to make it work is a file-based key with extra steps — and a device that will eventually be left plugged into a server rack. The right credential for a machine is a machine credential.

10. Failure tests and token removal behavior

Work through each test in order. These are the behaviors students who skip this section will encounter unexpectedly in production.

Standard failure tests

Client-side — load credential first
ssh-add -K

Wrong PIN: Enter an incorrect PIN when prompted.

ssh localhost

Expected: sign_and_send_pubkey: signing failed for ED25519-SK or FIDO_ERR_PIN_INVALID. No password fallback offered. The server presents Permission denied (publickey).

No touch: Enter the correct PIN but do not touch the token when it flashes.

ssh localhost

Expected: the token times out waiting for touch; authentication fails. No fallback.

Password attempt after FIDO enforcement (section 11 required): After configuring the Match User block in section 11, remove the sk- key from the agent and try with password only.

ssh-add -D
ssh localhost

Expected: Permission denied (publickey). Password auth is disabled for this user.

Token removal — new connections vs. existing sessions

This is the most important behavioral distinction between the PAM lab and SSH. Students completing the PAM-FIDO lab will assume that removing the token locks the session. It does not.

Open two terminals before beginning this exercise.

Test 1 — Token removed mid-session:

  1. Terminal A: ssh localhost — authenticate with PIN + touch, stay in the shell.
  2. Remove the FIDO2 token from the USB port.
  3. Terminal A: run commands in the open session. ls, date, whoami. The session continues without interruption.
  4. Terminal B: ssh localhost — authentication fails. The agent has the handle but the signing requires the token hardware, which is absent.

An established SSH session runs on negotiated symmetric session keys. Once the handshake is complete, the agent and the token are not involved in maintaining the channel. The token passively closes the door to new connections — it does not evict sessions already inside.

There is no built-in SSH equivalent of the udev greeter-switch. In the PAM lab, token removal triggered a udev rule that actively called dm-tool switch-to-greeter within seconds. SSH has no such mechanism. If active session eviction on token removal is required by your threat model, that is a separate, deliberate mechanism at the host session layer — not something SSH provides for free. The right question before building it: is this an open-session problem or a new-connection problem?

Test 2 — TTL expiry during a file transfer:

This demonstrates that established transfers survive TTL expiry — and reveals the silent-failure behavior that trips up experienced engineers in production.

  1. Create a test file: dd if=/dev/urandom bs=1M count=200 of=/tmp/testfile.bin
  2. Load the credential with a 30-second TTL: ssh-add -K -t 30
  3. Terminal A: start the transfer: scp /tmp/testfile.bin localhost:/tmp/testfile-copy.bin
  4. Terminal B: watch the agent: watch -n1 ssh-add -l — observe the key disappear when the TTL expires.
  5. Terminal A: observe that the transfer continues and completes normally after the TTL expires.
  6. After the transfer completes, attempt a new connection: ssh localhost date

Expected: Permission denied (publickey). The error message looks identical to a misconfigured authorized_keys or wrong username. Nothing says "your credential expired."

Correct diagnostic sequence:

Client-side
ssh-add -l

Output: The agent has no identities. — the credential expired. Recovery:

Client-side
ssh-add -K

PIN + touch, full ceremony, credential reloaded.

Students who have not seen this failure will debug the wrong thing. The first instinct is to check authorized_keys, sshd configuration, or file permissions. The correct first diagnostic is always ssh-add -l to confirm the agent state before investigating anything else.

Test 3 — Token removed with TTL still active:

  1. Load with TTL: ssh-add -K -t 2h — PIN + touch.
  2. Remove the token.
  3. Attempt a connection: ssh localhost — fails. The agent has the handle but signing requires the token hardware.
  4. Reinsert the token.
  5. Attempt a connection: ssh localhost — succeeds with touch only. No PIN re-entry because the handle is still valid within the TTL window.

This confirms: the TTL retains the credential handle, not a cached signature. Each new connection still involves the token hardware. The TTL only controls how long you avoid re-running ssh-add -K.

Test 4 — ControlMaster resilience (if configured in section 7):

  1. Load with a short TTL: ssh-add -K -t 30
  2. Connect via ControlMaster: ssh localhost — PIN + touch, master established.
  3. Wait 35 seconds for TTL to expire. Confirm with ssh-add -l.
  4. In the same terminal: ssh localhost date — succeeds without PIN or touch. The ControlMaster channel is still alive and handles the new session.
  5. In a new terminal, bypass ControlMaster: ssh -o ControlMaster=no localhost date — fails. The agent is empty and a new master cannot be established.

Channel-level persistence (ControlMaster) and agent-level persistence (TTL) are independent mechanisms. Understanding which is active in a given situation determines the correct diagnostic when auth fails.

11. Server-side hardening: disable password auth for FIDO user

After verifying that FIDO2 SSH works reliably, disable password-based SSH for FIDO_USER specifically. Use a Match User block to scope the change — the breakglass account retains password access.

Do not proceed until sections 8, 9, and 10 all pass cleanly. Keep a working password-authenticated terminal session open throughout this section.
Server-side — sudo
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo editor /etc/ssh/sshd_config

Append to the end of the file:

sshd_config addition
Match User FIDO_USER
    PasswordAuthentication no

Reload sshd:

Server-side — sudo
sudo sshd -t
sudo systemctl reload ssh

sshd -t performs a configuration syntax check before reloading. If it reports errors, correct them before reloading.

Test

Client-side
ssh-add -D
ssh localhost

Expected: Permission denied (publickey). Password auth is no longer offered for FIDO_USER. Now reload the credential and confirm FIDO2 still works:

Client-side
ssh-add -K
ssh localhost whoami

Expected: PIN + touch, then your username.

Confirm the breakglass user (any user not matched by the Match User block) can still authenticate with a password from a separate terminal.

12. Backup token enrollment

Enroll a second token so that losing the primary does not lock you out of SSH.

Insert the backup token. Generate a second sk- resident key:

Client-side
ssh-keygen -t ed25519-sk -O resident -O verify-required -C "fido2-ssh-lab-backup" -f /tmp/backup_key

Append the backup public key to authorized_keys:

Server-side — edit as FIDO_USER
cat /tmp/backup_key.pub >> ~/.ssh/authorized_keys

Add verify-required to the new line as you did in section 6. Then clean up the temporary file:

Client-side
rm /tmp/backup_key /tmp/backup_key.pub

Test both tokens independently. Swap tokens and confirm each authenticates:

Client-side
ssh-add -D
ssh-add -K
ssh localhost whoami

Repeat with the other token. Both should succeed independently.

The backup token's credential is independent. It has its own PIN (set the same or different — your choice). Losing one token does not affect the other's authentication.

13. Resident key portability

Resident credentials stay on the token. When you move to a new machine — or after deleting the stub file — you can re-derive the client-side handle without re-enrolling.

Client-side — on the new machine, or after deleting the stub
ssh-keygen -K

Enter the FIDO2 PIN and touch the token. OpenSSH queries the token for all discoverable credentials and writes stub files to the current directory (id_ed25519_sk_rk and id_ed25519_sk_rk.pub by default). Move them to ~/.ssh/ and rename as needed, or load directly into the agent:

Client-side
ssh-add -K

The public key that was already in authorized_keys on the server is unchanged. No re-enrollment required on the server side — the credential is the same, just re-derived on the new client.

Token capacity. Most FIDO2 tokens hold 25 discoverable credentials. If you enroll many services, slots fill up. fido2-token -L && fido2-token -I /dev/hidraw1 shows remaining capacity on supported tokens. Non-resident keys do not consume on-token slots.

14. Git hosting services — context note

The same FIDO2 SSH key you enrolled in this lab can authenticate to GitHub, GitLab, Codeberg, and any other service that accepts SSH public keys. No additional enrollment on the token, no new key generated — the same sk-ssh-ed25519@openssh.com public key goes into the service's SSH key settings page, and the same PIN + touch flow authenticates git clone, git push, and git pull operations.

This lab does not require you to create or use personal accounts on any external service. The following is for context — "here is what you would do" — not a required step.

# GitHub example — no account required in this lab
# 1. Copy the public key:
cat ~/.ssh/id_ed25519_sk.pub

# 2. Paste it into: GitHub → Settings → SSH and GPG keys → New SSH key

# 3. Test (requires a GitHub account):
ssh -T git@github.com

# Expected: "Hi username! You've successfully authenticated..."
# Flow: PIN prompt, then touch — identical to the loopback lab

The hardware credential is not service-specific. One token can authenticate to many SSH endpoints — each with its own authorized_keys or equivalent, all trusting the same public key. This is a significant operational advantage over password-per-service: rotating the token credential requires only updating the public key in each service's settings, not remembering which services use which password.

rsync --partial is good practice for any large transfer to a remote host where network interruptions are plausible. It is not a FIDO2 mitigation — as demonstrated in section 10, TTL expiry and token removal do not interrupt established transfers. Use it as general SSH hygiene, not as a workaround for credential behavior.

15. Connection to the blue/yellow separation model

In this lab, one token handled both local login (via the PAM-FIDO lab) and SSH authentication. The same physical device, two different enrollment paths, two different trust domains.

The privilege separation lab (fido_separation_lab_v0.1) extends this into a deliberate two-token model:

Key colorRoleEnrolled for
BlueIdentity / loginPAM login (fido2-local-auth); SSH to personal hosts
YellowPrivilege elevationsudo PAM (fido-yellow-sudo); SSH to privileged or production hosts only
RedBreak-glass emergencyPassword-only account; offline passphrase

Applied to SSH: the blue token's public key goes in authorized_keys on personal development hosts. The yellow token's public key goes in authorized_keys only on hosts that require elevated trust. A credential compromise of the blue key does not grant access to yellow-gated hosts. A stolen token without the PIN grants access to nothing.

This is the same trust separation architecture, extended from local login to networked access.

16. Rollback

SSH and PAM are fully independent. Rolling back this lab requires no PAM changes.

Remove the FIDO2 public key from authorized_keys

Server-side — edit as FIDO_USER
nano ~/.ssh/authorized_keys

Delete the line beginning with sk-ssh-ed25519@openssh.com (and the backup key line if enrolled). Save the file.

Revert sshd_config

Server-side — sudo
sudo test -e /etc/ssh/sshd_config.bak && sudo cp -a /etc/ssh/sshd_config.bak /etc/ssh/sshd_config || true
sudo sshd -t
sudo systemctl reload ssh

Clean up agent and stub files

Client-side
ssh-add -D
rm -f ~/.ssh/id_ed25519_sk ~/.ssh/id_ed25519_sk.pub
rm -f ~/.ssh/cm-*

Remove ControlMaster config (if added)

Client-side
nano ~/.ssh/config

Remove or comment out the Host localhost ControlMaster block.

The resident credential on the token is not removed by rollback. It remains on the token until explicitly deleted with fido2-token -D /dev/hidraw1 or the token is reset. Removing the public key from authorized_keys is sufficient to revoke access — the credential on the token becomes harmless without a trusting server.

17. Troubleshooting

Always start with ssh -vvv localhost and ssh-add -l before investigating anything else.

SymptomLikely causeDiagnostic / fix
Permission denied (publickey)Agent empty (TTL expired or never loaded); public key not in authorized_keys; wrong userssh-add -l — if empty, run ssh-add -K. If loaded, check ~/.ssh/authorized_keys on the server for the matching sk- line.
sign_and_send_pubkey: signing failed for ED25519-SKToken interaction failed — wrong PIN, no touch within timeout, token removed during authConfirm token is inserted; rerun ssh localhost; enter correct PIN; touch when LED flashes.
agent refused operationAgent has no key loadedssh-add -l (confirm empty); ssh-add -K to load.
no mutual signature algorithmServer OpenSSH too old to accept sk- key types (pre-8.2)ssh -V on the server. Lab VM is confirmed at 10.3p1. Only relevant when connecting to external servers.
Token not found by fido2-token -LUSB passthrough not configured in VirtualBox; token not insertedPower off VM; VirtualBox Settings → USB → add filter for the token; power on; reinsert token.
PIN prompt never appears during ssh-add -KToken not inserted; libfido2 not finding the devicefido2-token -L; confirm /dev/hidraw1 is present; check ls -l /dev/hidraw* permissions.
ControlMaster connection not reusedDifferent hostname used (localhost vs 127.0.0.1); ControlPath socket missingls ~/.ssh/cm-*; use the exact hostname specified in ~/.ssh/config.
Backup token fails to authenticateBackup public key not in authorized_keys; verify-required not prepended to backup key linecat ~/.ssh/authorized_keys — confirm both sk- lines are present with verify-required.
ssh -vvv shows the client's side onlyClient verbose output can't see why the server rejected the key — a Match block, verify-required enforcement, or key-type restriction all live server-sidesudo journalctl -u ssh -f on the server (same box, in this lab) while retrying the connection.

Reading journalctl -u ssh for FIDO2 failures (server side)

sshd is a real systemd unit — unlike PAM debugging elsewhere in this course, no grepping is needed, since the unit itself scopes the log to exactly this daemon:

sudo journalctl -u ssh -f

Run that in a spare terminal, then retry ssh localhost from the client. A key rejected for missing verify-required looks different from one rejected for an unrecognized key type — the journal line names which check failed, where ssh -vvv on the client just sees "signing failed" or "no mutual signature algorithm" without the server's reasoning.

Reading ssh -vvv output for FIDO2 failures

Look for these lines in verbose output:

# Agent offering the key:
debug1: Offering public key: ... SK ...

# Server accepts the key type, challenge issued:
debug1: Server accepts key: ...

# Token interaction attempt:
debug1: sign_and_send_pubkey: signing using sk-ssh-ed25519@openssh.com

# Failure at token:
sign_and_send_pubkey: signing failed for ED25519-SK ... agent refused operation

If the server never accepts the key type, the issue is in authorized_keys or server configuration. If the server accepts but signing fails, the issue is token interaction — PIN, touch, or token presence.

18. References and source notes

  • ssh-keygen(1) — documents -t ed25519-sk, -O resident, -O verify-required, -K (download resident credentials).
  • ssh-add(1) — documents -K (load resident credentials), -t (TTL), -d (delete), -l (list).
  • sshd_config(5) — documents Match User, PasswordAuthentication, PubkeyAuthentication, PubkeyAuthOptions verify-required.
  • ssh_config(5) — documents ControlMaster, ControlPath, ControlPersist.
  • OpenSSH PROTOCOL.u2f — wire protocol specification for sk- key types; documents the credential handle format and assertion flow.
  • FIDO Alliance CTAP2 specification — discoverable credential (resident key) design; explains why the credential handle in the stub file cannot be used without the token's secure element.
  • Yubico developer documentation — FIDO2 SSH key setup, resident credential management, and token capacity guidance.