1. Background and philosophy
This lab extends the FIDO2 hardware token from local PAM login — covered in the FIDO2 lab — to SSH authentication. The same physical token, the same PIN, the same touch gesture. A different enrollment path and a different trust domain.
How FIDO2 SSH differs from what you already built
| Property | PAM login (Module 1) | FIDO2 SSH (this lab) |
|---|---|---|
| Enrollment tool | pamu2fcfg | ssh-keygen |
| Credential stored in | /etc/security/u2f_keys | authorized_keys on server |
| Authentication path | PAM stack → pam_u2f.so | SSH key exchange → libfido2 |
| Private key location | Token secure element | Token secure element |
| PIN enforcement | pinverification=1 in PAM | -O verify-required at keygen + authorized_keys |
| Rollback | Remove PAM include, restore backup files | Remove public key from authorized_keys |
What the network adds — and what it introduces
This project teaches isolated-system security as a foundation, not a limitation. A system secured without assuming a network connection is often more defensible and lower-maintenance than one that depends on the network for its security properties. SSH is covered because it is a natural and important extension of hardware-backed authentication into networked access — not as a replacement for thinking carefully about what the network introduces.
The network adds reach: one token can authenticate to many hosts. It introduces attack surface: credential forwarding risks, lateral movement, and the possibility that an established session outlives the intended access window. Understanding both sides is the goal of this lab.
| Auth method | Private key location | Copied without token? | Requires PIN? | Requires touch? |
|---|---|---|---|---|
| Password SSH | Server (hashed) | N/A | Yes (the password) | No |
| File-based key SSH | ~/.ssh/id_ed25519 on disk | Yes — trivially | Optional (passphrase) | No |
| FIDO2 SSH (this lab) | Token secure element | No — physically impossible | Yes (verify-required) | Yes |
2. Lab topology: loopback as server and client
This lab uses a single VM as both the SSH server and the SSH client, connecting over loopback (ssh localhost). This removes the second-machine setup burden without obscuring the conceptual model.
The security property being demonstrated does not require a real network: the token, the key material, and the authentication flow are identical whether the server is on loopback or across the internet. If the lab works on loopback, it works on a real network.
Role labeling
Throughout this lab, every command block is labeled with the role it represents — even though both roles are on the same machine.
/etc/ssh/sshd_config, managing service state. Requires sudo.
Client-side — actions the end user takes: key generation, agent management, making connections.
Enable openssh-server
Server-side — sudosudo apt install openssh-server
sudo systemctl enable --now ssh
sudo systemctl status ssh
Confirm password login works before touching FIDO2
Verify that basic SSH works over loopback with a password before any FIDO2 configuration. This confirms the server is running and is your fallback throughout the lab.
Client-sidessh localhost
You should be prompted for your Unix password and land in a shell. Type exit to close the test session. Keep this terminal open — it is your breakglass connection throughout the lab.
3. Prerequisites and safety
Confirm OpenSSH version
Client-sidessh -V
Expected output on the lab VM:
OpenSSH_10.3p1 Debian-1, OpenSSL 3.6.2 7 Apr 2026
sk- key type support requires OpenSSH 8.2 or later. Version 10.3p1 is confirmed on Debian 13.5 trixie.
Confirm token is visible
Client-sidefido2-token -L
Expected: a line such as /dev/hidraw1: KEY-ID FIDO+U2F. If nothing is returned, confirm the token is inserted and USB passthrough is configured in VirtualBox. hidraw1 is illustrative — the actual number depends on other HID devices already present, so always use the path this command reports rather than assuming hidraw1.
Packages
Server-side — sudosudo apt update
sudo apt install openssh-client openssh-server libfido2-1
libfido2-1 is typically already installed if you completed the PAM lab. openssh-client and openssh-server are usually present on the lab VM.
Replace these placeholders before running commands
| Placeholder | Meaning |
|---|---|
FIDO_USER | The local user account that will use FIDO2 SSH. |
4. Generate a resident FIDO2 SSH key
Generate the key
Insert the FIDO2 token. Run the following as FIDO_USER:
ssh-keygen -t ed25519-sk -O resident -O verify-required -C "fido2-ssh-lab"
What each flag means:
| Flag | Effect |
|---|---|
-t ed25519-sk | Ed25519 curve with security key. Preferred over ecdsa-sk on current OpenSSH. Use ecdsa-sk only if connecting to a server running OpenSSH older than 8.2. |
-O resident | Store the credential as a discoverable (resident) credential on the token itself. The token holds the full credential — no private key material is written to disk. |
-O verify-required | Bake PIN (user verification) into the credential at enrollment. The token will require the PIN on every signing operation regardless of what the client or server requests. |
-C "fido2-ssh-lab" | Comment. Appears in authorized_keys and ssh-add -l output. Use something meaningful — token model, host, or purpose. |
During generation you will be prompted for the FIDO2 PIN, then asked to touch the token when it flashes. Accept the default file location (~/.ssh/id_ed25519_sk) and leave the passphrase blank when prompted — the token PIN is the protection.
Output files
Client-sidels -l ~/.ssh/id_ed25519_sk ~/.ssh/id_ed25519_sk.pub
cat ~/.ssh/id_ed25519_sk.pub
The public key line will look like:
sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab
The stub file — what it is and is not
The file id_ed25519_sk is called a stub file. Inspect it:
file ~/.ssh/id_ed25519_sk
head -1 ~/.ssh/id_ed25519_sk
/home/FIDO_USER/.ssh/id_ed25519_sk: OpenSSH private key
-----BEGIN OPENSSH PRIVATE KEY-----
Despite the header and the filename, this is not an exportable private key. The private key material is stored in the token's secure element and never leaves it. What the stub file contains is a credential handle — an identifier that tells the SSH agent which resident credential on the token to use when signing. The stub file alone cannot authenticate anything: it requires the physical token to complete any signing operation.
You can verify this: copy the stub file to a different machine, or try to use it with a different FIDO2 token of the same model. Authentication will fail in both cases. The credential is bound to the specific token that enrolled it.
Keep vs. delete — a usability argument
The stub file's only practical value is convenience: when it is present in ~/.ssh/ with a standard name, the SSH client discovers and offers it automatically — exactly like a traditional key file. Without it, you must run ssh-add -K explicitly at the start of each session to load the credential from the token into the agent.
That extra step is small, but it matters. Security controls that create friction get worked around. A user who finds the manual load step annoying enough will start looking for ways to skip the token entirely — that is the real risk, not the stub file existing on disk. The stub file alone is useless without the physical token: it contains a credential handle, not key material. An attacker who reads your home directory cannot authenticate with it.
The security difference between keeping and deleting the stub is therefore minimal. The usability difference is real. Keeping the stub is a reasonable choice on a personal, single-user machine. Deleting it is appropriate on a shared or sensitive machine where no handle should exist on disk, and the explicit ssh-add -K ceremony is the deliberate policy.
If you delete the stub and want to reduce the friction of the manual load step, add an alias or a small function to your shell profile:
Client-side — add to~/.bashrc or ~/.zshrc
# Load FIDO2 SSH credential from token into agent (with 2-hour TTL)
alias fido-load='ssh-add -K -t 2h'
Then source ~/.bashrc (or open a new terminal) and run fido-load once at the start of a session. One short command, PIN + touch, and the credential is loaded for two hours — functionally identical to the automatic stub discovery, without anything on disk.
ssh-keygen -K downloads the resident credential handle from the token and writes a new stub to the current directory.Exercise: delete the stub and verify auth still works
Remove the key from the agent first, then delete the stub:
Client-sidessh-add -d ~/.ssh/id_ed25519_sk
rm ~/.ssh/id_ed25519_sk
ls ~/.ssh/
Only id_ed25519_sk.pub should remain. Now reload the credential directly from the token:
ssh-add -K
ssh-add -l
The key should appear in the agent output. Authentication will work in section 8. Decide whether to keep the stub absent (and use the alias above) or recreate it for automatic discovery — document your reasoning.
5. Non-resident key comparison
Generate a second key without -O resident to see the difference:
ssh-keygen -t ed25519-sk -O verify-required -C "fido2-ssh-lab-nonresident" -f ~/.ssh/id_ed25519_sk_nr
Inspect both stub files:
Client-sidels -l ~/.ssh/id_ed25519_sk* 2>/dev/null
wc -c ~/.ssh/id_ed25519_sk_nr ~/.ssh/id_ed25519_sk.pub
| Property | Resident (-O resident) | Non-resident (default) |
|---|---|---|
| Credential stored on | Token only (discoverable) | Token + stub file required |
| Portability | Token alone; ssh-keygen -K re-derives stub anywhere | Must carry stub file to each client machine |
| If stub is lost | Re-derive from token with ssh-keygen -K | Credential is inaccessible — cannot recover the handle |
| Token storage slots used | One slot per credential (tokens have limited capacity, typically 25) | No on-token storage slot used |
| When to prefer | Carry one token, use on multiple machines | Fixed single client; low token slot availability; file on encrypted volume |
rm -f ~/.ssh/id_ed25519_sk_nr ~/.ssh/id_ed25519_sk_nr.pub
6. Copy the public key to the server
Append to authorized_keys
On loopback, the client and server are the same machine and the same user. Use ssh-copy-id:
ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub localhost
You will be prompted for your Unix password (the existing SSH session is password-authenticated). This appends the public key to ~/.ssh/authorized_keys.
Verify:
Client-sidecat ~/.ssh/authorized_keys
The line beginning with sk-ssh-ed25519@openssh.com is your FIDO2 public key.
Add server-side verify-required enforcement
The -O verify-required flag at keygen time bakes PIN enforcement into the credential itself — the token requires the PIN regardless of server policy. The server-side verify-required option in authorized_keys is an additional, independent layer: the server will reject a signature that was produced without user verification, even if the token did not require it.
Use both. Edit authorized_keys and prepend verify-required to the sk- key line:
nano ~/.ssh/authorized_keys
Change the sk- line from:
sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab
To:
verify-required sk-ssh-ed25519@openssh.com AAAA... fido2-ssh-lab
authorized_keys. If your password-based SSH public key is also in that file, leave it untouched — it is your current session's key.Set correct permissions
Client-sidechmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
7. ssh-agent, credential TTL, and ControlMaster
Confirm the agent is running
Client-sideecho $SSH_AUTH_SOCK
Expected: a path like /run/user/1000/keyring/ssh or /tmp/ssh-XXXX/agent.XXXX. If the variable is empty, start an agent: eval $(ssh-agent -s).
Load the resident credential
Client-sidessh-add -K
Enter the FIDO2 PIN when prompted, then touch the token when it flashes. The credential is now loaded into the agent.
ssh-add -K means "store in the macOS Keychain" — a completely unrelated operation. The FIDO2 resident key load command on macOS is different. This lab runs on Debian 13.5 where ssh-add -K correctly loads FIDO2 resident keys. Do not copy SSH commands from macOS-specific documentation without checking the flag meanings.Confirm the key is loaded
Client-sidessh-add -l
Expected: a line containing SK and your comment (fido2-ssh-lab).
Remove a key manually before TTL expires
Client-sidessh-add -d ~/.ssh/id_ed25519_sk.pub
Or remove all keys: ssh-add -D. Run ssh-add -K to reload.
Credential TTL — loading with a time limit
Instead of loading the credential indefinitely, load it with a lifetime:
Client-sidessh-add -K -t 2h
The agent will automatically drop the credential after two hours. The next SSH connection attempt after expiry will fail with Permission denied (publickey) — run ssh-add -l to confirm the agent is empty, then ssh-add -K to reload.
The TTL bounds the window during which an unattended machine with the token still inserted could be misused. Token removal closes that window immediately regardless of TTL.
sk- keys, the signing operation occurs on the token hardware at each new connection attempt. The token must be physically present. The TTL controls how long the agent keeps the handle; it does not cache signatures. This is testable: load with a TTL, remove the token, attempt a connection — it fails. Reinsert the token within the TTL window — it succeeds with touch only (no PIN re-entry, because the handle is still valid).ControlMaster — one auth ceremony, many connections
A complementary pattern to the TTL is ControlMaster: a persistent master SSH connection that multiplexes subsequent connections without re-invoking the agent. One PIN + touch authentication covers many git push, scp, and ssh commands in sequence.
Add to ~/.ssh/config (create the file if it does not exist):
nano ~/.ssh/config
Host localhost 127.0.0.1
ControlMaster auto
ControlPath ~/.ssh/cm-%r@%h:%p
ControlPersist 2h
With this in place, the first ssh localhost authenticates with PIN + touch and establishes the master. Subsequent ssh localhost commands in the same or other terminals reuse the master channel silently — no agent, no token, no prompt. After two hours of inactivity the master closes and the next connection starts a new ceremony.
ControlMaster to specific hosts you connect to frequently — not Host * — to avoid unintentional multiplexing to sensitive targets.8. Test the connection
With the credential loaded in the agent and the public key in authorized_keys:
ssh-add -K
ssh localhost
Expected behavior:
- The agent presents the
sk-key to the server. - The server issues an authentication challenge.
- The agent sends the challenge to the token; the token prompts for PIN (because
verify-requiredwas set at enrollment). - After PIN entry, the token flashes — touch it.
- Authentication succeeds; you receive a shell prompt.
whoami
exit
Subsequent connections within the same agent session
Client-sidessh localhost whoami
ssh localhost date
Each connection requires a fresh token interaction (PIN + touch) because verify-required is set. This is correct behavior for Scenario A. Sections 7 and 9 cover how to calibrate this with a TTL or ControlMaster when per-connection friction is too high for your workflow.
If a connection fails here, ssh -vvv shows the client's side — but since sshd is a systemd unit, sudo journalctl -u ssh -f run alongside it shows the server's reasoning too. Section 17 covers reading both together.
9. Scenario lab: verify-required, TTL, and touch-required
You have now used verify-required with per-connection PIN + touch. This section works through three scenarios that represent different operational contexts and tradeoffs. The goal is not to identify the "most secure" option — it is to understand what each control proves, in what context, for what user, at what moment in their session.
The options
| Option | PIN required? | Touch required? | When appropriate |
|---|---|---|---|
| Neither | No | No | Never for FIDO2 — defeats the purpose of hardware authentication |
touch-required (default for sk-) | No | Yes | Within an active session where per-connection PIN friction is undesirable and presence confirmation alone is sufficient; not for automation — FIDO2 tokens are human credentials |
verify-required | Yes | Yes | Interactive user login; high-value or infrequent access; contexts where proving identity (not just presence) matters at each connection |
Scenario A — High-value or infrequent access
Context: You are connecting to a production server, a privileged account, or any target where each connection is a meaningful event. Full PIN + touch on every connection is appropriate. The friction cost is accepted because each connection carries real risk.
This is what you have already configured and tested. No changes needed. PIN + touch on every ssh invocation.
Scenario B — Active working session with calibrated TTL
Context: The same developer, same token, connecting to a development host repeatedly during an afternoon of work. Full PIN + touch once to load the agent; subsequent connections within a two-hour window are transparent.
Client-sidessh-add -D
ssh-add -K -t 2h
ssh-add -l
The -t 2h flag loads the credential with a two-hour lifetime. Now connect several times:
ssh localhost date
ssh localhost whoami
ssh localhost uptime
First connection: PIN + touch. Subsequent connections within the TTL: touch only (the PIN was proven at load time; the TTL certifies the handle has not been refreshed since then, bounding the window).
This is stronger than a certificate file on disk: re-loading after TTL expiry requires the physical token and PIN — a file on disk does not decay and does not require re-proof of presence.
Scenario C — Calibration exercise
What threat does the TTL actually address? Think through each before reading the answer:
- You walk away from your desk and leave the token plugged in. A colleague sits down.
- You leave a session open overnight.
- You set the TTL to zero (disable caching). What changes?
- You set the TTL to 8 hours. What changes?
Answers:
- 1: A short TTL (e.g., 30 minutes) limits how long the plugged-in token can be used to make new connections without you. Token removal closes this window immediately.
- 2: An overnight TTL means the credential is gone by morning. No action from you required. The session itself (if you left one open) is unaffected — see section 10.
- 3: Every connection requires
ssh-add -K: PIN + touch, full ceremony. Maximum assurance, maximum friction. On a workflow with dozens ofgit pushoperations per hour this creates enough friction to drive workarounds. - 4: Matches a typical working day. At the end of the day the credential expires on its own. You need to make a conscious decision about the right value for your actual session length and risk.
Student deliverable: Write one paragraph justifying the TTL value you would choose for your own working context. What threat does it address? What friction does it accept? What does it not address?
Note on automation and service accounts
10. Failure tests and token removal behavior
Work through each test in order. These are the behaviors students who skip this section will encounter unexpectedly in production.
Standard failure tests
Client-side — load credential firstssh-add -K
Wrong PIN: Enter an incorrect PIN when prompted.
ssh localhost
Expected: sign_and_send_pubkey: signing failed for ED25519-SK or FIDO_ERR_PIN_INVALID. No password fallback offered. The server presents Permission denied (publickey).
No touch: Enter the correct PIN but do not touch the token when it flashes.
ssh localhost
Expected: the token times out waiting for touch; authentication fails. No fallback.
Password attempt after FIDO enforcement (section 11 required): After configuring the Match User block in section 11, remove the sk- key from the agent and try with password only.
ssh-add -D
ssh localhost
Expected: Permission denied (publickey). Password auth is disabled for this user.
Token removal — new connections vs. existing sessions
Open two terminals before beginning this exercise.
Test 1 — Token removed mid-session:
- Terminal A:
ssh localhost— authenticate with PIN + touch, stay in the shell. - Remove the FIDO2 token from the USB port.
- Terminal A: run commands in the open session.
ls,date,whoami. The session continues without interruption. - Terminal B:
ssh localhost— authentication fails. The agent has the handle but the signing requires the token hardware, which is absent.
An established SSH session runs on negotiated symmetric session keys. Once the handshake is complete, the agent and the token are not involved in maintaining the channel. The token passively closes the door to new connections — it does not evict sessions already inside.
dm-tool switch-to-greeter within seconds. SSH has no such mechanism. If active session eviction on token removal is required by your threat model, that is a separate, deliberate mechanism at the host session layer — not something SSH provides for free. The right question before building it: is this an open-session problem or a new-connection problem?Test 2 — TTL expiry during a file transfer:
This demonstrates that established transfers survive TTL expiry — and reveals the silent-failure behavior that trips up experienced engineers in production.
- Create a test file:
dd if=/dev/urandom bs=1M count=200 of=/tmp/testfile.bin - Load the credential with a 30-second TTL:
ssh-add -K -t 30 - Terminal A: start the transfer:
scp /tmp/testfile.bin localhost:/tmp/testfile-copy.bin - Terminal B: watch the agent:
watch -n1 ssh-add -l— observe the key disappear when the TTL expires. - Terminal A: observe that the transfer continues and completes normally after the TTL expires.
- After the transfer completes, attempt a new connection:
ssh localhost date
Expected: Permission denied (publickey). The error message looks identical to a misconfigured authorized_keys or wrong username. Nothing says "your credential expired."
Correct diagnostic sequence:
Client-sidessh-add -l
Output: The agent has no identities. — the credential expired. Recovery:
ssh-add -K
PIN + touch, full ceremony, credential reloaded.
authorized_keys, sshd configuration, or file permissions. The correct first diagnostic is always ssh-add -l to confirm the agent state before investigating anything else.Test 3 — Token removed with TTL still active:
- Load with TTL:
ssh-add -K -t 2h— PIN + touch. - Remove the token.
- Attempt a connection:
ssh localhost— fails. The agent has the handle but signing requires the token hardware. - Reinsert the token.
- Attempt a connection:
ssh localhost— succeeds with touch only. No PIN re-entry because the handle is still valid within the TTL window.
This confirms: the TTL retains the credential handle, not a cached signature. Each new connection still involves the token hardware. The TTL only controls how long you avoid re-running ssh-add -K.
Test 4 — ControlMaster resilience (if configured in section 7):
- Load with a short TTL:
ssh-add -K -t 30 - Connect via ControlMaster:
ssh localhost— PIN + touch, master established. - Wait 35 seconds for TTL to expire. Confirm with
ssh-add -l. - In the same terminal:
ssh localhost date— succeeds without PIN or touch. The ControlMaster channel is still alive and handles the new session. - In a new terminal, bypass ControlMaster:
ssh -o ControlMaster=no localhost date— fails. The agent is empty and a new master cannot be established.
Channel-level persistence (ControlMaster) and agent-level persistence (TTL) are independent mechanisms. Understanding which is active in a given situation determines the correct diagnostic when auth fails.
11. Server-side hardening: disable password auth for FIDO user
After verifying that FIDO2 SSH works reliably, disable password-based SSH for FIDO_USER specifically. Use a Match User block to scope the change — the breakglass account retains password access.
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo editor /etc/ssh/sshd_config
Append to the end of the file:
sshd_config additionMatch User FIDO_USER
PasswordAuthentication no
Reload sshd:
Server-side — sudosudo sshd -t
sudo systemctl reload ssh
sshd -t performs a configuration syntax check before reloading. If it reports errors, correct them before reloading.
Test
Client-sidessh-add -D
ssh localhost
Expected: Permission denied (publickey). Password auth is no longer offered for FIDO_USER. Now reload the credential and confirm FIDO2 still works:
ssh-add -K
ssh localhost whoami
Expected: PIN + touch, then your username.
Confirm the breakglass user (any user not matched by the Match User block) can still authenticate with a password from a separate terminal.
12. Backup token enrollment
Enroll a second token so that losing the primary does not lock you out of SSH.
Insert the backup token. Generate a second sk- resident key:
ssh-keygen -t ed25519-sk -O resident -O verify-required -C "fido2-ssh-lab-backup" -f /tmp/backup_key
Append the backup public key to authorized_keys:
cat /tmp/backup_key.pub >> ~/.ssh/authorized_keys
Add verify-required to the new line as you did in section 6. Then clean up the temporary file:
rm /tmp/backup_key /tmp/backup_key.pub
Test both tokens independently. Swap tokens and confirm each authenticates:
Client-sidessh-add -D
ssh-add -K
ssh localhost whoami
Repeat with the other token. Both should succeed independently.
13. Resident key portability
Resident credentials stay on the token. When you move to a new machine — or after deleting the stub file — you can re-derive the client-side handle without re-enrolling.
Client-side — on the new machine, or after deleting the stubssh-keygen -K
Enter the FIDO2 PIN and touch the token. OpenSSH queries the token for all discoverable credentials and writes stub files to the current directory (id_ed25519_sk_rk and id_ed25519_sk_rk.pub by default). Move them to ~/.ssh/ and rename as needed, or load directly into the agent:
ssh-add -K
The public key that was already in authorized_keys on the server is unchanged. No re-enrollment required on the server side — the credential is the same, just re-derived on the new client.
fido2-token -L && fido2-token -I /dev/hidraw1 shows remaining capacity on supported tokens. Non-resident keys do not consume on-token slots.14. Git hosting services — context note
The same FIDO2 SSH key you enrolled in this lab can authenticate to GitHub, GitLab, Codeberg, and any other service that accepts SSH public keys. No additional enrollment on the token, no new key generated — the same sk-ssh-ed25519@openssh.com public key goes into the service's SSH key settings page, and the same PIN + touch flow authenticates git clone, git push, and git pull operations.
This lab does not require you to create or use personal accounts on any external service. The following is for context — "here is what you would do" — not a required step.
# GitHub example — no account required in this lab
# 1. Copy the public key:
cat ~/.ssh/id_ed25519_sk.pub
# 2. Paste it into: GitHub → Settings → SSH and GPG keys → New SSH key
# 3. Test (requires a GitHub account):
ssh -T git@github.com
# Expected: "Hi username! You've successfully authenticated..."
# Flow: PIN prompt, then touch — identical to the loopback lab
The hardware credential is not service-specific. One token can authenticate to many SSH endpoints — each with its own authorized_keys or equivalent, all trusting the same public key. This is a significant operational advantage over password-per-service: rotating the token credential requires only updating the public key in each service's settings, not remembering which services use which password.
rsync --partial is good practice for any large transfer to a remote host where network interruptions are plausible. It is not a FIDO2 mitigation — as demonstrated in section 10, TTL expiry and token removal do not interrupt established transfers. Use it as general SSH hygiene, not as a workaround for credential behavior.
15. Connection to the blue/yellow separation model
In this lab, one token handled both local login (via the PAM-FIDO lab) and SSH authentication. The same physical device, two different enrollment paths, two different trust domains.
The privilege separation lab (fido_separation_lab_v0.1) extends this into a deliberate two-token model:
| Key color | Role | Enrolled for |
|---|---|---|
| Blue | Identity / login | PAM login (fido2-local-auth); SSH to personal hosts |
| Yellow | Privilege elevation | sudo PAM (fido-yellow-sudo); SSH to privileged or production hosts only |
| Red | Break-glass emergency | Password-only account; offline passphrase |
Applied to SSH: the blue token's public key goes in authorized_keys on personal development hosts. The yellow token's public key goes in authorized_keys only on hosts that require elevated trust. A credential compromise of the blue key does not grant access to yellow-gated hosts. A stolen token without the PIN grants access to nothing.
This is the same trust separation architecture, extended from local login to networked access.
16. Rollback
SSH and PAM are fully independent. Rolling back this lab requires no PAM changes.
Remove the FIDO2 public key from authorized_keys
Server-side — edit as FIDO_USERnano ~/.ssh/authorized_keys
Delete the line beginning with sk-ssh-ed25519@openssh.com (and the backup key line if enrolled). Save the file.
Revert sshd_config
Server-side — sudosudo test -e /etc/ssh/sshd_config.bak && sudo cp -a /etc/ssh/sshd_config.bak /etc/ssh/sshd_config || true
sudo sshd -t
sudo systemctl reload ssh
Clean up agent and stub files
Client-sidessh-add -D
rm -f ~/.ssh/id_ed25519_sk ~/.ssh/id_ed25519_sk.pub
rm -f ~/.ssh/cm-*
Remove ControlMaster config (if added)
Client-sidenano ~/.ssh/config
Remove or comment out the Host localhost ControlMaster block.
fido2-token -D /dev/hidraw1 or the token is reset. Removing the public key from authorized_keys is sufficient to revoke access — the credential on the token becomes harmless without a trusting server.17. Troubleshooting
Always start with ssh -vvv localhost and ssh-add -l before investigating anything else.
| Symptom | Likely cause | Diagnostic / fix |
|---|---|---|
Permission denied (publickey) | Agent empty (TTL expired or never loaded); public key not in authorized_keys; wrong user | ssh-add -l — if empty, run ssh-add -K. If loaded, check ~/.ssh/authorized_keys on the server for the matching sk- line. |
sign_and_send_pubkey: signing failed for ED25519-SK | Token interaction failed — wrong PIN, no touch within timeout, token removed during auth | Confirm token is inserted; rerun ssh localhost; enter correct PIN; touch when LED flashes. |
agent refused operation | Agent has no key loaded | ssh-add -l (confirm empty); ssh-add -K to load. |
no mutual signature algorithm | Server OpenSSH too old to accept sk- key types (pre-8.2) | ssh -V on the server. Lab VM is confirmed at 10.3p1. Only relevant when connecting to external servers. |
Token not found by fido2-token -L | USB passthrough not configured in VirtualBox; token not inserted | Power off VM; VirtualBox Settings → USB → add filter for the token; power on; reinsert token. |
PIN prompt never appears during ssh-add -K | Token not inserted; libfido2 not finding the device | fido2-token -L; confirm /dev/hidraw1 is present; check ls -l /dev/hidraw* permissions. |
| ControlMaster connection not reused | Different hostname used (localhost vs 127.0.0.1); ControlPath socket missing | ls ~/.ssh/cm-*; use the exact hostname specified in ~/.ssh/config. |
| Backup token fails to authenticate | Backup public key not in authorized_keys; verify-required not prepended to backup key line | cat ~/.ssh/authorized_keys — confirm both sk- lines are present with verify-required. |
ssh -vvv shows the client's side only | Client verbose output can't see why the server rejected the key — a Match block, verify-required enforcement, or key-type restriction all live server-side | sudo journalctl -u ssh -f on the server (same box, in this lab) while retrying the connection. |
Reading journalctl -u ssh for FIDO2 failures (server side)
sshd is a real systemd unit — unlike PAM debugging elsewhere in this course, no grepping is needed, since the unit itself scopes the log to exactly this daemon:
sudo journalctl -u ssh -f
Run that in a spare terminal, then retry ssh localhost from the client. A key rejected for missing verify-required looks different from one rejected for an unrecognized key type — the journal line names which check failed, where ssh -vvv on the client just sees "signing failed" or "no mutual signature algorithm" without the server's reasoning.
Reading ssh -vvv output for FIDO2 failures
Look for these lines in verbose output:
# Agent offering the key:
debug1: Offering public key: ... SK ...
# Server accepts the key type, challenge issued:
debug1: Server accepts key: ...
# Token interaction attempt:
debug1: sign_and_send_pubkey: signing using sk-ssh-ed25519@openssh.com
# Failure at token:
sign_and_send_pubkey: signing failed for ED25519-SK ... agent refused operation
If the server never accepts the key type, the issue is in authorized_keys or server configuration. If the server accepts but signing fails, the issue is token interaction — PIN, touch, or token presence.
18. References and source notes
ssh-keygen(1)— documents-t ed25519-sk,-O resident,-O verify-required,-K(download resident credentials).ssh-add(1)— documents-K(load resident credentials),-t(TTL),-d(delete),-l(list).sshd_config(5)— documentsMatch User,PasswordAuthentication,PubkeyAuthentication,PubkeyAuthOptions verify-required.ssh_config(5)— documentsControlMaster,ControlPath,ControlPersist.- OpenSSH
PROTOCOL.u2f— wire protocol specification forsk-key types; documents the credential handle format and assertion flow. - FIDO Alliance CTAP2 specification — discoverable credential (resident key) design; explains why the credential handle in the stub file cannot be used without the token's secure element.
- Yubico developer documentation — FIDO2 SSH key setup, resident credential management, and token capacity guidance.