6 ms·

Do Users Verify SSH Keys? (2011) [pdf]

49 points by 8organicbits 1y ago

EPendragon 1y ago
The abstract for this paper is fire
egberts1 1y ago
Yeah and whatever you do, don't be deploying SSHFP DNS record unless its DNS server is running DNSSEC AND ... AND your web browser/client is also using DNSSEC-verified query responses too (either thru your dedicated DNSSEC-enabled resolver and/or specially-modified /etc/resolv.conf Source: https://egbert.net/blog/tags/sshfp.html https://egbert.net/blog/tags/sshfp.html
- tptacek 1y ago
  SSHFP is baffling to me. The entire point of the DNS is to make introductions between parties with no preexisting relationship, which is exactly not how an SSH cluster works. SSH already has a (very good) certificate system that solves the same problem. Why would you include the global DNS among your trust anchors!?
  - egberts1 1y ago
    It was for me too. But when ordered by my boss to implement SSHFP, the hacker in me took it apart and made this checklist. In hindsight, it isn't a very useable DNS RCODE. And yes, it is preferable to use SSH certificates (for as long as you are aware that private keys must be guarded jealously). We need PAKE capability in SSH, preferably Signal-like protocol for authentication.
    - egberts1 1y ago
      Oh yeah, you are better off having a separate but hardened SSH authentication server that is to be consulted by such SSH gateways, just to protect the private keys. And use DTLS (two sets of client and server PKI , one for each direction) to guard that link between SSH gatewat(s) and SSH authentication (certificate-based) server. The starting point of all this: https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand/ https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...
radial_symmetry 1y ago
I appreciate the to-the-point abstract
- tobinfricke 1y ago
  Also in compliance with Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."
kqr 1y ago
I feel like this is unnecessarily reductive. The initial handshake is always fraught with security problems. I struggle to see a scenario in which a bad actor is able to give me the address of a bad machine, yet not be able to trick me into their host key being the correct one. I would definitely however spend effort into verifying a host key that changes unexpectedly.
- jrochkind1 1y ago
  I'll be honest, I have never spent effort on a host key that had changed unexpectedly, and at least a few have.
  - kragen 1y ago
    I've often called people on the phone and stuff. It depends somewhat on what's at stake. Authenticating users with SSH passwords puts much more at stake than using public keys, since an attacker who can get you to send your unencrypted password to a malicious server once can steal your account; deploying PAKE algorithms (successors to SRP, see https://eprint.iacr.org/2021/1492.pdf https://eprint.iacr.org/2021/1492.pdf) could mitigate that, but I don't think any shipped SSH version has ever supported a PAKE algorithm.
- amelius 1y ago
  We need a new protocol, where installing the OS of a new machine automatically installs a trusted key from an inserted USB drive, so that the machine automatically becomes part of the "enclave". Or something like that.
  - organsnyder 1y ago
    This is common in corporate environments.
  - MaxMatti 1y ago
    The paper does mention that you can have your ssh keys signed by a ca, so in a company the it staff could configure everybodys os to only trust ssh keys signed by the organization.
dcminter 1y ago
I guess the interesting question to me is: how often does this matter? How many successful mitm attacks on ssh connections are there and in what sort of circumstances do they occur? It seems like it ought to matter, but if roughly nobody verifies and yet the sky has not fallen - does it?
- 1oooqooq 1y ago
  nobody robbed my house in years. i still lock the door. it's so banal to check host keys.
- marcosdumay 1y ago
  It's only for the first connection, and it's very rare that targets are valuable on the first connection. On the other hand, we know of at least two suppliers of software that run with elevated access everywhere (including the dev side of every advanced military) that have been breached by unknown parties for years. The most likely explanation, by far, is that the sky only didn't fall yet because nobody wants it to. And that leaves us vulnerable to somebody suddenly wanting it.
vbezhenar 1y ago
It's really ridiculous that ssh does not use standard PKI which is deployed everywhere. So unsecure.
- dspillett 1y ago
  To use the style of server identity management we use for, say, HTTPS, you need a “trusted” 3rd party involved to sign certificates. This is impractical for SSH in many (most?) cases for several reasons (SSH does support cert based identification and authentication, but there are not many circumstances where this is more practical than, or otherwise preferable to, TOFU for SSH). In fact, many people who don't properly understand SSH's trust-on-first-use system (so don't actually verify server certificate fingerprints) argue for browsers to support it as an option alongside the current certificate signing & verification regimes.
- kragen 1y ago
  By "standard PKI which is deployed everywhere" do you mean SSL certificates? That would make you vulnerable to dozens of poorly secured CAs throughout the world; any attacker who could penetrate one of them could then use that access to MITM any SSH connection in the world (if they could additionally spoof DNS). SSL certificates are probably the best we can do for the "talk to a server you've never heard of" scenario, but we can do enormously better for the scenario where you're SSHing into a server you already have a pre-existing trust relationship with.
  - vbezhenar 1y ago
    It is good enough for websites, it will be good enough for ssh. "pre-existing trust relationship" prevents from rotation keys which is standard security measure (unheard of in ssh, of course).
    - kbolino 1y ago
      Let us set aside the reasons why SSH adopted a different certificate format (namely, that X.509 is much more complex than they needed at the time). WebPKI only realistically serves a small portion of the SSH hosts out there. This is quite different from the situation with HTTPS. Even so, this would still be very convenient and useful. As I said elsewhere, I think this is sub-5% of SSH servers. X.509 more broadly could replace SSH certificates. Many institutional settings already have trust stores set up to include their in-house CAs. Public clouds and major hosting providers could also set up their own CAs, but they would have trouble distributing them (cf. AWS RDS, for example). Now we're probably up to 25% or so of deployed SSH servers. In the case of clouds, though, this adds a massive new exploitation vector (IP reassignment) and thus puts pressure on expiration/revocation. The rest are going to need self-signed certs. Between the non-WebPKI CA distribution problem and the probable predominance of self-signed certs, trust-on-first-use would still be the norm, and so relying on pre-existing trust relationships would still be necessary. We could augment TOFU/known-hosts with some kind of certificate or CA pinning rather than just key pinning, though. So, again, while I think adopting X.509 isn't a bad idea, and makes a lot more sense today than it did in 2010 (pre-Heartbleed!) when SSH added certificates, it's not really solving the problem that SSH has much better than today's solutions, no matter how well it solves the problem that HTTPS has.
kemotep 1y ago
Thanks for sharing this! Yesterday I was just wondering about ssh key verification techniques for third party services. SSH keys are amazing, portable and in some ways easier to use than Passkeys. But for them to successfully replace passwords and account configuration, which works decently well for a service like pico.sh, the user experience needs to be improved significantly. Not impossible but what does become a continuous and ongoing problem is verification.
erikerikson 1y ago
Fails to mention that you can paste in the expected key. Of course if there is a compromise of the source the key is copied from that no help but that's a higher bar. Still easy and doesn't rely on human frailty.
- jon-wood 1y ago
  Or you can use SSH certificates, where you work on the basis that if the host key is signed by the correct CA then it's legit. No more tofu required beyond need to trust whatever source you got your CA's public key from.
hk1337 1y ago
Coupled with that you have to have a matching private key that you created and using ssh-config, is this even necessary?
teeray 1y ago
I kinda like the approach Github does: they just publish their fingerprints here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... This is served over TLS, so it's no worse than TLS. You can also benefit from the paved road that LetsEncrypt has provided. It might not be as smooth as SSH CAs once they're set up, but setting those up and the Day 2 operations involved isn't nearly as straightforward.
deleted 1y ago
[deleted]
beala 1y ago
Terminal.shop lets you order coffee over ssh, which is kind of novel and fun. I did it, and the coffee was good! This post reminded me that they've gotten enough questions about security that they've added this to their FAQ: > is ordering via ssh secure?# you bet it is. arguably more secure than your browser. ssh incorporates encryption and authentication via a process called public key cryptography. if that doesn’t sound secure we don’t know what does. [1] I think this is wrong though for exactly the reasons described in this post. TLS verifies that the URL matches the cert through the chain of trust, whereas SSH leaves this up to the user to do out-of-band, which of course no one does. But then the author of this article goes on to say (emphasis mine): > This result represents good news for both the SSL/TLS PKI camps and the SSH non-PKI camps, since SSH advocates can rejoice over the fact that the expensive PKI-based approach is no better than the SSH one, while PKI advocates can rest assured that their solution is no less secure than the SSH one. Which feels like it comes out of left field. Certainly the chain of trust adds some security, even if it's imperfect. I know many people just click through the warning, but I certainly don't. [1] https://www.terminal.shop/faq https://www.terminal.shop/faq
- tw04 1y ago
  >TLS verifies that the URL matches cert through the chain of trust, I think you need to point out that TLS utilizes the browsers cert store for that chain of trust. If a bad actor acquires an entity that has a trusted cert, or your cert store is compromised, that embedded cert store is almost entirely useless which has happened on more than one occassion (Chinese government and Symantec most recently). https://expeditedsecurity.com/blog/control-the-ssl-cas-your-browser-trusts/ https://expeditedsecurity.com/blog/control-the-ssl-cas-your-... This is typically caught pretty quickly but there's almost nothing a user can do to defend against a chain of trust attack. With SSH, while nobody does it, at least you have the ability to protect yourself.
- zie 1y ago
  in SSH, it's a two-way handshake, the client ordering the coffee also gets a cert to prove their identity. In browser land, the client browser doesn't get a cert to prove their identity, it's one-way only. Certainly TLS supports client certs, browsers(at least some) technically even implement a version, but the UX is SOOOO horrible that nobody uses it. Some people have tried, the only people that have ever seen any success with client side authentication certificates over a web browser are webauthn/passkeys and the US Military(their ID cards have a cert in them). webauthn/passkeys are not fully baked yet, so time will tell if they will actually be a success, but so far their usage is growing.
zie 1y ago
We transfer ACH files(i.e. paychecks) via SSH(SFTP) to several banks. You better believe I check keys. One of the banks forces key rotation every 2-ish years. I absolutely verify it every rotation and delete the old keys. Occasionally it fails, almost always it's something unexpected happening, but occasionally we catch their errors(verified by connecting from various endpoints/DNS queries/etc). We used to call them all the time whenever that happened. Now we just auto-retry on failure in an hour and that fixes the issue all of the time(so far). We only re-try once and then fail with a ticket. Most of us like our paychecks, so we are pretty good about getting that ticket resolved quickly.
chuckadams 1y ago
No, and expecting users to actually do so is a sign that something is very wrong about the process. TOFU turns out to be good enough for most purposes anyway, but if a key changes (perhaps the server was reimaged) then verifying it is about as friendly as a tax audit. Or using GPG.
- SoftTalker 1y ago
  Yeah that is my experience. Users don't understand public key cryptography. You ask them for their public key and they send you the private one. They use the same key everywhere. They don't understand the difference between a host key and a login key. Ask them to do anything with their authorized_keys file and your next ticket will be "I'm locked out of my system." They do understand passwords, and most can manage an SMS code as a second factor. That's about the limit of what you can count on.
  - chuckadams 1y ago
    I've been doing this for 30 years and sometimes I give the wrong key file on the command line by forgetting to add '.pub' to the end. Far as I remember, I've always caught it before I managed to send it somewhere public, and thankfully most of my keys nowadays have a passphrase that gets remembered in my OS's keychain. But the UX is really that bad.
  - franga2000 1y ago
    Users can understand asymmetric crypto, but the tools are so convoluted for no reason that they usually just give up. I've had no trouble explaining it to "average" computer users and they got it completely, but then actually using the tools for signing or authentication was the nearly impossible part. Your key has to parts: public and private. You give your public part to the server so it knows it's talking to you because only you have the private part. The server has its own pair and it gives you its public part so you know you're not talking to an impostor server. The private key is never sent, it stays on your computer, but it does some fancy math so the server can know you have it.
- franga2000 1y ago
foxyv 1y ago
Until SSH servers implement PKIX based Host Key verification, it's always going to be fraught with issues like this. Users will just keep blindly accepting host keys because they "Don't got time for that."

49 points by 8organicbits 1y ago

The abstract for this paper is fire

Yeah and whatever you do, don't be deploying SSHFP DNS record unless its DNS server is running DNSSEC AND ... AND your web browser/client is also using DNSSEC-verified query responses too (either thru your dedicated DNSSEC-enabled resolver and/or specially-modified /etc/resolv.conf Source: https://egbert.net/blog/tags/sshfp.html https://egbert.net/blog/tags/sshfp.html

tptacek 1y ago
SSHFP is baffling to me. The entire point of the DNS is to make introductions between parties with no preexisting relationship, which is exactly not how an SSH cluster works. SSH already has a (very good) certificate system that solves the same problem. Why would you include the global DNS among your trust anchors!?
- egberts1 1y ago
  It was for me too. But when ordered by my boss to implement SSHFP, the hacker in me took it apart and made this checklist. In hindsight, it isn't a very useable DNS RCODE. And yes, it is preferable to use SSH certificates (for as long as you are aware that private keys must be guarded jealously). We need PAKE capability in SSH, preferably Signal-like protocol for authentication.
  - egberts1 1y ago
    Oh yeah, you are better off having a separate but hardened SSH authentication server that is to be consulted by such SSH gateways, just to protect the private keys. And use DTLS (two sets of client and server PKI , one for each direction) to guard that link between SSH gatewat(s) and SSH authentication (certificate-based) server. The starting point of all this: https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand/ https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...

radial_symmetry 1y ago

I appreciate the to-the-point abstract

tobinfricke 1y ago
Also in compliance with Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."

kqr 1y ago

I feel like this is unnecessarily reductive. The initial handshake is always fraught with security problems. I struggle to see a scenario in which a bad actor is able to give me the address of a bad machine, yet not be able to trick me into their host key being the correct one. I would definitely however spend effort into verifying a host key that changes unexpectedly.

jrochkind1 1y ago
I'll be honest, I have never spent effort on a host key that had changed unexpectedly, and at least a few have.
- kragen 1y ago
  I've often called people on the phone and stuff. It depends somewhat on what's at stake. Authenticating users with SSH passwords puts much more at stake than using public keys, since an attacker who can get you to send your unencrypted password to a malicious server once can steal your account; deploying PAKE algorithms (successors to SRP, see https://eprint.iacr.org/2021/1492.pdf https://eprint.iacr.org/2021/1492.pdf) could mitigate that, but I don't think any shipped SSH version has ever supported a PAKE algorithm.
amelius 1y ago
We need a new protocol, where installing the OS of a new machine automatically installs a trusted key from an inserted USB drive, so that the machine automatically becomes part of the "enclave". Or something like that.
- organsnyder 1y ago
  This is common in corporate environments.
- MaxMatti 1y ago
  The paper does mention that you can have your ssh keys signed by a ca, so in a company the it staff could configure everybodys os to only trust ssh keys signed by the organization.

dcminter 1y ago

I guess the interesting question to me is: how often does this matter? How many successful mitm attacks on ssh connections are there and in what sort of circumstances do they occur? It seems like it ought to matter, but if roughly nobody verifies and yet the sky has not fallen - does it?

1oooqooq 1y ago
nobody robbed my house in years. i still lock the door. it's so banal to check host keys.
marcosdumay 1y ago
It's only for the first connection, and it's very rare that targets are valuable on the first connection. On the other hand, we know of at least two suppliers of software that run with elevated access everywhere (including the dev side of every advanced military) that have been breached by unknown parties for years. The most likely explanation, by far, is that the sky only didn't fall yet because nobody wants it to. And that leaves us vulnerable to somebody suddenly wanting it.

vbezhenar 1y ago

It's really ridiculous that ssh does not use standard PKI which is deployed everywhere. So unsecure.

dspillett 1y ago
To use the style of server identity management we use for, say, HTTPS, you need a “trusted” 3rd party involved to sign certificates. This is impractical for SSH in many (most?) cases for several reasons (SSH does support cert based identification and authentication, but there are not many circumstances where this is more practical than, or otherwise preferable to, TOFU for SSH). In fact, many people who don't properly understand SSH's trust-on-first-use system (so don't actually verify server certificate fingerprints) argue for browsers to support it as an option alongside the current certificate signing & verification regimes.
kragen 1y ago
By "standard PKI which is deployed everywhere" do you mean SSL certificates? That would make you vulnerable to dozens of poorly secured CAs throughout the world; any attacker who could penetrate one of them could then use that access to MITM any SSH connection in the world (if they could additionally spoof DNS). SSL certificates are probably the best we can do for the "talk to a server you've never heard of" scenario, but we can do enormously better for the scenario where you're SSHing into a server you already have a pre-existing trust relationship with.
- vbezhenar 1y ago
  It is good enough for websites, it will be good enough for ssh. "pre-existing trust relationship" prevents from rotation keys which is standard security measure (unheard of in ssh, of course).
  - kbolino 1y ago
    Let us set aside the reasons why SSH adopted a different certificate format (namely, that X.509 is much more complex than they needed at the time). WebPKI only realistically serves a small portion of the SSH hosts out there. This is quite different from the situation with HTTPS. Even so, this would still be very convenient and useful. As I said elsewhere, I think this is sub-5% of SSH servers. X.509 more broadly could replace SSH certificates. Many institutional settings already have trust stores set up to include their in-house CAs. Public clouds and major hosting providers could also set up their own CAs, but they would have trouble distributing them (cf. AWS RDS, for example). Now we're probably up to 25% or so of deployed SSH servers. In the case of clouds, though, this adds a massive new exploitation vector (IP reassignment) and thus puts pressure on expiration/revocation. The rest are going to need self-signed certs. Between the non-WebPKI CA distribution problem and the probable predominance of self-signed certs, trust-on-first-use would still be the norm, and so relying on pre-existing trust relationships would still be necessary. We could augment TOFU/known-hosts with some kind of certificate or CA pinning rather than just key pinning, though. So, again, while I think adopting X.509 isn't a bad idea, and makes a lot more sense today than it did in 2010 (pre-Heartbleed!) when SSH added certificates, it's not really solving the problem that SSH has much better than today's solutions, no matter how well it solves the problem that HTTPS has.

kemotep 1y ago

Thanks for sharing this! Yesterday I was just wondering about ssh key verification techniques for third party services. SSH keys are amazing, portable and in some ways easier to use than Passkeys. But for them to successfully replace passwords and account configuration, which works decently well for a service like pico.sh, the user experience needs to be improved significantly. Not impossible but what does become a continuous and ongoing problem is verification.

erikerikson 1y ago

Fails to mention that you can paste in the expected key. Of course if there is a compromise of the source the key is copied from that no help but that's a higher bar. Still easy and doesn't rely on human frailty.

jon-wood 1y ago
Or you can use SSH certificates, where you work on the basis that if the host key is signed by the correct CA then it's legit. No more tofu required beyond need to trust whatever source you got your CA's public key from.

hk1337 1y ago

Coupled with that you have to have a matching private key that you created and using ssh-config, is this even necessary?

teeray 1y ago

I kinda like the approach Github does: they just publish their fingerprints here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... This is served over TLS, so it's no worse than TLS. You can also benefit from the paved road that LetsEncrypt has provided. It might not be as smooth as SSH CAs once they're set up, but setting those up and the Day 2 operations involved isn't nearly as straightforward.

deleted 1y ago

[deleted]

beala 1y ago

Terminal.shop lets you order coffee over ssh, which is kind of novel and fun. I did it, and the coffee was good! This post reminded me that they've gotten enough questions about security that they've added this to their FAQ: > is ordering via ssh secure?# you bet it is. arguably more secure than your browser. ssh incorporates encryption and authentication via a process called public key cryptography. if that doesn’t sound secure we don’t know what does. [1] I think this is wrong though for exactly the reasons described in this post. TLS verifies that the URL matches the cert through the chain of trust, whereas SSH leaves this up to the user to do out-of-band, which of course no one does. But then the author of this article goes on to say (emphasis mine): > This result represents good news for both the SSL/TLS PKI camps and the SSH non-PKI camps, since SSH advocates can rejoice over the fact that the expensive PKI-based approach is no better than the SSH one, while PKI advocates can rest assured that their solution is no less secure than the SSH one. Which feels like it comes out of left field. Certainly the chain of trust adds some security, even if it's imperfect. I know many people just click through the warning, but I certainly don't. [1] https://www.terminal.shop/faq https://www.terminal.shop/faq

tw04 1y ago
>TLS verifies that the URL matches cert through the chain of trust, I think you need to point out that TLS utilizes the browsers cert store for that chain of trust. If a bad actor acquires an entity that has a trusted cert, or your cert store is compromised, that embedded cert store is almost entirely useless which has happened on more than one occassion (Chinese government and Symantec most recently). https://expeditedsecurity.com/blog/control-the-ssl-cas-your-browser-trusts/ https://expeditedsecurity.com/blog/control-the-ssl-cas-your-... This is typically caught pretty quickly but there's almost nothing a user can do to defend against a chain of trust attack. With SSH, while nobody does it, at least you have the ability to protect yourself.
zie 1y ago
in SSH, it's a two-way handshake, the client ordering the coffee also gets a cert to prove their identity. In browser land, the client browser doesn't get a cert to prove their identity, it's one-way only. Certainly TLS supports client certs, browsers(at least some) technically even implement a version, but the UX is SOOOO horrible that nobody uses it. Some people have tried, the only people that have ever seen any success with client side authentication certificates over a web browser are webauthn/passkeys and the US Military(their ID cards have a cert in them). webauthn/passkeys are not fully baked yet, so time will tell if they will actually be a success, but so far their usage is growing.

zie 1y ago

We transfer ACH files(i.e. paychecks) via SSH(SFTP) to several banks. You better believe I check keys. One of the banks forces key rotation every 2-ish years. I absolutely verify it every rotation and delete the old keys. Occasionally it fails, almost always it's something unexpected happening, but occasionally we catch their errors(verified by connecting from various endpoints/DNS queries/etc). We used to call them all the time whenever that happened. Now we just auto-retry on failure in an hour and that fixes the issue all of the time(so far). We only re-try once and then fail with a ticket. Most of us like our paychecks, so we are pretty good about getting that ticket resolved quickly.

chuckadams 1y ago

No, and expecting users to actually do so is a sign that something is very wrong about the process. TOFU turns out to be good enough for most purposes anyway, but if a key changes (perhaps the server was reimaged) then verifying it is about as friendly as a tax audit. Or using GPG.

SoftTalker 1y ago
Yeah that is my experience. Users don't understand public key cryptography. You ask them for their public key and they send you the private one. They use the same key everywhere. They don't understand the difference between a host key and a login key. Ask them to do anything with their authorized_keys file and your next ticket will be "I'm locked out of my system." They do understand passwords, and most can manage an SMS code as a second factor. That's about the limit of what you can count on.
- chuckadams 1y ago
  I've been doing this for 30 years and sometimes I give the wrong key file on the command line by forgetting to add '.pub' to the end. Far as I remember, I've always caught it before I managed to send it somewhere public, and thankfully most of my keys nowadays have a passphrase that gets remembered in my OS's keychain. But the UX is really that bad.
- franga2000 1y ago
  Users can understand asymmetric crypto, but the tools are so convoluted for no reason that they usually just give up. I've had no trouble explaining it to "average" computer users and they got it completely, but then actually using the tools for signing or authentication was the nearly impossible part. Your key has to parts: public and private. You give your public part to the server so it knows it's talking to you because only you have the private part. The server has its own pair and it gives you its public part so you know you're not talking to an impostor server. The private key is never sent, it stays on your computer, but it does some fancy math so the server can know you have it.
franga2000 1y ago

foxyv 1y ago

Until SSH servers implement PKIX based Host Key verification, it's always going to be fraught with issues like this. Users will just keep blindly accepting host keys because they "Don't got time for that."