7 ms·

> As far as I know, nobody ever verifies the signatures in a systematic way. If you'll forgive a semi-relevant ramble about SSH: The situation seems similar w

> As far as I know, nobody ever verifies the signatures in a systematic way. If you'll forgive a semi-relevant ramble about SSH: The situation seems similar with SSH. As far as I can tell, just about everyone goes with the approach of trust-blindly-on-first-use. I've made a habit of manually checking the public-key when I SSH into a new EC2 instance using PuTTY. It is not obvious how to do this. Existing tooling just isn't set up with it in mind. The fingerprint reported on the EC2 system-log uses a different hash function than PuTTY uses. (This is PuTTY's fault. It's behind OpenSSH.) I was able to get an answer on ServerFault, but I was surprised no-one had asked the question before. [0] With some lesser-known distros it seems to be outright impossible, as the fingerprint is not written to the system-log at all. Presumably the distro maintainers never check SSH fingerprints. Still less relevantly: the trust-blindly-on-first-use antipattern is known euphemistically as TOFU, for trust on first use. This term can also refer to where the public key is manually verified by the user on first use. [1] Very unhelpful for a term-of-art to be so ambiguous. [0] https://serverfault.com/q/996828/ https://serverfault.com/q/996828/ [1] https://en.wikipedia.org/wiki/Trust_on_first_use https://en.wikipedia.org/wiki/Trust_on_first_use

still_grokking 6y ago
I would recommend using the original tooling. The ssh CLI shows the fingerprint, and explicitly asks: $ ssh example.com The authenticity of host 'example.com (1.2.3.4)' can't be established. RSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. Are you sure you want to continue connecting (yes/no/[fingerprint])?
- MaxBarraclough 6y ago
  That's roughly what I've ended up doing. [0] Both OpenSSH and PuTTY ask you to confirm the fingerprint, but PuTTY's fingerprint scheme is MD5/hex, whereas modern versions of the OpenSSH client use SHA256/Base64. [0] https://serverfault.com/questions/996828/#comment1296514_996864 https://serverfault.com/questions/996828/#comment1296514_996...
forty 6y ago
SSH certificates avoid this issue and are used in real life. We make them with hashicorp vault, it works well and is really convenient.
- GordonS 6y ago
  Huh, I had no idea there was even such a thing as SSH certificates! Presuming you mean X.509 certificates, is this part of the standard spec (e.g. should work with OpenSSH etc)? Do you know if it works with PuTTY?
  - jbreiding 6y ago
    Here is a guide I've found useful in the subject, https://engineering.fb.com/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/security/scalable-and-secure-acce....
  - TimWolla 6y ago
    They are not based on X.509. I recommend taking a look at the CERTIFICATES section of the ssh-keygen man page, because I am unable to find a good reference on the web right now.
- MaxBarraclough 6y ago
  Have to admit I'm not very familiar with SSH certs. I'm generally just spinning up a fresh Ubuntu instance on EC2, so I have an entirely 'vanilla' SSH set-up. Small scale, so I don't especially mind the ritual of verifying the fingerprint. It's annoying though that it takes a few minutes for the EC2 system-log to be populated with the 'true value' of the fingerprint.
kccqzy 6y ago
Big companies that care about security use short-lived certificates for SSH, not simple private/public keys.
closeparen 6y ago
The instance itself needs to get its SSH host key signed, or upload its fingerprints somewhere, on first boot. This is easy enough to write, but making it any more secure than plain old TOFU is tricky.
- MaxBarraclough 6y ago
  > making it any more secure than plain old TOFU is tricky. To refer again to my complaint about this term: this isn't meaningful. It's not clear whether you mean trust blindly on first use (total failure to cryptographically verify the identity of the server) or manually check SSH fingerprint on first use (secure but inconvenient). Both get called TOFU. One meaning of the term is a valid means of establishing a secure channel, and the other is cybersecurity negligence. I'm not just being obtuse here, I really don't know which you mean. Manually checking fingerprints on first use seems like a pretty solid approach to SSH security, if you can trust the client to actually perform the check. Short-lived keys are the obvious improvement that this approach doesn't really support. Is that what you meant?
  - closeparen 6y ago
    Where do you get the fingerprints to check against? Three intuitive but wrong answers: a) By spawning the instance with a script that will post them to some API. Now you are just blindly trusting whatever gets posted to that API. If your provider offers instance identity documents, you could sign the request with that, but an attacker who can MITM within your VPC can presumably impersonate anyone to the instance metadata service. And if MITM within the VPC is not in your threat model, then blind trust (from another host in the VPC) is fine. b) By having an SSH CA service connect to the instance to grab its public key. The CA is extending the same blind trust as the admin would have in doing it manually. c) By having generated the host keypair somewhere other than the host. Now you have key material in the wrong places. I don't think there's such a clear-cut line here. "Verify" just pushes the blind trust somewhere else. I can think of some potentially better answers, for example write the public key to a block device, detach it from the new host, attach it to a verification service host and read it there. But this is kind of just reinventing networking... why do we trust the cloud provider's SAN more than its SDN?
laegooose 6y ago
Thank you for asking serverfault question and writing about it here. When I started to work with EC2 and SSH few years ago I encountered this problem. However I was overwhelmed with the stuff I had to do for actual business and had no energy to investigate. Next question is, how do we protect against this attack without waiting for few minutes and making all those clicks to find the fingerprints every time we launch an instance?
- MaxBarraclough 6y ago
  Someone mentioned SSH certificates. My understanding is that it's the 'proper' solution, but is more work to set-up. Whether it's worth it will presumably be a question of scale.
ryanlol 6y ago
> I've made a habit of manually checking the public-key when I SSH into a new EC2 instance using PuTTY. It is not obvious how to do this. Existing tooling just isn't set up with it in mind. This is a not necessarily useful if you use key auth. The attacker can’t perform an useful MITM attack on the connection no matter what key you accept from the server. At best the attacker could let you log into their server, but presumably you’d notice that you’re logged into the wrong server. I guess there are a few corner cases where this might help, such as blind scp uploads.
- MaxBarraclough 6y ago
  > The attacker can’t perform an useful MiTM attack on the connection no matter what key you accept from the server. True, thanks to public key crypto. > At best the attacker could let you log into their server This is another way of saying the machine you end up trusting could be under the control of an attacker. So yes, that's a serious concern. > presumably you’d notice that you’re logged into the wrong server Not if it's a fresh Ubuntu instance. Even if not, I'd expect SSH security to answer that definitively, on principle. SSH shouldn't be any sloppier than HTTPS.
  - DangitBobby 6y ago
    Maybe I'm being naive here, but if you spin up a new instance and immediately SSH into that via the given public IP, isn't the likelihood that someone could have meaningfully injected themselves between you and that previously unknown IP address vanishingly small?
    - lysium 6y ago
      Just monitor and mitm all ssh connections. It’s really easy.
      - MaxBarraclough 6y ago
        Right. The attacker might have compromised my local network, for instance.
vbezhenar 6y ago
Here's an approach that I'm using with small VPS providers. I'm using web remote access to install Linux (from pristine distribution ISO). That web remote access uses HTTPS, so I can be sure that all input/output is E2E protected from provider to my computer. After I install Linux, I boot it first time and access it using that HTTPS remove console. Then I'm running something like ssh-keygen -l -f /etc/ssh/ssh_host_ed2519.pub -E md5 (I might be wrong about this command, check mans before use, also it does not require root, so login under ordinary user) and I can see md5 hash of server ssh public key. I write it down (or rather print-screen), connect with Putty and verify it. I believe that this method provides as much chain of trust as possible.
- MaxBarraclough 6y ago
  That's the approach I use with Linode. Unlike EC2 there's no browser-viewable system-log that shows the public key, but it can be retrieved using a web-based terminal session. To find the fingerprint in PuTTY's obsolete format, the command I use is equivalent to the one you've given. It was provided in the ServerFault answer I linked: ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ed25519_key.pub I don't like comparing fingerprints by eye, but PuTTY doesn't make it easy to do otherwise. This issue doesn't arise with PowerShell's OpenSSH client, where I can just copy the fingerprint (in the new format) to the clipboard. (It lacks features like mouse support though, so I prefer to use PuTTY.) To find the fingerprint in the new format for OpenSSH, I believe the command is: ssh-keygen -l -f /etc/ssh/ssh_host_ed25519_key.pub > I believe that this method provides as much chain of trust as possible. Yes, can't do better than that. If your HTTPS session with your VPS provider has been compromised, it's all over anyway.
MertsA 6y ago
What's absolutely incredibly frustrating is that it's absurdly common to ignore SSH fingerprints even if they're staring you in the face. I just spun up a Fedora 32 cloud image on RamNode over the weekend and the default image oh-so-helpfully displays the SSH fingerprints as part of the MOTD so you can verify it just looking at the login prompt on console. Except apparently the script that generates that runs before the SSH keys are regenerated on the first boot of the image, so the fingerprints displayed don't match the keys on disk. Either no one else has noticed this bug yet or nobody has been bothered enough to fix it. There is a solution though, SSHFP records for the host. I just set them up after catching that issue just because of how frustrating blindly trusting on first use is. Now I can SSH to my servers even without anything for them being in known_hosts and it checks for SSHFP records and verifies the fingerprint and DNSSEC protects the SSHFP records.
- tptacek 6y ago
  SSHFP requires you to DNSSEC-sign your zones, which virtually nobody in the technology industry does, for a variety of reasons, some involving security. A more straightforward solution that doesn't require you to hitch your wagon to a moribund infrastructure overhaul is to use SSH certificates. SSH certificate infrastructures have other benefits beyond resolving the introduction problem; for instance, they're easier to manage than keys once set up, and, perhaps most importantly, they make it possible to issue short-lifespan keys based on single signon, which is much safer than having everyone register a canonical personal key.

still_grokking 6y ago

I would recommend using the original tooling. The ssh CLI shows the fingerprint, and explicitly asks: $ ssh example.com The authenticity of host 'example.com (1.2.3.4)' can't be established. RSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. Are you sure you want to continue connecting (yes/no/[fingerprint])?

MaxBarraclough 6y ago
That's roughly what I've ended up doing. [0] Both OpenSSH and PuTTY ask you to confirm the fingerprint, but PuTTY's fingerprint scheme is MD5/hex, whereas modern versions of the OpenSSH client use SHA256/Base64. [0] https://serverfault.com/questions/996828/#comment1296514_996864 https://serverfault.com/questions/996828/#comment1296514_996...

forty 6y ago

SSH certificates avoid this issue and are used in real life. We make them with hashicorp vault, it works well and is really convenient.

GordonS 6y ago
Huh, I had no idea there was even such a thing as SSH certificates! Presuming you mean X.509 certificates, is this part of the standard spec (e.g. should work with OpenSSH etc)? Do you know if it works with PuTTY?
- jbreiding 6y ago
  Here is a guide I've found useful in the subject, https://engineering.fb.com/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/security/scalable-and-secure-acce....
- TimWolla 6y ago
  They are not based on X.509. I recommend taking a look at the CERTIFICATES section of the ssh-keygen man page, because I am unable to find a good reference on the web right now.
MaxBarraclough 6y ago
Have to admit I'm not very familiar with SSH certs. I'm generally just spinning up a fresh Ubuntu instance on EC2, so I have an entirely 'vanilla' SSH set-up. Small scale, so I don't especially mind the ritual of verifying the fingerprint. It's annoying though that it takes a few minutes for the EC2 system-log to be populated with the 'true value' of the fingerprint.

kccqzy 6y ago

Big companies that care about security use short-lived certificates for SSH, not simple private/public keys.

closeparen 6y ago

The instance itself needs to get its SSH host key signed, or upload its fingerprints somewhere, on first boot. This is easy enough to write, but making it any more secure than plain old TOFU is tricky.

MaxBarraclough 6y ago
> making it any more secure than plain old TOFU is tricky. To refer again to my complaint about this term: this isn't meaningful. It's not clear whether you mean trust blindly on first use (total failure to cryptographically verify the identity of the server) or manually check SSH fingerprint on first use (secure but inconvenient). Both get called TOFU. One meaning of the term is a valid means of establishing a secure channel, and the other is cybersecurity negligence. I'm not just being obtuse here, I really don't know which you mean. Manually checking fingerprints on first use seems like a pretty solid approach to SSH security, if you can trust the client to actually perform the check. Short-lived keys are the obvious improvement that this approach doesn't really support. Is that what you meant?
- closeparen 6y ago
  Where do you get the fingerprints to check against? Three intuitive but wrong answers: a) By spawning the instance with a script that will post them to some API. Now you are just blindly trusting whatever gets posted to that API. If your provider offers instance identity documents, you could sign the request with that, but an attacker who can MITM within your VPC can presumably impersonate anyone to the instance metadata service. And if MITM within the VPC is not in your threat model, then blind trust (from another host in the VPC) is fine. b) By having an SSH CA service connect to the instance to grab its public key. The CA is extending the same blind trust as the admin would have in doing it manually. c) By having generated the host keypair somewhere other than the host. Now you have key material in the wrong places. I don't think there's such a clear-cut line here. "Verify" just pushes the blind trust somewhere else. I can think of some potentially better answers, for example write the public key to a block device, detach it from the new host, attach it to a verification service host and read it there. But this is kind of just reinventing networking... why do we trust the cloud provider's SAN more than its SDN?

laegooose 6y ago

Thank you for asking serverfault question and writing about it here. When I started to work with EC2 and SSH few years ago I encountered this problem. However I was overwhelmed with the stuff I had to do for actual business and had no energy to investigate. Next question is, how do we protect against this attack without waiting for few minutes and making all those clicks to find the fingerprints every time we launch an instance?

MaxBarraclough 6y ago
Someone mentioned SSH certificates. My understanding is that it's the 'proper' solution, but is more work to set-up. Whether it's worth it will presumably be a question of scale.

ryanlol 6y ago

> I've made a habit of manually checking the public-key when I SSH into a new EC2 instance using PuTTY. It is not obvious how to do this. Existing tooling just isn't set up with it in mind. This is a not necessarily useful if you use key auth. The attacker can’t perform an useful MITM attack on the connection no matter what key you accept from the server. At best the attacker could let you log into their server, but presumably you’d notice that you’re logged into the wrong server. I guess there are a few corner cases where this might help, such as blind scp uploads.

MaxBarraclough 6y ago
> The attacker can’t perform an useful MiTM attack on the connection no matter what key you accept from the server. True, thanks to public key crypto. > At best the attacker could let you log into their server This is another way of saying the machine you end up trusting could be under the control of an attacker. So yes, that's a serious concern. > presumably you’d notice that you’re logged into the wrong server Not if it's a fresh Ubuntu instance. Even if not, I'd expect SSH security to answer that definitively, on principle. SSH shouldn't be any sloppier than HTTPS.
- DangitBobby 6y ago
  Maybe I'm being naive here, but if you spin up a new instance and immediately SSH into that via the given public IP, isn't the likelihood that someone could have meaningfully injected themselves between you and that previously unknown IP address vanishingly small?
  - lysium 6y ago
    Just monitor and mitm all ssh connections. It’s really easy.
    - MaxBarraclough 6y ago
      Right. The attacker might have compromised my local network, for instance.

vbezhenar 6y ago

Here's an approach that I'm using with small VPS providers. I'm using web remote access to install Linux (from pristine distribution ISO). That web remote access uses HTTPS, so I can be sure that all input/output is E2E protected from provider to my computer. After I install Linux, I boot it first time and access it using that HTTPS remove console. Then I'm running something like ssh-keygen -l -f /etc/ssh/ssh_host_ed2519.pub -E md5 (I might be wrong about this command, check mans before use, also it does not require root, so login under ordinary user) and I can see md5 hash of server ssh public key. I write it down (or rather print-screen), connect with Putty and verify it. I believe that this method provides as much chain of trust as possible.

MaxBarraclough 6y ago
That's the approach I use with Linode. Unlike EC2 there's no browser-viewable system-log that shows the public key, but it can be retrieved using a web-based terminal session. To find the fingerprint in PuTTY's obsolete format, the command I use is equivalent to the one you've given. It was provided in the ServerFault answer I linked: ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ed25519_key.pub I don't like comparing fingerprints by eye, but PuTTY doesn't make it easy to do otherwise. This issue doesn't arise with PowerShell's OpenSSH client, where I can just copy the fingerprint (in the new format) to the clipboard. (It lacks features like mouse support though, so I prefer to use PuTTY.) To find the fingerprint in the new format for OpenSSH, I believe the command is: ssh-keygen -l -f /etc/ssh/ssh_host_ed25519_key.pub > I believe that this method provides as much chain of trust as possible. Yes, can't do better than that. If your HTTPS session with your VPS provider has been compromised, it's all over anyway.

MertsA 6y ago

What's absolutely incredibly frustrating is that it's absurdly common to ignore SSH fingerprints even if they're staring you in the face. I just spun up a Fedora 32 cloud image on RamNode over the weekend and the default image oh-so-helpfully displays the SSH fingerprints as part of the MOTD so you can verify it just looking at the login prompt on console. Except apparently the script that generates that runs before the SSH keys are regenerated on the first boot of the image, so the fingerprints displayed don't match the keys on disk. Either no one else has noticed this bug yet or nobody has been bothered enough to fix it. There is a solution though, SSHFP records for the host. I just set them up after catching that issue just because of how frustrating blindly trusting on first use is. Now I can SSH to my servers even without anything for them being in known_hosts and it checks for SSHFP records and verifies the fingerprint and DNSSEC protects the SSHFP records.

tptacek 6y ago
SSHFP requires you to DNSSEC-sign your zones, which virtually nobody in the technology industry does, for a variety of reasons, some involving security. A more straightforward solution that doesn't require you to hitch your wagon to a moribund infrastructure overhaul is to use SSH certificates. SSH certificate infrastructures have other benefits beyond resolving the introduction problem; for instance, they're easier to manage than keys once set up, and, perhaps most importantly, they make it possible to issue short-lifespan keys based on single signon, which is much safer than having everyone register a canonical personal key.