7 ms·

TOFU: Do You Check?

107 points by cedws 3y ago

NoZebra120vClip 3y ago
I was using a system at work, and so eventually I needed to ssh into it, and I was confronted with the usual TOFU prompt. So I contacted Support, to ask how I could ascertain the Host Keys in use and check their fingerprints. And they replied, "Mmm, we don't do that."
tedunangst 3y ago
Nobody ever mentions that TOFU requires the attacker to commit to the attack before they know whether you know the fingerprint. Somebody could set up on that public wifi and intercept ssh, but they'd be detected as soon as anybody with a previously known key tries to connect. They can't wait for you to connect to an unknown server, because they don't know what you don't know. If you're not sure about the network, you can connect to a known host first to check for funny business. Or turn on the option for ascii art keys.
- c7DJTLrn 3y ago
  Good point. If we're being imaginative, there's various side channels an attacker could use to carefully target their attack, such as seeing a new MAC or DHCP lease on the local network.
  - deleted 3y ago
    [deleted]
  - xg15 3y ago
    True. Also depends on what the target is. I.e., if the attacker wants to impersonate machines that are only avaliable on the local network, then it's probably reasonable to conclude that a machine which was never seen on that LAN before also hasn't yet connected to the impersonated host. However, that isn't true for "public" SSH endpoints, such as github, gitlab etc - e.g. if the attacker is impersonating the wifi if the nearest Starbucks to snoop on all the hipster solo-devs there, he'll probably be unable to impersonate github, because even people who use that wifi for the first time have probably connected to github before, on a different network.
- SoftTalker 3y ago
  > they'd be detected as soon as anybody with a previously known key tries to connect Would they? Or would the person just delete the old key from their ~/.ssh/known_hosts and accept the new one?
  - tedunangst 3y ago
    If you're deleting keys from known_hosts, you're not trusting on first use.
    - mjochim 3y ago
      That is correct. And still, many, many devs and admins would do exactly that. Which is why in many, many cases, Tofu really isn't Tofu but "trust whatever." When designing or evaluating security, one should not ignore that this is a part of reality.
      - arp242 3y ago
        That's a bit of a different issue from the "Let’s say you’re on a fresh machine" that this article is about. I agree that "automatic trust on first use" is "good enough" for most cases and people (especially with sshfp records), and to be honest I think the warning you get once that fails is strong enough: % git clone git@github.com:madmurphy/libconfini.git Cloning into 'libconfini'... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU. Please contact your system administrator. Add correct host key in /home/martin/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /home/martin/.ssh/known_hosts:118 Host key for github.com has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. It's a strong warning, with a manual fix that's more than "just press ok" (probably intentionally), and if you choose to ignore that then that's your problem. I'm not really sure what could be done better? A centralized https-like system comes with its own downsides. (The only complaint is that last "Please make sure you have the correct access rights and the repository exists" line, which is from git and not ssh, and a tad confusing; maybe it's possible for git to do better here?)
m_x 3y ago
Aren't SSH host certificates the answer? Why are they not more widely used?
- fragmede 3y ago
  Especially since they're distributable via DNS. If your server has an entry in DNS, you shouldn't need to confirm host key.
  - pritambaral 3y ago
    If the MITM is in the client's LAN, DNS can be spoofed too. A simpler solution would be to just use SSH certs for host keys.
- c7DJTLrn 3y ago
  According to the guy who invented them: "I'm a shitty shitty publicist" (Damien Miller)
- surfer7837 3y ago
  Wouldn't that require a responsible body (ie a CA) to verify who they say are. Not sure that's needed for SSH
  - cpach 3y ago
    Yes, but it can be an internal CA. There are even FOSS products for that: https://goteleport.com/ https://goteleport.com/
- tetha 3y ago
  We've switched to them based on Hashicorp Vault and I think they are an improvement in security and convenience. Before SSH host certificates, using tooling like ansible was seriously annoying, especially to the monthly ansible users, compared to the daily ansible users. If you tried using ansible once a month without our host key certificates, you'd have ansible barf about a dozen unknown host keys due to them being rebuilt and other things. Then you fix that, then they do that one ansible run, then it lies around for a month and back you go to square one. I'm not sure HOW you'd stage an actual attack there, but people got used to just accept SSH host keys whatsoever, so the vulnerability was there. Now we have host key signing based on Vault. This is a huge improvement in my book. Base infra guys know when they've reset a VM to a base image (which resets SSH host keys), so they know when to expect a TOFU request from their SSH upon first connect to the just rebuilt system, so they accept those. Afterwards, one of the first things the config management does is to sign those host keys and then the accepted host keys are usually deleted again. There probably is a window of vulnerability in there, but that's getting pretty hard to attack and I'm sure there are easier ones in the overall infrastructure. The main problem at this however is that you need some safe and secure place for the CA, and ideally automation to sign those host keys. And it needs to be enough of a problem to do all of that.
NovemberWhiskey 3y ago
Ideally, the DNS SSHFP record would help solve this problem from Internet-exposed services, in combination with DNSSEC of course.
Apreche 3y ago
I don't check. That's because I'm one of those weirdos who uses certificates, so the prompt never appears.
cesarb 3y ago
It's implied in the examples on this post, but when using public key authentication, an attacker with a stolen SSH host key can pretend to be the server to the client, but cannot pretend to be the client to the server. Which means that the attacker has to pretend to be the server, without being able to see how the server would answer. For instance, if the ssh connection was a "git pull", the attacker has to know enough about the repository state (and about how much of that state the git client has already seen) to avoid complaints from git; if the ssh connection is a shell connection, the attacker has to know enough about the target machine to make the command outputs seem realistic enough for that particular machine; and so on. And as for "SSH has no forward secrecy", I thought SSH used Diffie-Hellman or similar to derive the session key? The SSH host key (and the client's public key) is for authentication only, stealing them does not reveal anything about past sessions.
- mindslight 3y ago
  > [the attacker] cannot pretend to be the client to the server Isn't the most common way of identifying ssh clients still just public key authorization? Meaning the attacker can MITM the real client connecting to the server, satisfy the real client's request, and then likely carry on doing whatever it likes. edit: actually no - I can see where my intuition is wrong (out of date from the time of widespread password auth). If the ssh server only accepts input that's been effectively signed by the authentication key, then there's no way for the MITM to send its own requests. Yet another reason to not use password auth.
- progmetaldev 3y ago
  SSH does have forward secrecy. The OP stated in a previous comment that they have fixed this in the article.
dheera 3y ago
No, often I don't check, because my managers rush me to produce results on tight timelines, so I don't have time to "do things right". Slack message at 3pm asking for updates to be in a doc by 6pm. What do you expect? I guess the problem is less serious on a corporate VPN, but still, the principle applies everywhere. If you want security, give your employees time to do things right. Usually that means at least double what you think it will take.
- shadowgovt 3y ago
  It seriously shoots any Zero Trust claims a corporation makes in the foot if people are "yes"ing through that dialog. Of course, to really have Zero Trust you have to make it impractical for users to "yes" through the dialog.
  - dinosaurdynasty 3y ago
    For SSH: setup an SSH CA and then disable asking in the client config.
SoftTalker 3y ago
When I set up a system for a user, I pre-populate /etc/ssh/ssh_known_hosts for all our internal systems. But I've certainly been guilty of just hitting "yes" on first connection to a machine I'm unfamiliar with.
dmurray 3y ago
Trust On First Use is exactly what I do. If I have to connect to a new machine, I identify it by its hostname, and send it my credentials (whatever credentials the owner just sent me in plaintext). If I get the warning and I'm pretty sure I connected to this machine before, I ask whoever owns the machine "hey did you do something weird with this machine? Like reimage it or rotate SSL certs or change the DNS entry so it's actually a different machine?" 99% of the time I'm not connecting to a machine for the first time, so (if I may be allowed some bad maths) this reduces my attack surface by a factor of 100. (If the owner didn't send me plaintext credentials but instead added my SSH key, I don't expose anything by using that, but I immediately follow up by doing other sensitive things like scp'ing code or credentials there).
- shadowgovt 3y ago
  "hey did you do something weird with this machine? Like reimage it or rotate SSL certs or change the DNS entry so it's actually a different machine?" "I dunno."
  - Muromec 3y ago
    "Maybe the other department reprovisioned the bastion about 2 months ago, but maybe it was another fiscal year"
morelisp 3y ago
I tried to check but 99% of our clients rotate host keys whenever they restart the container or the LB sends you to a different node or whatever. And if you email them to verify they take 3 weeks to respond, which is 20 daily reports they now don't get and so don't want to pay for, so... nope.
3pt14159 3y ago
I had this phase where I tried to do everything the right way. I used to check whenever I thought I could, including to my own servers. And you know what? 100% of the time the check passed. After a while I sat back and thought to myself "is this really worth the cost in time?" And I concluded that no it wasn't. Plenty of people run scanners looking for bad stuff online, ranging from poisoned SSH to supply chain attacks on OSS. These are the people that find the baddies and the baddies eventually get targeted. Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint. It would be better if I scanned the web app that I'm working on for, say, SQL injection bugs or misapplied CORS headers or whatever else. Ultimately security checks are an economic activity. The likelihood you are targeted via a given threat, the payoff to the attacker if you are targeted, the downside to the attacker if they are caught, etc. I think being realistic is prudent.
- Muromec 3y ago
  >Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint ... and when you don't you drive up the probability of enabling the 4-stage attack that leaks private keys to Chines govt. Of course the probability is only 0.01% and it's worth to spend time on something better. And when you do check and other 1000 devs in the organization don't, what's the point, right? This is how this stuff happens and we all read a nice post morten here somewhat 2 years later.
- taberiand 3y ago
  I think that's the crux of the problem: we should do this check, and perhaps the first few times we do - but it's a chore every time and infrequent enough that the process has to be rediscovered, with the associated uncertainty that entails (sure I checked - but did I do it properly?) If it's so important to validate then it shouldn't be left up to a human to choose, it should be mandatory - but the process needs to be straightforward. Why isn't there a built in tool that we can trust to check for us when these alerts happen? Or at least streamline the process and provide certainty
  - 3pt14159 3y ago
seabass-labrax 3y ago
MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public. One of the security principles I live by is not to upload anything to the Internet that I wouldn't mind being shared completely in public. The only exceptions I make are video conferencing and email. Jitsi provides great end-to-end encryption for the former (although the 8x8 server's recent prohibition of anonymous use is disappointing).
- ghusto 3y ago
  Same. I presume every company I have an account on will be compromised at some point, and all my account data will be out in the wild. This presumption hasn't disappointed so far ;)
- cesarb 3y ago
  > MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public. This is not just about secrecy, but also about integrity. Someone who MITMs your connection could not only look at the code you published, but also modify it. Then later someone who was trusting you (or possibly even yourself on a new machine) will end up downloading and running the modified code.
  - seabass-labrax 3y ago
    In this particular case, I'm not sure it would be possible to modify the code in transit, because although you could trick the user into submitting the code into Fake GitHub, you wouldn't be able to authenticate as the user to submit your modified copy to Real GitHub. More importantly, in my opinion, the end-user has no way of verifying the integrity of the code if authentication was the only security measure. By signing releases or the commits themselves (which can be done with both PGP and SSH keys currently) the end-user can verify the integrity themselves.
tjk_ 3y ago
Bitbucket dev here. Firstly, thanks for the post, this topic is too often skimmed and as someone who worked directly on the host key rotation here, it was shocking to find how many devs had zero idea what "that prompt from git on a new computer" was. Secondly, I just wanted to say that there is an interesting side-effect to everything covered in that blog that effectively leads to a gap in most people's mental models (or maybe just mine). Because of the way these systems were defined/the way people interact with them, there are several conundrums when a large centralized host needs to deprecate an old key. If a key were suspected of being compromised, it seems obvious that you could just stop serving the old key and tell all of your users to be prepared to re-TOFU. However, since _most_ devs blindly type "yes" when interacting with a large git host, this effectively primes the pumps for any bad actor that have MITM control but have not actually stolen the keys. This gives them a relatively long (if not infinitely long) window of time where a user will not be surprised to see "the prompt" and blindly accept trust of a host that could be controlled by the bad actor. If a key was completely successfully compromised by a bad actor, and said bad actor had MITM control of the victim network, then requests with the _old_ key would never actually reach the correct host and just quietly continue working (assuming the bad actor was savy enough to setup a remote system to properly behave as a git host without a prior copy of the target repository). Damned if you do, damned if you don't, so always carefully TOFU. Also,not sure why the author linked off to a Bitbucket Cloud blog post for the SSH keys, they are documented here[1] along with our recommended best practices WRT TOFU. 1 - https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/ https://support.atlassian.com/bitbucket-cloud/docs/configure...
- c7DJTLrn 3y ago
  Thanks. I linked to the blog post because it was one of the first results in the search engine. How are you guys tackling host key rotation? Do you do it periodically, or on compromise? How do you protect such an important set of keys?
  - MaxBarraclough 3y ago
    I don't work for them, but: BitBucket changed their SSH keys earlier this year due to possible compromise of their secret keys. [0] I don't think they change them periodically. [0] https://bitbucket.org/blog/ssh-host-key-changes https://bitbucket.org/blog/ssh-host-key-changes edit I see eichin also linked to the same page
MarkSweep 3y ago
Maybe there should be an “expected key” argument to “git clone”. GitHub could include it in the copy-pasted clone command. Problem solved.
thealistra 3y ago
Isn’t this mechanism by default to check if the prompt shows on a used machine to detect cert rotation? And then you can ask the admin. Seems like too much work for checking the fingerprints if not in an obvious security first context.
pxx 3y ago
> Not to mention that SSH has no forward secrecy. What? This is not right, not even for the obsolete v1 ssh: https://utcc.utoronto.ca/~cks/space/blog/tech/SshForwardSecrecy https://utcc.utoronto.ca/~cks/space/blog/tech/SshForwardSecr...
- c7DJTLrn 3y ago
  Fixed, thanks.
foobiekr 3y ago
In the real world, "trust on first use" is literally true - it's never "check on first use." SSHFP DNS records would help a lot, or some other scheme. For all of the positive image ssh has, mostly because telnet and ftp were so ridiculously terrible, ssh-as-used is pretty mediocre. And the reality is: "No." https://www.usenix.net/system/files/login/articles/105484-Gutmann.pdf https://www.usenix.net/system/files/login/articles/105484-Gu...
- MaxBarraclough 3y ago
  > it's never "check on first use." Personally I always manually check on first use, but I agree this puts me in a small minority. I said something similar in a comment last year. [0] It's unfortunate that the term TOFU is ambiguous. In practice it's used to refer to check on first use and blindly assume it's ok on first use. [0] https://news.ycombinator.com/item?id=30366424 https://news.ycombinator.com/item?id=30366424
myself248 3y ago
Yeah I've done it. Not for Git but for SSH. I'll call the admin out-of-band and read him part of the fingerprint, and ask him to read another part back. Only done it a few times when I'm setting up a new machine and don't have an already-authenticated one to check against. If I've got suspicions about the network connection I'm on, I'll first ask him some questions about our adventures years ago, that nobody else would likely know about. And one time when the host key changed, and I wasn't expecting it, he said my call was how he knew he replaced the right box. Tongue-in-cheek, but only barely. The only time I immediately dismiss the key-changed message is when I'm cycling yet another raspberry pi image into the same IP as an old one or whatever, and I'm on the local network so I know exactly what's going on.
ghusto 3y ago
I think the attack in all cases would require pointing DNS of gitprovider.net to their own server. This means they control DNS on that network, which means they're running it. What am I doing on a network I don't trust with a machine I've never used? The example given is that I've started a new job. Well okay, their machine, their network, and I guess their corporate gitprovider.net account they're getting me to use, so to heck with it. If we're talking about using _my_ gitprovider.net account however, that's a different story. I simply wouldn't use my own account on a laptop that's not mine.
- c7DJTLrn 3y ago
  On the local network, there are ways to MITM without touching DNS, like ARP spoofing.
- toxik 3y ago
  Another good question is, why not post the fingerprint in a TXT record?
  - frutiger 3y ago
    They are more likely to be found in SSHFP records.
upofadown 3y ago
SSH could really use a better default key fingerprint. It's 42 characters of base64. So a hodgepodge of upper and lower case Latin characters with some numbers and special characters mixed in. Not even any spaces to give it some structure. Basically just the output of of a 256 bit hash dumped directly to the terminal. Perhaps more people would check it if the usability was improved...
- bentcorner 3y ago
  Usability would be improved if there was a one-click way to verify it. Off the top of my head a QR code you could scan so it'd be enough "out of band" that the odds of a compromise affecting both paths would be unlikely. Ideally someone smarter than me would come up with a way that it wouldn't require TOFU at all.
  - m-p-3 3y ago
    A well-known URI that provides SSH keys would be nice.
    - dambi0 3y ago
      How would you encrypt / authenticate the URI? Assuming TLS adding this dependency to configure SSH may not be that desirable in some cases. The original SSH threat model allowed for MITM, so you’d have to also allow for that happen to the SSH key distribution URI. Context is key, it’s an interesting trade-off
  - dinosaurdynasty 3y ago
    Or you know SSHFP records + DNSSEC to verify hostkeys. Which is something OpenSSH already supports.
- tedunangst 3y ago
  Turn on ascii art.
- susam 3y ago
  > SSH could really use a better default key fingerprint. It's 42 characters of base64. You can get an ASCII graphic for the host key with the VisualHostKey option. For example: ssh -o VisualHostKey=yes tty.sdf.org Alternatively, you can set this option permanently in the SSH client configuration like this: echo VisualHostKey yes >> ~/.ssh/config ssh tty.sdf.org Here is an example output: $ ssh -o VisualHostKey=yes tty.sdf.org Host key fingerprint is SHA256:ZjwbO7AU8rHJExYrmZS2LqGZ7WfdoELfMrF54W92PYA +--[ED25519 256]--+ | ... | | .oo o | | .=.* | | . .* B | | = o O S. | |+ + o.o*E=. | | o o O.++ o | | o X = +.. o | | + + +.. . | +----[SHA256]-----+ user@tty.sdf.org's password: This graphic looks like a pigeon to me. Similarly, the one for github.com looks like a cat to me. For each server I connect to via SSH I know what the ASCII art for the host key looks like. This familiarity aids in performing TOFU in a hassle-free manner.
neilv 3y ago
The last time I was jump-starting an internal wiki, for a place that had no docs... when I was documenting how to set up a workstation for checking out and running the code from Git SSH... it turned out that no one seemed to document or know what the fingerprint should be. Not even any documentation I could find from the major Git provider. I can understand not expecting every user to do a one-time 10-second diligence process. But the "enterprise" provider seemingly not expecting any of its customers' IT people to even do a one-time 10-second check when integrating mission-critical infrastructure... is a concern.
MenhirMike 3y ago
If you don't check and just allow it, does that count as OpenTOFU?
Am4TIfIsER0ppos 3y ago
Only once. ssh needs to make the fingerprint shown in the terminal match what is stored in known_hosts. If I already have public keys why can't I match them against the new connection? I could preload the file but why not copy it from my desktop's file into the terminal.
TheRealPomax 3y ago
As a font person: this is not the use of tofu I was familiar with.
jheriko 3y ago
here is a better one, just don't use the disaster unusable ssh garbage. no problem ever.
- fragmede 3y ago
  Why is it garbage? do you have recommendations on what to use instead?
SyberShock 3y ago
> "I wonder if SSH should really be used at all for git. While TLS certificates are relatively short lived these days thanks to Let’s Encrypt, SSH host keys live a very long time. If GitHub’s SSH host keys were stolen, how long would an attacker get away with intercepting communications?" The solution is to use SSH Certificate Authentication as opposed to SSH Key Authentication. The Certificate has a forced expiry and is verified by a CA. Then interception is rendered a moot problem since keys are verified with the CA before use.

107 points by cedws 3y ago

I was using a system at work, and so eventually I needed to ssh into it, and I was confronted with the usual TOFU prompt. So I contacted Support, to ask how I could ascertain the Host Keys in use and check their fingerprints. And they replied, "Mmm, we don't do that."

tedunangst 3y ago

Nobody ever mentions that TOFU requires the attacker to commit to the attack before they know whether you know the fingerprint. Somebody could set up on that public wifi and intercept ssh, but they'd be detected as soon as anybody with a previously known key tries to connect. They can't wait for you to connect to an unknown server, because they don't know what you don't know. If you're not sure about the network, you can connect to a known host first to check for funny business. Or turn on the option for ascii art keys.

c7DJTLrn 3y ago
Good point. If we're being imaginative, there's various side channels an attacker could use to carefully target their attack, such as seeing a new MAC or DHCP lease on the local network.
- deleted 3y ago
  [deleted]
- xg15 3y ago
  True. Also depends on what the target is. I.e., if the attacker wants to impersonate machines that are only avaliable on the local network, then it's probably reasonable to conclude that a machine which was never seen on that LAN before also hasn't yet connected to the impersonated host. However, that isn't true for "public" SSH endpoints, such as github, gitlab etc - e.g. if the attacker is impersonating the wifi if the nearest Starbucks to snoop on all the hipster solo-devs there, he'll probably be unable to impersonate github, because even people who use that wifi for the first time have probably connected to github before, on a different network.
SoftTalker 3y ago
> they'd be detected as soon as anybody with a previously known key tries to connect Would they? Or would the person just delete the old key from their ~/.ssh/known_hosts and accept the new one?
- tedunangst 3y ago
  If you're deleting keys from known_hosts, you're not trusting on first use.
  - mjochim 3y ago
    That is correct. And still, many, many devs and admins would do exactly that. Which is why in many, many cases, Tofu really isn't Tofu but "trust whatever." When designing or evaluating security, one should not ignore that this is a part of reality.
    - arp242 3y ago
      That's a bit of a different issue from the "Let’s say you’re on a fresh machine" that this article is about. I agree that "automatic trust on first use" is "good enough" for most cases and people (especially with sshfp records), and to be honest I think the warning you get once that fails is strong enough: % git clone git@github.com:madmurphy/libconfini.git Cloning into 'libconfini'... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU. Please contact your system administrator. Add correct host key in /home/martin/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /home/martin/.ssh/known_hosts:118 Host key for github.com has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. It's a strong warning, with a manual fix that's more than "just press ok" (probably intentionally), and if you choose to ignore that then that's your problem. I'm not really sure what could be done better? A centralized https-like system comes with its own downsides. (The only complaint is that last "Please make sure you have the correct access rights and the repository exists" line, which is from git and not ssh, and a tad confusing; maybe it's possible for git to do better here?)

m_x 3y ago

Aren't SSH host certificates the answer? Why are they not more widely used?

fragmede 3y ago
Especially since they're distributable via DNS. If your server has an entry in DNS, you shouldn't need to confirm host key.
- pritambaral 3y ago
  If the MITM is in the client's LAN, DNS can be spoofed too. A simpler solution would be to just use SSH certs for host keys.
c7DJTLrn 3y ago
According to the guy who invented them: "I'm a shitty shitty publicist" (Damien Miller)
surfer7837 3y ago
Wouldn't that require a responsible body (ie a CA) to verify who they say are. Not sure that's needed for SSH
- cpach 3y ago
  Yes, but it can be an internal CA. There are even FOSS products for that: https://goteleport.com/ https://goteleport.com/
tetha 3y ago
We've switched to them based on Hashicorp Vault and I think they are an improvement in security and convenience. Before SSH host certificates, using tooling like ansible was seriously annoying, especially to the monthly ansible users, compared to the daily ansible users. If you tried using ansible once a month without our host key certificates, you'd have ansible barf about a dozen unknown host keys due to them being rebuilt and other things. Then you fix that, then they do that one ansible run, then it lies around for a month and back you go to square one. I'm not sure HOW you'd stage an actual attack there, but people got used to just accept SSH host keys whatsoever, so the vulnerability was there. Now we have host key signing based on Vault. This is a huge improvement in my book. Base infra guys know when they've reset a VM to a base image (which resets SSH host keys), so they know when to expect a TOFU request from their SSH upon first connect to the just rebuilt system, so they accept those. Afterwards, one of the first things the config management does is to sign those host keys and then the accepted host keys are usually deleted again. There probably is a window of vulnerability in there, but that's getting pretty hard to attack and I'm sure there are easier ones in the overall infrastructure. The main problem at this however is that you need some safe and secure place for the CA, and ideally automation to sign those host keys. And it needs to be enough of a problem to do all of that.

NovemberWhiskey 3y ago

Ideally, the DNS SSHFP record would help solve this problem from Internet-exposed services, in combination with DNSSEC of course.

Apreche 3y ago

I don't check. That's because I'm one of those weirdos who uses certificates, so the prompt never appears.

cesarb 3y ago

It's implied in the examples on this post, but when using public key authentication, an attacker with a stolen SSH host key can pretend to be the server to the client, but cannot pretend to be the client to the server. Which means that the attacker has to pretend to be the server, without being able to see how the server would answer. For instance, if the ssh connection was a "git pull", the attacker has to know enough about the repository state (and about how much of that state the git client has already seen) to avoid complaints from git; if the ssh connection is a shell connection, the attacker has to know enough about the target machine to make the command outputs seem realistic enough for that particular machine; and so on. And as for "SSH has no forward secrecy", I thought SSH used Diffie-Hellman or similar to derive the session key? The SSH host key (and the client's public key) is for authentication only, stealing them does not reveal anything about past sessions.

mindslight 3y ago
> [the attacker] cannot pretend to be the client to the server Isn't the most common way of identifying ssh clients still just public key authorization? Meaning the attacker can MITM the real client connecting to the server, satisfy the real client's request, and then likely carry on doing whatever it likes. edit: actually no - I can see where my intuition is wrong (out of date from the time of widespread password auth). If the ssh server only accepts input that's been effectively signed by the authentication key, then there's no way for the MITM to send its own requests. Yet another reason to not use password auth.
progmetaldev 3y ago
SSH does have forward secrecy. The OP stated in a previous comment that they have fixed this in the article.

dheera 3y ago

No, often I don't check, because my managers rush me to produce results on tight timelines, so I don't have time to "do things right". Slack message at 3pm asking for updates to be in a doc by 6pm. What do you expect? I guess the problem is less serious on a corporate VPN, but still, the principle applies everywhere. If you want security, give your employees time to do things right. Usually that means at least double what you think it will take.

shadowgovt 3y ago
It seriously shoots any Zero Trust claims a corporation makes in the foot if people are "yes"ing through that dialog. Of course, to really have Zero Trust you have to make it impractical for users to "yes" through the dialog.
- dinosaurdynasty 3y ago
  For SSH: setup an SSH CA and then disable asking in the client config.

SoftTalker 3y ago

When I set up a system for a user, I pre-populate /etc/ssh/ssh_known_hosts for all our internal systems. But I've certainly been guilty of just hitting "yes" on first connection to a machine I'm unfamiliar with.

dmurray 3y ago

Trust On First Use is exactly what I do. If I have to connect to a new machine, I identify it by its hostname, and send it my credentials (whatever credentials the owner just sent me in plaintext). If I get the warning and I'm pretty sure I connected to this machine before, I ask whoever owns the machine "hey did you do something weird with this machine? Like reimage it or rotate SSL certs or change the DNS entry so it's actually a different machine?" 99% of the time I'm not connecting to a machine for the first time, so (if I may be allowed some bad maths) this reduces my attack surface by a factor of 100. (If the owner didn't send me plaintext credentials but instead added my SSH key, I don't expose anything by using that, but I immediately follow up by doing other sensitive things like scp'ing code or credentials there).

shadowgovt 3y ago
"hey did you do something weird with this machine? Like reimage it or rotate SSL certs or change the DNS entry so it's actually a different machine?" "I dunno."
- Muromec 3y ago
  "Maybe the other department reprovisioned the bastion about 2 months ago, but maybe it was another fiscal year"

morelisp 3y ago

I tried to check but 99% of our clients rotate host keys whenever they restart the container or the LB sends you to a different node or whatever. And if you email them to verify they take 3 weeks to respond, which is 20 daily reports they now don't get and so don't want to pay for, so... nope.

3pt14159 3y ago

I had this phase where I tried to do everything the right way. I used to check whenever I thought I could, including to my own servers. And you know what? 100% of the time the check passed. After a while I sat back and thought to myself "is this really worth the cost in time?" And I concluded that no it wasn't. Plenty of people run scanners looking for bad stuff online, ranging from poisoned SSH to supply chain attacks on OSS. These are the people that find the baddies and the baddies eventually get targeted. Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint. It would be better if I scanned the web app that I'm working on for, say, SQL injection bugs or misapplied CORS headers or whatever else. Ultimately security checks are an economic activity. The likelihood you are targeted via a given threat, the payoff to the attacker if you are targeted, the downside to the attacker if they are caught, etc. I think being realistic is prudent.

Muromec 3y ago
>Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint ... and when you don't you drive up the probability of enabling the 4-stage attack that leaks private keys to Chines govt. Of course the probability is only 0.01% and it's worth to spend time on something better. And when you do check and other 1000 devs in the organization don't, what's the point, right? This is how this stuff happens and we all read a nice post morten here somewhat 2 years later.
taberiand 3y ago
I think that's the crux of the problem: we should do this check, and perhaps the first few times we do - but it's a chore every time and infrequent enough that the process has to be rediscovered, with the associated uncertainty that entails (sure I checked - but did I do it properly?) If it's so important to validate then it shouldn't be left up to a human to choose, it should be mandatory - but the process needs to be straightforward. Why isn't there a built in tool that we can trust to check for us when these alerts happen? Or at least streamline the process and provide certainty
- 3pt14159 3y ago

seabass-labrax 3y ago

MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public. One of the security principles I live by is not to upload anything to the Internet that I wouldn't mind being shared completely in public. The only exceptions I make are video conferencing and email. Jitsi provides great end-to-end encryption for the former (although the 8x8 server's recent prohibition of anonymous use is disappointing).

ghusto 3y ago
Same. I presume every company I have an account on will be compromised at some point, and all my account data will be out in the wild. This presumption hasn't disappointed so far ;)
cesarb 3y ago
> MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public. This is not just about secrecy, but also about integrity. Someone who MITMs your connection could not only look at the code you published, but also modify it. Then later someone who was trusting you (or possibly even yourself on a new machine) will end up downloading and running the modified code.
- seabass-labrax 3y ago
  In this particular case, I'm not sure it would be possible to modify the code in transit, because although you could trick the user into submitting the code into Fake GitHub, you wouldn't be able to authenticate as the user to submit your modified copy to Real GitHub. More importantly, in my opinion, the end-user has no way of verifying the integrity of the code if authentication was the only security measure. By signing releases or the commits themselves (which can be done with both PGP and SSH keys currently) the end-user can verify the integrity themselves.

tjk_ 3y ago

Bitbucket dev here. Firstly, thanks for the post, this topic is too often skimmed and as someone who worked directly on the host key rotation here, it was shocking to find how many devs had zero idea what "that prompt from git on a new computer" was. Secondly, I just wanted to say that there is an interesting side-effect to everything covered in that blog that effectively leads to a gap in most people's mental models (or maybe just mine). Because of the way these systems were defined/the way people interact with them, there are several conundrums when a large centralized host needs to deprecate an old key. If a key were suspected of being compromised, it seems obvious that you could just stop serving the old key and tell all of your users to be prepared to re-TOFU. However, since _most_ devs blindly type "yes" when interacting with a large git host, this effectively primes the pumps for any bad actor that have MITM control but have not actually stolen the keys. This gives them a relatively long (if not infinitely long) window of time where a user will not be surprised to see "the prompt" and blindly accept trust of a host that could be controlled by the bad actor. If a key was completely successfully compromised by a bad actor, and said bad actor had MITM control of the victim network, then requests with the _old_ key would never actually reach the correct host and just quietly continue working (assuming the bad actor was savy enough to setup a remote system to properly behave as a git host without a prior copy of the target repository). Damned if you do, damned if you don't, so always carefully TOFU. Also,not sure why the author linked off to a Bitbucket Cloud blog post for the SSH keys, they are documented here[1] along with our recommended best practices WRT TOFU. 1 - https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/ https://support.atlassian.com/bitbucket-cloud/docs/configure...

c7DJTLrn 3y ago
Thanks. I linked to the blog post because it was one of the first results in the search engine. How are you guys tackling host key rotation? Do you do it periodically, or on compromise? How do you protect such an important set of keys?
- MaxBarraclough 3y ago
  I don't work for them, but: BitBucket changed their SSH keys earlier this year due to possible compromise of their secret keys. [0] I don't think they change them periodically. [0] https://bitbucket.org/blog/ssh-host-key-changes https://bitbucket.org/blog/ssh-host-key-changes edit I see eichin also linked to the same page

MarkSweep 3y ago

Maybe there should be an “expected key” argument to “git clone”. GitHub could include it in the copy-pasted clone command. Problem solved.

thealistra 3y ago

Isn’t this mechanism by default to check if the prompt shows on a used machine to detect cert rotation? And then you can ask the admin. Seems like too much work for checking the fingerprints if not in an obvious security first context.

pxx 3y ago

> Not to mention that SSH has no forward secrecy. What? This is not right, not even for the obsolete v1 ssh: https://utcc.utoronto.ca/~cks/space/blog/tech/SshForwardSecrecy https://utcc.utoronto.ca/~cks/space/blog/tech/SshForwardSecr...

c7DJTLrn 3y ago
Fixed, thanks.

foobiekr 3y ago

In the real world, "trust on first use" is literally true - it's never "check on first use." SSHFP DNS records would help a lot, or some other scheme. For all of the positive image ssh has, mostly because telnet and ftp were so ridiculously terrible, ssh-as-used is pretty mediocre. And the reality is: "No." https://www.usenix.net/system/files/login/articles/105484-Gutmann.pdf https://www.usenix.net/system/files/login/articles/105484-Gu...

MaxBarraclough 3y ago
> it's never "check on first use." Personally I always manually check on first use, but I agree this puts me in a small minority. I said something similar in a comment last year. [0] It's unfortunate that the term TOFU is ambiguous. In practice it's used to refer to check on first use and blindly assume it's ok on first use. [0] https://news.ycombinator.com/item?id=30366424 https://news.ycombinator.com/item?id=30366424

myself248 3y ago

Yeah I've done it. Not for Git but for SSH. I'll call the admin out-of-band and read him part of the fingerprint, and ask him to read another part back. Only done it a few times when I'm setting up a new machine and don't have an already-authenticated one to check against. If I've got suspicions about the network connection I'm on, I'll first ask him some questions about our adventures years ago, that nobody else would likely know about. And one time when the host key changed, and I wasn't expecting it, he said my call was how he knew he replaced the right box. Tongue-in-cheek, but only barely. The only time I immediately dismiss the key-changed message is when I'm cycling yet another raspberry pi image into the same IP as an old one or whatever, and I'm on the local network so I know exactly what's going on.

ghusto 3y ago

I think the attack in all cases would require pointing DNS of gitprovider.net to their own server. This means they control DNS on that network, which means they're running it. What am I doing on a network I don't trust with a machine I've never used? The example given is that I've started a new job. Well okay, their machine, their network, and I guess their corporate gitprovider.net account they're getting me to use, so to heck with it. If we're talking about using _my_ gitprovider.net account however, that's a different story. I simply wouldn't use my own account on a laptop that's not mine.

c7DJTLrn 3y ago
On the local network, there are ways to MITM without touching DNS, like ARP spoofing.
toxik 3y ago
Another good question is, why not post the fingerprint in a TXT record?
- frutiger 3y ago
  They are more likely to be found in SSHFP records.

upofadown 3y ago

SSH could really use a better default key fingerprint. It's 42 characters of base64. So a hodgepodge of upper and lower case Latin characters with some numbers and special characters mixed in. Not even any spaces to give it some structure. Basically just the output of of a 256 bit hash dumped directly to the terminal. Perhaps more people would check it if the usability was improved...

bentcorner 3y ago
Usability would be improved if there was a one-click way to verify it. Off the top of my head a QR code you could scan so it'd be enough "out of band" that the odds of a compromise affecting both paths would be unlikely. Ideally someone smarter than me would come up with a way that it wouldn't require TOFU at all.
- m-p-3 3y ago
  A well-known URI that provides SSH keys would be nice.
  - dambi0 3y ago
    How would you encrypt / authenticate the URI? Assuming TLS adding this dependency to configure SSH may not be that desirable in some cases. The original SSH threat model allowed for MITM, so you’d have to also allow for that happen to the SSH key distribution URI. Context is key, it’s an interesting trade-off
- dinosaurdynasty 3y ago
  Or you know SSHFP records + DNSSEC to verify hostkeys. Which is something OpenSSH already supports.
tedunangst 3y ago
Turn on ascii art.
susam 3y ago
> SSH could really use a better default key fingerprint. It's 42 characters of base64. You can get an ASCII graphic for the host key with the VisualHostKey option. For example: ssh -o VisualHostKey=yes tty.sdf.org Alternatively, you can set this option permanently in the SSH client configuration like this: echo VisualHostKey yes >> ~/.ssh/config ssh tty.sdf.org Here is an example output: $ ssh -o VisualHostKey=yes tty.sdf.org Host key fingerprint is SHA256:ZjwbO7AU8rHJExYrmZS2LqGZ7WfdoELfMrF54W92PYA +--[ED25519 256]--+ | ... | | .oo o | | .=.* | | . .* B | | = o O S. | |+ + o.o*E=. | | o o O.++ o | | o X = +.. o | | + + +.. . | +----[SHA256]-----+ user@tty.sdf.org's password: This graphic looks like a pigeon to me. Similarly, the one for github.com looks like a cat to me. For each server I connect to via SSH I know what the ASCII art for the host key looks like. This familiarity aids in performing TOFU in a hassle-free manner.

neilv 3y ago

The last time I was jump-starting an internal wiki, for a place that had no docs... when I was documenting how to set up a workstation for checking out and running the code from Git SSH... it turned out that no one seemed to document or know what the fingerprint should be. Not even any documentation I could find from the major Git provider. I can understand not expecting every user to do a one-time 10-second diligence process. But the "enterprise" provider seemingly not expecting any of its customers' IT people to even do a one-time 10-second check when integrating mission-critical infrastructure... is a concern.

MenhirMike 3y ago

If you don't check and just allow it, does that count as OpenTOFU?

Am4TIfIsER0ppos 3y ago

Only once. ssh needs to make the fingerprint shown in the terminal match what is stored in known_hosts. If I already have public keys why can't I match them against the new connection? I could preload the file but why not copy it from my desktop's file into the terminal.

TheRealPomax 3y ago

As a font person: this is not the use of tofu I was familiar with.

jheriko 3y ago

here is a better one, just don't use the disaster unusable ssh garbage. no problem ever.

fragmede 3y ago
Why is it garbage? do you have recommendations on what to use instead?

SyberShock 3y ago

> "I wonder if SSH should really be used at all for git. While TLS certificates are relatively short lived these days thanks to Let’s Encrypt, SSH host keys live a very long time. If GitHub’s SSH host keys were stolen, how long would an attacker get away with intercepting communications?" The solution is to use SSH Certificate Authentication as opposed to SSH Key Authentication. The Certificate has a forced expiry and is verified by a CA. Then interception is rendered a moot problem since keys are verified with the CA before use.