5 ms·

So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection s

So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection scripts locally. Yikes.

tialaramex 2y ago
One completely awful thing some scanners might choose to do is if you're offering RSA auth (which most SSH servers are and indeed the SecSH RFC says this is Mandatory To Implement) then you're "potentially vulnerable" which would encourage people to do password auth instead. Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.
- danparsonson 2y ago
  Have to admit I've never understood why password auth is considered so much worse than using a cert - surely a decent password (long, random, etc) is for all practical purposes unguessable, and so you're either using a private RSA key that no-one can guess, or a password that no-one can guess, and then what's the difference? With the added inconvenience of having to pass around a certificate if you want to login to the same account from from multiple sources.
  - denysvitali 2y ago
    The difference, IMHO, is that it's easier to pick an easy guessable password than it is to create an insecure RSA key (on purpose).
  - dboreham 2y ago
    It depends what happens to the password. Typically it's sent as a bearer credential. But there are auth schemes (not widely used these days) where the password isn't sent over the wire.
    - solardev 2y ago
      Isn't it pretty standard practice to salt and hash the password client-side before sending it over the wire?
      - cayde 2y ago
        No, usually it's sent in plain text to the server, however encapsulated.
        solardev 2y ago
        Wow, really? Ten years ago, it was drilled into me to never send a password like that, especially since the server shouldn't have the plain version anyway (so no reason for the client to send it). https://owasp.org/www-community/OWASP_Application_Security_FAQ https://owasp.org/www-community/OWASP_Application_Security_F... says "Salted hash for transmitting passwords is a good technique. This ensures that the password can not be stolen even if the SSL key is broken." I didn't want to believe you, but man, I just checked a few websites in the network inspector... and it seems like GMail, Hackernews, Wordpress, Wix, and Live.com all just sent it in plaintext with only SSL encryption :( That's a bit disappointing. But TIL. Thanks for letting me know!
        jbverschoor 2y ago
        That's exactly the case with HTTP credentials and most authentication frameworks...
        cayde 2y ago
        If you want to hop into a rabbit hole, try taking look in Steam's login send the user and pass)) If TLS break then all is untrusted anyway! If you read hash as MITM you can replay it as pass equivalent and log in with hash, do not need knowledge of the original pass. You can just inject the script to exfilatrate original pass before hashing. CSP is broken, since you can edit header to give your own script a inline nonce. I think everything is reliant on TLS in end. I think 10yr ago before TLS was 99%+ standard on all sites many people would come up with schemes, forums would md5 pass client side and send md5, all sorts were common. But now trust is in TLS.
  - teraflop 2y ago
    One of the biggest differences is that if you're using password auth, and you are tricked into connecting to a malicious server, that server now has your plaintext password and can impersonate you to other servers. If you use a different strong random password for every single server, this attack isn't a problem, but that adds a lot of management hassle compared to using a single private key. (It's also made more difficult by host key checking, but let's be honest, most of us don't diligently check the fingerprints every single time we get a mismatch warning.) In contrast, if you use an SSH key, then a compromised server never actually gets a copy of your private key unless you explicitly copy it over. (If you're have SSH agent forwarding turned on, then during the compromised connection, the server can run a "confused deputy" attack to authenticate other connections using your agent's identity. But it loses that ability when you disconnect.)
    - someplaceguy 2y ago
      > if you're using password auth, and you are tricked into connecting to a malicious server, that server now has your plaintext password and can impersonate you to other servers. Why would the password be sent in plaintext instead of, say, sending a hash of the password calculated with a salt that is unique per SSH server? Or something even more cryptographically sound. In fact, passwords in /etc/shadow already do have random salts, so why aren't these sent over to the SSH client so it can send a proper hash instead of the plaintext password?
      - danparsonson 2y ago
        If the hash permits a login then having a hash is essentially equivalent to having a password. The malicious user wouldn't be able to use it to sudo but they could deploy some other privilege escalation once logged in.
        deleted 2y ago
        [deleted]
      - djao 2y ago
        It's a little bit more complicated than just sending a hash of the password, but there are ways to authenticate using hashed passwords without sending the password over the wire, for example https://en.wikipedia.org/wiki/Digest_access_authentication https://en.wikipedia.org/wiki/Digest_access_authentication or https://en.wikipedia.org/wiki/Password-authenticated_key_agreement https://en.wikipedia.org/wiki/Password-authenticated_key_agr... Even so, these protocols require the server to know your actual password, not just a hash of the password, even though the password itself never traverses the network. So a compromised server can still lead to a compromised credential, and unless you use different passwords for every server, we're back to the same problem.
  - duskwuff 2y ago
    > Have to admit I've never understood why password auth is considered so much worse than using a cert Password auth involves sending your credentials to the server. They're encrypted, but not irreversibly; the server needs your plaintext username and password to validate them, and it can, in principle, record them to be reused elsewhere. Public key and certificate-based authentication only pass your username and a signature to the server. Even if you don't trust the server you're logging into, it can't do anything to compromise other servers that key has access to.
  - imrehg 2y ago
    On that last point, I wouldn't pass around the certificate to log in from multiple sources, rather each source would have its own certificate. That is easy & cheap to do (especially with ed25519 certs).
    - danparsonson 2y ago
      Ah right, that's useful, thanks. Presumably if you need to login from an untrusted source (e.g. in an emergency), then you're out of luck in that case? Do you maybe keep an emergency access cert stashed somewhere?
      - imrehg 2y ago
        That's a very good question. Likely depends on the circumstances. I don't quite know any ways of using untrusted sources safely. Maybe something where you can use temporary credentials (say 2FA), or the the likes of using AWS's EC2 Instance Connect, but there's always a problem of _something_ has to be on an untrusted location, I guess? Having some emergency access certs in a password manager might be a good backup (and rotating it after using it on an untrusted source?). The best way is, however, removing the need in emergencies to access a machine (e.g. more of the "cattle vs pets" way of thinking). But that's hard for sure.
        danparsonson 2y ago
        > ...rotating it after using it on an untrusted source?... > ...the "cattle vs pets" way of thinking... Good points both... To the former, of course you're right that once used, an emergency cert should be replaced, which could be onerous either from the point of view of having double the number of certs to manage (rather than one master key), or else having to rotate the master key on all servers. To the latter, I'm definitely thinking about pets, so I hadn't considered just throwing away the VM and starting again; that neatly sidesteps the issue. Thanks!
  - computerfriend 2y ago
    I sometimes ask this as an interview question. Hardly anybody knows the answer.
    - sureglymop 2y ago
      So don't you want to enlighten us with the answer?
      - computerfriend 2y ago
        The enlightenment has happened in the other comments, but the summary is: * this enforces high entropy, * the password is transmitted to the server, the private key is not.
  - Too 2y ago
    A lot of it has to do with centralizing administration. If you have more than one server and more than one user, certificates reduce a NxM problem into N+M instead. Certificates can be revoked, they can have short expiry dates and due to centralized administration, renewing them is not terribly inconvenient. On top of that they are a lot more difficult to read over the shoulder, to some degree that can be considered the second factor in a MFA scheme. Same reasons why passkeys are preferred over passwords lately. Not as secure as a HW-key, still miles better than “hunter2”.
  - bostik 2y ago
    > surely a decent password (long, random, etc) is for all practical purposes unguessable Sadly that is not how normies use passwords. WE know what passwords managers are for. Vast majority of people outside our confined sphere do not. In short: password rotation policies make passwords overall less secure, because in order to remember what the new password is, people apply patterns. Patterns are guessable. Patterns get applied to future password as well. This has been known to the infosec people since 1990's because they had to understand how people actually behave. It took a research paper[0], published in 2010, to finally provide sufficient data for that fact to become undeniable. It still took another 6-7 years until the information percolated through to the relevant regulatory bodies and for them to update their previous guidance. These days both NIST and NCSC tell in very clear terms to not require password rotation. 0: https://www.researchgate.net/publication/221517955_The_true_cost_of_unusable_password_policies https://www.researchgate.net/publication/221517955_The_true_...
CodesInChaos 2y ago
It might be possible to use timing information to detect this, since the signature verification code appears to only run if the client public key matches a specific fingerprint. The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan. (https://bench.cr.yp.to/results-sign.html https://bench.cr.yp.to/results-sign.html lists Ed448 verification at around 400k cycles, which at 4GHz amounts to 100us)
- pstrateman 2y ago
  However only probabilistic detection is possible that way and really 100us variance over the internet would require many many detection attempts to discern.
- Thorrez 2y ago
  According to[1], the backdoor introduces a much larger slowdown, without backdoor: 0m0.299s, with backdoor: 0m0.807s. I'm not sure exactly why the slowdown is so large. [1] https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
  - CodesInChaos 2y ago
    The effect of the slowdown on the total handshake time wouldn't work well for detection, since without a baseline you can't tell if it's slow due to the backdoor, or due to high network latency or a slow/busy CPU. The relative timing of different steps in the TCP and SSH handshakes on the other hand should work, since the backdoor should only affect one/some steps (RSA verification), while others remain unaffected (e.g. the TCP handshake).
Thorrez 2y ago
The tweet says "unreplayable". Can someone explain how it's not replayable? Does the backdoored sshd issue some challenge that the attacker is required to sign?
- candiodari 2y ago
  What it does is this: RSA_public_decrypt verifies a signature on the client's (I think) host key by a fixed Ed448 key, and then if it verifies, passes the payload to system(). If you send a request to SSH to associate (agree on a key for private communications), signed by a specific private key, it will send the rest of the request to the "system" call in libc, which will execute it in bash. So this is quite literally a "shellcode". Except, you know, it's on your system.
  - Thorrez 2y ago
    That sounds repayable though. If I did a tcpdump of the attacker attacking my system, I could replay that attack against someone other system. For it to not be replayable, there needs to be some challenge issued by the backdoored sshd. Of course since the backdoor was never widely deployed and is now public, I think it's unlikely the attacker will attempt to use it. So whether it's replayable doesn't have a practical impact now. I'm only asking about replayability because I'm curious how it's unreplayable.

So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection scripts locally. Yikes.

tialaramex 2y ago

One completely awful thing some scanners might choose to do is if you're offering RSA auth (which most SSH servers are and indeed the SecSH RFC says this is Mandatory To Implement) then you're "potentially vulnerable" which would encourage people to do password auth instead. Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.

danparsonson 2y ago
Have to admit I've never understood why password auth is considered so much worse than using a cert - surely a decent password (long, random, etc) is for all practical purposes unguessable, and so you're either using a private RSA key that no-one can guess, or a password that no-one can guess, and then what's the difference? With the added inconvenience of having to pass around a certificate if you want to login to the same account from from multiple sources.
- denysvitali 2y ago
  The difference, IMHO, is that it's easier to pick an easy guessable password than it is to create an insecure RSA key (on purpose).
- dboreham 2y ago
  It depends what happens to the password. Typically it's sent as a bearer credential. But there are auth schemes (not widely used these days) where the password isn't sent over the wire.
  - solardev 2y ago
    Isn't it pretty standard practice to salt and hash the password client-side before sending it over the wire?
    - cayde 2y ago
      No, usually it's sent in plain text to the server, however encapsulated.
      - solardev 2y ago
        Wow, really? Ten years ago, it was drilled into me to never send a password like that, especially since the server shouldn't have the plain version anyway (so no reason for the client to send it). https://owasp.org/www-community/OWASP_Application_Security_FAQ https://owasp.org/www-community/OWASP_Application_Security_F... says "Salted hash for transmitting passwords is a good technique. This ensures that the password can not be stolen even if the SSL key is broken." I didn't want to believe you, but man, I just checked a few websites in the network inspector... and it seems like GMail, Hackernews, Wordpress, Wix, and Live.com all just sent it in plaintext with only SSL encryption :( That's a bit disappointing. But TIL. Thanks for letting me know!
        jbverschoor 2y ago
        That's exactly the case with HTTP credentials and most authentication frameworks...
        cayde 2y ago
        If you want to hop into a rabbit hole, try taking look in Steam's login send the user and pass)) If TLS break then all is untrusted anyway! If you read hash as MITM you can replay it as pass equivalent and log in with hash, do not need knowledge of the original pass. You can just inject the script to exfilatrate original pass before hashing. CSP is broken, since you can edit header to give your own script a inline nonce. I think everything is reliant on TLS in end. I think 10yr ago before TLS was 99%+ standard on all sites many people would come up with schemes, forums would md5 pass client side and send md5, all sorts were common. But now trust is in TLS.
- teraflop 2y ago
  One of the biggest differences is that if you're using password auth, and you are tricked into connecting to a malicious server, that server now has your plaintext password and can impersonate you to other servers. If you use a different strong random password for every single server, this attack isn't a problem, but that adds a lot of management hassle compared to using a single private key. (It's also made more difficult by host key checking, but let's be honest, most of us don't diligently check the fingerprints every single time we get a mismatch warning.) In contrast, if you use an SSH key, then a compromised server never actually gets a copy of your private key unless you explicitly copy it over. (If you're have SSH agent forwarding turned on, then during the compromised connection, the server can run a "confused deputy" attack to authenticate other connections using your agent's identity. But it loses that ability when you disconnect.)
  - someplaceguy 2y ago
    > if you're using password auth, and you are tricked into connecting to a malicious server, that server now has your plaintext password and can impersonate you to other servers. Why would the password be sent in plaintext instead of, say, sending a hash of the password calculated with a salt that is unique per SSH server? Or something even more cryptographically sound. In fact, passwords in /etc/shadow already do have random salts, so why aren't these sent over to the SSH client so it can send a proper hash instead of the plaintext password?
    - danparsonson 2y ago
      If the hash permits a login then having a hash is essentially equivalent to having a password. The malicious user wouldn't be able to use it to sudo but they could deploy some other privilege escalation once logged in.
      - deleted 2y ago
        [deleted]
    - djao 2y ago
      It's a little bit more complicated than just sending a hash of the password, but there are ways to authenticate using hashed passwords without sending the password over the wire, for example https://en.wikipedia.org/wiki/Digest_access_authentication https://en.wikipedia.org/wiki/Digest_access_authentication or https://en.wikipedia.org/wiki/Password-authenticated_key_agreement https://en.wikipedia.org/wiki/Password-authenticated_key_agr... Even so, these protocols require the server to know your actual password, not just a hash of the password, even though the password itself never traverses the network. So a compromised server can still lead to a compromised credential, and unless you use different passwords for every server, we're back to the same problem.
- duskwuff 2y ago
  > Have to admit I've never understood why password auth is considered so much worse than using a cert Password auth involves sending your credentials to the server. They're encrypted, but not irreversibly; the server needs your plaintext username and password to validate them, and it can, in principle, record them to be reused elsewhere. Public key and certificate-based authentication only pass your username and a signature to the server. Even if you don't trust the server you're logging into, it can't do anything to compromise other servers that key has access to.
- imrehg 2y ago
  On that last point, I wouldn't pass around the certificate to log in from multiple sources, rather each source would have its own certificate. That is easy & cheap to do (especially with ed25519 certs).
  - danparsonson 2y ago
    Ah right, that's useful, thanks. Presumably if you need to login from an untrusted source (e.g. in an emergency), then you're out of luck in that case? Do you maybe keep an emergency access cert stashed somewhere?
    - imrehg 2y ago
      That's a very good question. Likely depends on the circumstances. I don't quite know any ways of using untrusted sources safely. Maybe something where you can use temporary credentials (say 2FA), or the the likes of using AWS's EC2 Instance Connect, but there's always a problem of _something_ has to be on an untrusted location, I guess? Having some emergency access certs in a password manager might be a good backup (and rotating it after using it on an untrusted source?). The best way is, however, removing the need in emergencies to access a machine (e.g. more of the "cattle vs pets" way of thinking). But that's hard for sure.
      - danparsonson 2y ago
        > ...rotating it after using it on an untrusted source?... > ...the "cattle vs pets" way of thinking... Good points both... To the former, of course you're right that once used, an emergency cert should be replaced, which could be onerous either from the point of view of having double the number of certs to manage (rather than one master key), or else having to rotate the master key on all servers. To the latter, I'm definitely thinking about pets, so I hadn't considered just throwing away the VM and starting again; that neatly sidesteps the issue. Thanks!
- computerfriend 2y ago
  I sometimes ask this as an interview question. Hardly anybody knows the answer.
  - sureglymop 2y ago
    So don't you want to enlighten us with the answer?
    - computerfriend 2y ago
      The enlightenment has happened in the other comments, but the summary is: * this enforces high entropy, * the password is transmitted to the server, the private key is not.
- Too 2y ago
  A lot of it has to do with centralizing administration. If you have more than one server and more than one user, certificates reduce a NxM problem into N+M instead. Certificates can be revoked, they can have short expiry dates and due to centralized administration, renewing them is not terribly inconvenient. On top of that they are a lot more difficult to read over the shoulder, to some degree that can be considered the second factor in a MFA scheme. Same reasons why passkeys are preferred over passwords lately. Not as secure as a HW-key, still miles better than “hunter2”.
- bostik 2y ago
  > surely a decent password (long, random, etc) is for all practical purposes unguessable Sadly that is not how normies use passwords. WE know what passwords managers are for. Vast majority of people outside our confined sphere do not. In short: password rotation policies make passwords overall less secure, because in order to remember what the new password is, people apply patterns. Patterns are guessable. Patterns get applied to future password as well. This has been known to the infosec people since 1990's because they had to understand how people actually behave. It took a research paper[0], published in 2010, to finally provide sufficient data for that fact to become undeniable. It still took another 6-7 years until the information percolated through to the relevant regulatory bodies and for them to update their previous guidance. These days both NIST and NCSC tell in very clear terms to not require password rotation. 0: https://www.researchgate.net/publication/221517955_The_true_cost_of_unusable_password_policies https://www.researchgate.net/publication/221517955_The_true_...

CodesInChaos 2y ago

It might be possible to use timing information to detect this, since the signature verification code appears to only run if the client public key matches a specific fingerprint. The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan. (https://bench.cr.yp.to/results-sign.html https://bench.cr.yp.to/results-sign.html lists Ed448 verification at around 400k cycles, which at 4GHz amounts to 100us)

pstrateman 2y ago
However only probabilistic detection is possible that way and really 100us variance over the internet would require many many detection attempts to discern.
Thorrez 2y ago
According to[1], the backdoor introduces a much larger slowdown, without backdoor: 0m0.299s, with backdoor: 0m0.807s. I'm not sure exactly why the slowdown is so large. [1] https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
- CodesInChaos 2y ago
  The effect of the slowdown on the total handshake time wouldn't work well for detection, since without a baseline you can't tell if it's slow due to the backdoor, or due to high network latency or a slow/busy CPU. The relative timing of different steps in the TCP and SSH handshakes on the other hand should work, since the backdoor should only affect one/some steps (RSA verification), while others remain unaffected (e.g. the TCP handshake).

Thorrez 2y ago

The tweet says "unreplayable". Can someone explain how it's not replayable? Does the backdoored sshd issue some challenge that the attacker is required to sign?

candiodari 2y ago
What it does is this: RSA_public_decrypt verifies a signature on the client's (I think) host key by a fixed Ed448 key, and then if it verifies, passes the payload to system(). If you send a request to SSH to associate (agree on a key for private communications), signed by a specific private key, it will send the rest of the request to the "system" call in libc, which will execute it in bash. So this is quite literally a "shellcode". Except, you know, it's on your system.
- Thorrez 2y ago
  That sounds repayable though. If I did a tcpdump of the attacker attacking my system, I could replay that attack against someone other system. For it to not be replayable, there needs to be some challenge issued by the backdoored sshd. Of course since the backdoor was never widely deployed and is now public, I think it's unlikely the attacker will attempt to use it. So whether it's replayable doesn't have a practical impact now. I'm only asking about replayability because I'm curious how it's unreplayable.