3 ms·

Does anyone have a good explanation or introduction into the performance testing that was done to find this? And how to get started? Actually measuring performa

Does anyone have a good explanation or introduction into the performance testing that was done to find this? And how to get started? Actually measuring performance always seemed to be a very hard task and I'd like to be able to do similar testing as the person which found this backdoor.

glibg10b 3y ago
https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 > == Observing Impact on openssh server == > > With the backdoored liblzma installed, logins via ssh become a lot slower. > > time ssh nonexistant@...alhost > > before: > nonexistant@...alhost: Permission denied (publickey). > > before: > real 0m0.299s > user 0m0.202s > sys 0m0.006s > > after: > nonexistant@...alhost: Permission denied (publickey). > > real 0m0.807s > user 0m0.202s > sys 0m0.006s
- deleted 3y ago
  [deleted]
intelVISA 3y ago
> measuring performance always seemed to be a very hard task It's not hard: it's either easy, or impossible, depending on the culture at your shop. At the basic level it's trivial - you instrument code and interpret the results against the HW and lower level machine counters. Depending on $lang you have many good libraries for this kicking around, the hard part is working with a corp that values performance (99% do not) so none of the required mindset and surrounding infra will be setup to permit this in any useful capacity.
cesarb 3y ago
From what I understand, it wasn't even the performance testing itself that caught the backdoor. The developer wanted to have the machine as quiescent as possible (so that nothing else running on it would interfere) before starting the performance tests, but sshd was using much more CPU than expected (and this could be observed with simple tools like "top"). My guess is that the usual "backscatter" of password guessing ssh login attempts from all over the Internet normally uses very little CPU time in sshd before being rejected, but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet).
- anarazel 3y ago
  Nearly spot on. I indeed was seeing sshd usage via top. > but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker Absurdly enough, the high cpu usage is well before it even tries to figure that out. Due to avoiding "suspicious" strings in both the binary and memory, it has a more expensive string matching routine. Finding all the symbols it from the in-memory symbol tables ends up slow due to that. That's why even sshd -h (i.e. printing help) is slow when started in the right environment. There's not enough visibility into other things that early during process startup (this happens long before main() is called), so they couldn't check what key is being presented or such. They really "should" have deferred much more of their initialization until after the ed448 check happened. > (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet) Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root).
  - cesarb 3y ago
    > Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Wow, that must have really sucked. It's one thing to have a rented office downtown (a VPS or similar) be backdoored, but to find a burglar broke into your home while it was under construction (or renovation in case it was a distro upgrade) and added a secret passage to it from the outside? What else did the burglar mess with while you weren't looking? > Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root). Most people's threat model is "everything except sshd is risky; openssh is from these cool paranoid people at OpenBSD and, as long as nobody can password guess, it's safe from non-authenticated attackers". That is, often sshd is the only open port (which helps explain why the attacker fixated on it).

glibg10b 3y ago

https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 > == Observing Impact on openssh server == > > With the backdoored liblzma installed, logins via ssh become a lot slower. > > time ssh nonexistant@...alhost > > before: > nonexistant@...alhost: Permission denied (publickey). > > before: > real 0m0.299s > user 0m0.202s > sys 0m0.006s > > after: > nonexistant@...alhost: Permission denied (publickey). > > real 0m0.807s > user 0m0.202s > sys 0m0.006s

deleted 3y ago
[deleted]

intelVISA 3y ago

> measuring performance always seemed to be a very hard task It's not hard: it's either easy, or impossible, depending on the culture at your shop. At the basic level it's trivial - you instrument code and interpret the results against the HW and lower level machine counters. Depending on $lang you have many good libraries for this kicking around, the hard part is working with a corp that values performance (99% do not) so none of the required mindset and surrounding infra will be setup to permit this in any useful capacity.

cesarb 3y ago

From what I understand, it wasn't even the performance testing itself that caught the backdoor. The developer wanted to have the machine as quiescent as possible (so that nothing else running on it would interfere) before starting the performance tests, but sshd was using much more CPU than expected (and this could be observed with simple tools like "top"). My guess is that the usual "backscatter" of password guessing ssh login attempts from all over the Internet normally uses very little CPU time in sshd before being rejected, but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet).

anarazel 3y ago
Nearly spot on. I indeed was seeing sshd usage via top. > but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker Absurdly enough, the high cpu usage is well before it even tries to figure that out. Due to avoiding "suspicious" strings in both the binary and memory, it has a more expensive string matching routine. Finding all the symbols it from the in-memory symbol tables ends up slow due to that. That's why even sshd -h (i.e. printing help) is slow when started in the right environment. There's not enough visibility into other things that early during process startup (this happens long before main() is called), so they couldn't check what key is being presented or such. They really "should" have deferred much more of their initialization until after the ed448 check happened. > (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet) Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root).
- cesarb 3y ago
  > Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Wow, that must have really sucked. It's one thing to have a rented office downtown (a VPS or similar) be backdoored, but to find a burglar broke into your home while it was under construction (or renovation in case it was a distro upgrade) and added a secret passage to it from the outside? What else did the burglar mess with while you weren't looking? > Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root). Most people's threat model is "everything except sshd is risky; openssh is from these cool paranoid people at OpenBSD and, as long as nobody can password guess, it's safe from non-authenticated attackers". That is, often sshd is the only open port (which helps explain why the attacker fixated on it).