4 ms·

OpenSSH Keystroke Obfuscation Bypass

200 points by pabs3 2y ago

thadt 2y ago
Good work. But - this is an implementation bug right? If the underlying stream of packets gives you a distinguisher (in this case, slightly larger packets), then this attack works. So adjusting the chaff and payload packet sizes to some fixed capacity restores obfuscation, if I'm understanding this correctly. And response packets - those also have to be managed to prevent leakage.
- KyleSanderson 2y ago
  The implementation is just wrong from what's been presented. basic jitter (20-100ms), and a dynamic payload size are what's actually missing here. The question now becomes though how interactive should your session be. Timing the connection latency might help to an extent, but this is about mitm and you don't necessarily know where your adversary is (first hop, or towards the end). Batching keystrokes would also help here.
  - tialaramex 2y ago
    Basically there are two "modes" in normal mode OpenSSH is content to idle with no packets moving except any requested keepalives. In "chaff mode" currently they try to send a chaff packet every time they can to disguise your keypresses, but they forgot keystrokes will just get bundled into the existing chaff packet, growing it, so it stands out as special. All they need to do is retain the "chaff mode" but when they have a keystroke ready to be sent they should suppress the chaff that would otherwise go in the same packet. No need for "basic jitter" or "dynamic payload size" that I can see, with this change the packets are indistinguishable in terms of size or (encrypted) content, and they have no more or less jitter than would be normal for the network they're traversing. [Various small edits to clarify]
    - aidenn0 2y ago
      They will also need to increase the minimum size of a non-chaff keystroke to be the size of the largest keystroke that they wish to keep confidential; 3 bytes is a minimum for the basic control characters (e.g. arrow keys which are ESC '[' X where X depends on the arrow direction). I did find it interesting that the return keystroke was of a differing size to other characters; on unix systems it should just send a ^J.
      - ec109685 2y ago
        Can’t they just send large control characters as three separate key presses?
        aidenn0 2y ago
        That will confuse screen/tmux which needs to distinguish between escape codes and a press of the escape key.
SoftTalker 2y ago
I'm not an expert in this area at all but I recall reading that trying to hide a signal in random noise doesn't really work, as you can still find the signal with statistical analysis or looking for distingishing characteristics that are not obscured. That was my first thought when I read about this new feature in OpenSSH, and it seems to have proven correct. Edit: wanted to add that I recognize that the people working on OpenSSH know a lot more about this than I do, and I had assumed they wouldn't bother implementing this if it wasn't a good idea, so willing to accept that "proven correct" may be an overstatement or even flat-out wrong.
- milliams 2y ago
  The advantage that OpenSSH have here is that they don't have to just hide the signal using noise, they can actually change the signal to look like the noise (or vice versa). For example they are making the signal packets some at a regular interval (since the "signal" being discussed is the timings). That alone would not hide the number of keypresses, but adding the chaff should do so. In this case, as said above, it seems like there's an implementation issue with actually doing those obfuscations, allowing the signal to be identified.
- 6502nerdface 2y ago
  > trying to hide a signal in random noise doesn't really work Actually it works perfectly and it's called a one-time pad!
qhwudbebd 2y ago
This was present in versions 9.5 - 9.7 of OpenSSH and fixed in 9.8p1 released at the beginning of July. Philippos (along with others) is credited for reporting it in the release notes: https://www.openssh.com/releasenotes.html#9.8p1
- palisade 2y ago
  Worth noting too, that other linux systems using 8.9, e.g. ubuntu are 8.9p1 and have the patch applied but to an older version so they're also safe. Just in case anyone was worried about that.
  - 0x0 2y ago
    I'd be very surprised if ubuntu has backported the keystroke obfuscation feature (which was introduced in 9.5) to 8.9. When the feature is missing, you're not safe from keystroke interception at all, which is what this bug is all about, so any 8.9 version would be actually considered "unsafe" until the whole feature is backported, which seems unlikely to happen?
keyboardcaper 2y ago
There is a market for keyboards which obscure keypress timings.
gunapologist99 2y ago
Yet another reason to switch to SSH keys instead of passwords wherever possible. Simple and bulletproof, with a minimum of footguns. EDIT: @tedunangst is right, wouldn't have helped. Even so, switch to keys!
- tedunangst 2y ago
  Does nothing here.
  - deleted 2y ago
    [deleted]
hi-v-rocknroll 2y ago
Perhaps instead shooting off individual keypresses with some random chaff packets, it would be wiser to have constant interval, constant low data rate, (say 8 KBps) data rate trickling packets through and let the other side throw away pick out the needles from the haystack of no ops. Good luck finding a side-channel in that.
- pshc 2y ago
  Sounds good until (for example) you find out the SSH client library wasn't queueing up packets ahead of time, but instead building them at send time, and a chaff packet takes less time to build than a real packet... so many ways to get side-channeled.
  - bubblesnort 2y ago
    Crypto requires a good source of random entropy. I suppose if you want more traffic for obscurity, you'll also have to find more random bits. Without enough entropy, you'd sacrifice security for obscurity. Just my ¢2, ymmv, etc
mrguyorama 2y ago
Regardless of what the "cause" of this is (ie bug or bad design), it's obvious nobody even took a look at this after it was coded up and merged in. The larger packets and double server ACKs are very clearly obviously giving the game away. This feature just doesn't even come close to what it supposedly does, was it even shipped as a feature or something that wasn't yet done? Not exactly encouraging, to see a system so integral to security seemingly shipping code without even a minor test of intended function.
- tedunangst 2y ago
  I think it's worth asking how important the feature is in the grand scheme of things. But what about the CIA timing my keystrokes is more of a nerd forum thing than an actual attack vector you read about in intrusion post mortems.
  - talkin 2y ago
    Well, or it is important, and then you add the countermeasures. These countermeasures are quite easy to mess up, so doing the validation (on an ongoing basis!) MUST be part of the deal. Or if you think it’s not important enough to do those assertions in CI, then it might be better to just reject the obfuscation attempts. There’s no middleground: doing the implementation without checks, means you added complexity, you dont know if security improved (or worsened!), and the the release note might come down to a false sense of security.
jonmon6691 2y ago
Most people would consider streaming music to be a negligible amount of bandwidth. Seems to me like SSH in a "high security mode" could just send X Kbps of bi-directional pad at a layer directly above encryption but below application. Then just use that channel for all the normal SSH traffic. You could either treat this as a bandwidth limited channel or do some slow time-constant ramping up and down at the risk of leaking information about file downloads or large command outputs.
- advael 2y ago
  This seems like a reasonable solution to me, and I often stream music through my ssh sessions via port forwards so I may well already be getting the benefit in some places
- mrngm 2y ago
  This and a similar suggestion in another thread may sound nice and easy, "just add a constant stream of noise", but it assumes you can generate enough constant noise and be able to intersperse the noise with valid commands without being able to distinguish these events. The problem is not necessarily that you want to hide (to a network adversary) that you've been typing. It's that you do not want to reveal, through some side-channel, what the exact contents were. On the openssh-unix-dev mailing list, someone recently pointed out[0] that just periodically (without jitter) sending out packets may be problematic due to subtle differences in clock timing. Aside, they also link to a presentation[1] [PDF] that shows influence of temperature on clock skew (especially page 18) and that this gives a possibility for fingerprinting. Then there's the challenge of keeping SSH interactive enough that people do not experience too much input lag while typing. What if the user typed a character, but due to such a timing side-channel preventive measure, that character needs to be sent in the next packet, adding latency to the user experience? Surely it improves security, but it may add too much frustration for regular usage. [0] https://marc.info/?l=openssh-unix-dev&m=169402700622936&w=2 https://marc.info/?l=openssh-unix-dev&m=169402700622936&w=2 [1] https://murdoch.is/talks/ccs06hotornot.pdf https://murdoch.is/talks/ccs06hotornot.pdf ([2006] Hot or Not: Revealing hidden services by their clock skew, see also https://doi.org/10.1145/1180405.1180410 https://doi.org/10.1145/1180405.1180410 and an HN thread from 2014: https://news.ycombinator.com/item?id=7694612 https://news.ycombinator.com/item?id=7694612)
jgeowugejsb 2y ago
[flagged]

200 points by pabs3 2y ago

Good work. But - this is an implementation bug right? If the underlying stream of packets gives you a distinguisher (in this case, slightly larger packets), then this attack works. So adjusting the chaff and payload packet sizes to some fixed capacity restores obfuscation, if I'm understanding this correctly. And response packets - those also have to be managed to prevent leakage.

KyleSanderson 2y ago
The implementation is just wrong from what's been presented. basic jitter (20-100ms), and a dynamic payload size are what's actually missing here. The question now becomes though how interactive should your session be. Timing the connection latency might help to an extent, but this is about mitm and you don't necessarily know where your adversary is (first hop, or towards the end). Batching keystrokes would also help here.
- tialaramex 2y ago
  Basically there are two "modes" in normal mode OpenSSH is content to idle with no packets moving except any requested keepalives. In "chaff mode" currently they try to send a chaff packet every time they can to disguise your keypresses, but they forgot keystrokes will just get bundled into the existing chaff packet, growing it, so it stands out as special. All they need to do is retain the "chaff mode" but when they have a keystroke ready to be sent they should suppress the chaff that would otherwise go in the same packet. No need for "basic jitter" or "dynamic payload size" that I can see, with this change the packets are indistinguishable in terms of size or (encrypted) content, and they have no more or less jitter than would be normal for the network they're traversing. [Various small edits to clarify]
  - aidenn0 2y ago
    They will also need to increase the minimum size of a non-chaff keystroke to be the size of the largest keystroke that they wish to keep confidential; 3 bytes is a minimum for the basic control characters (e.g. arrow keys which are ESC '[' X where X depends on the arrow direction). I did find it interesting that the return keystroke was of a differing size to other characters; on unix systems it should just send a ^J.
    - ec109685 2y ago
      Can’t they just send large control characters as three separate key presses?
      - aidenn0 2y ago
        That will confuse screen/tmux which needs to distinguish between escape codes and a press of the escape key.

SoftTalker 2y ago

I'm not an expert in this area at all but I recall reading that trying to hide a signal in random noise doesn't really work, as you can still find the signal with statistical analysis or looking for distingishing characteristics that are not obscured. That was my first thought when I read about this new feature in OpenSSH, and it seems to have proven correct. Edit: wanted to add that I recognize that the people working on OpenSSH know a lot more about this than I do, and I had assumed they wouldn't bother implementing this if it wasn't a good idea, so willing to accept that "proven correct" may be an overstatement or even flat-out wrong.

milliams 2y ago
The advantage that OpenSSH have here is that they don't have to just hide the signal using noise, they can actually change the signal to look like the noise (or vice versa). For example they are making the signal packets some at a regular interval (since the "signal" being discussed is the timings). That alone would not hide the number of keypresses, but adding the chaff should do so. In this case, as said above, it seems like there's an implementation issue with actually doing those obfuscations, allowing the signal to be identified.
6502nerdface 2y ago
> trying to hide a signal in random noise doesn't really work Actually it works perfectly and it's called a one-time pad!

qhwudbebd 2y ago

This was present in versions 9.5 - 9.7 of OpenSSH and fixed in 9.8p1 released at the beginning of July. Philippos (along with others) is credited for reporting it in the release notes: https://www.openssh.com/releasenotes.html#9.8p1

palisade 2y ago
Worth noting too, that other linux systems using 8.9, e.g. ubuntu are 8.9p1 and have the patch applied but to an older version so they're also safe. Just in case anyone was worried about that.
- 0x0 2y ago
  I'd be very surprised if ubuntu has backported the keystroke obfuscation feature (which was introduced in 9.5) to 8.9. When the feature is missing, you're not safe from keystroke interception at all, which is what this bug is all about, so any 8.9 version would be actually considered "unsafe" until the whole feature is backported, which seems unlikely to happen?

keyboardcaper 2y ago

There is a market for keyboards which obscure keypress timings.

gunapologist99 2y ago

Yet another reason to switch to SSH keys instead of passwords wherever possible. Simple and bulletproof, with a minimum of footguns. EDIT: @tedunangst is right, wouldn't have helped. Even so, switch to keys!

tedunangst 2y ago
Does nothing here.
- deleted 2y ago
  [deleted]

hi-v-rocknroll 2y ago

Perhaps instead shooting off individual keypresses with some random chaff packets, it would be wiser to have constant interval, constant low data rate, (say 8 KBps) data rate trickling packets through and let the other side throw away pick out the needles from the haystack of no ops. Good luck finding a side-channel in that.

pshc 2y ago
Sounds good until (for example) you find out the SSH client library wasn't queueing up packets ahead of time, but instead building them at send time, and a chaff packet takes less time to build than a real packet... so many ways to get side-channeled.
- bubblesnort 2y ago
  Crypto requires a good source of random entropy. I suppose if you want more traffic for obscurity, you'll also have to find more random bits. Without enough entropy, you'd sacrifice security for obscurity. Just my ¢2, ymmv, etc

mrguyorama 2y ago

Regardless of what the "cause" of this is (ie bug or bad design), it's obvious nobody even took a look at this after it was coded up and merged in. The larger packets and double server ACKs are very clearly obviously giving the game away. This feature just doesn't even come close to what it supposedly does, was it even shipped as a feature or something that wasn't yet done? Not exactly encouraging, to see a system so integral to security seemingly shipping code without even a minor test of intended function.

tedunangst 2y ago
I think it's worth asking how important the feature is in the grand scheme of things. But what about the CIA timing my keystrokes is more of a nerd forum thing than an actual attack vector you read about in intrusion post mortems.
- talkin 2y ago
  Well, or it is important, and then you add the countermeasures. These countermeasures are quite easy to mess up, so doing the validation (on an ongoing basis!) MUST be part of the deal. Or if you think it’s not important enough to do those assertions in CI, then it might be better to just reject the obfuscation attempts. There’s no middleground: doing the implementation without checks, means you added complexity, you dont know if security improved (or worsened!), and the the release note might come down to a false sense of security.

jonmon6691 2y ago

Most people would consider streaming music to be a negligible amount of bandwidth. Seems to me like SSH in a "high security mode" could just send X Kbps of bi-directional pad at a layer directly above encryption but below application. Then just use that channel for all the normal SSH traffic. You could either treat this as a bandwidth limited channel or do some slow time-constant ramping up and down at the risk of leaking information about file downloads or large command outputs.

advael 2y ago
This seems like a reasonable solution to me, and I often stream music through my ssh sessions via port forwards so I may well already be getting the benefit in some places
mrngm 2y ago
This and a similar suggestion in another thread may sound nice and easy, "just add a constant stream of noise", but it assumes you can generate enough constant noise and be able to intersperse the noise with valid commands without being able to distinguish these events. The problem is not necessarily that you want to hide (to a network adversary) that you've been typing. It's that you do not want to reveal, through some side-channel, what the exact contents were. On the openssh-unix-dev mailing list, someone recently pointed out[0] that just periodically (without jitter) sending out packets may be problematic due to subtle differences in clock timing. Aside, they also link to a presentation[1] [PDF] that shows influence of temperature on clock skew (especially page 18) and that this gives a possibility for fingerprinting. Then there's the challenge of keeping SSH interactive enough that people do not experience too much input lag while typing. What if the user typed a character, but due to such a timing side-channel preventive measure, that character needs to be sent in the next packet, adding latency to the user experience? Surely it improves security, but it may add too much frustration for regular usage. [0] https://marc.info/?l=openssh-unix-dev&m=169402700622936&w=2 https://marc.info/?l=openssh-unix-dev&m=169402700622936&w=2 [1] https://murdoch.is/talks/ccs06hotornot.pdf https://murdoch.is/talks/ccs06hotornot.pdf ([2006] Hot or Not: Revealing hidden services by their clock skew, see also https://doi.org/10.1145/1180405.1180410 https://doi.org/10.1145/1180405.1180410 and an HN thread from 2014: https://news.ycombinator.com/item?id=7694612 https://news.ycombinator.com/item?id=7694612)

jgeowugejsb 2y ago

[flagged]