6 ms·

Debunking the LZ4 "20 years old bug" myth

178 points by bronson 12y ago

iclelland 12y ago
This all looks like a lot more drama than it really is. There's a lot of reasonable discussion going on between the two parties now. See http://seclists.org/oss-sec/2014/q2/681 http://seclists.org/oss-sec/2014/q2/681 and https://code.google.com/p/lz4/issues/detail?id=52&can=1 https://code.google.com/p/lz4/issues/detail?id=52&can=1 for more recent, measured responses.
aortega 12y ago
"besides TCP messages have a hard-code limit of 64KB."
- hcles 12y ago
  A legit criticism, but who would transmit an 8Mb datablock using LZ4 over a network? There are far better compression schemes whose lower performance tradeoffs become negligible when network latency is involved. I am using LZ4 currently for a project compressing Netflow-derived data for a major ISP, even then it's in 64Kb chunks for quick access and decompression.
  - Xylakant 12y ago
    > but who would transmit an 8Mb datablock using LZ4 over a network Well, an attacker would. The question is whether the listener accepts that block, decodes it and gets pwned.
    - derefr 12y ago
      The question is whether every single router between the attacker and the destination accepts that block and passes it on. TCP window scaling will get an ISO across a building faster, or data from EC2 to S3 faster, but it won't get your 8MB block through the internet backbone. And, keeping in mind, that if you do manage to break something, it'll likely be the very first router in the chain, not the destination system. Exploiting said router in such a way that it will exactly rebroadcast the problematic packet (i.e. perform its intended function), from within your own shellcode--and then continuing that chain all the way to the destination--would be quite a feat.
      - jacquesm 12y ago
        Doesn't the compression happen at the frame level rather than at the connection level?
      - Xylakant 12y ago
        Depending on where the attacker and where the potential victim are located and depending the protocol used and on how the network is structured all of that may or may not be a problem for the attacker. Maybe he already owned the last router before the victim via a totally unrelated issue and it's just a single hop. I concur that this bug is unlikely to be exploited in the wild but I'm generally opposed to the statement "yes, that may be a critical problem, but who would do that?!" People that want to attack you do that. Today I read an article where the response to a critical flaw in a process was "but that's not a real world scenario, those actions would need to be done deliberately and that would require criminal energy." Phew, we're safe then. Only positive energy around, move along, nothing to see here.
        derefr 12y ago
        I wasn't so much saying "phew, you can relax" as I was saying "this is why you haven't seen a new Internet worm based on this concept." It can still, of course, be done in one-off scenarios (attacking a peer on a guest wi-fi network is one easy possibility) but it's not Heartbleed-level scary, because you can't just "scan the Internet" for the vulnerability, and attack every vulnerable thing you find to useful result.
      - noselasd 12y ago
        That's not particularly relevant, you're thinking at the wrong layer. Seen from a program sitting on top of TCP, there's no such thing as a TCP message/block. You can very well grab a single 16MB chunk with 1 read() call, if the socket buffer is sized for that. Or you can append incoming data to a big internal buffer which you pass through decompression once that buffer fills up. You would have to look at protocols sitting above TCP that transfers compressed data, that typically would define a message based protocol, and how an implementation decompresses that data.
    - bleakcabal 12y ago
      Even then, like it's mentioned in the article you need a 16mb datablock.
    - mtdewcmu 12y ago
      What about Martians? Remember, the bug is on Mars.
thegeomaster 12y ago
There is a chance that e.g. some internal formats within companies use LZ4 in a way that's vulnerable. They have legacy code and legacy systems and who knows what's at play out there. Plugging this vuln would be a very good idea. The author hasn't convinced me completely that absolutely no one uses it that way.
hblanks 12y ago
Lobste.rs, of course, toplists the original citation in place of Yann's response. I find it better reading.
dang 12y ago
The original article was posted at https://news.ycombinator.com/item?id=7951324 https://news.ycombinator.com/item?id=7951324.
lostlogin 12y ago
And now its hit the front (Internet) page of a big paper. The Mars Rover hardly gets a mention, despite the headline. http://www.telegraph.co.uk/technology/news/10930090/Software-bug-found-on-Mars-Curiosity-rover.html http://www.telegraph.co.uk/technology/news/10930090/Software...

178 points by bronson 12y ago

This all looks like a lot more drama than it really is. There's a lot of reasonable discussion going on between the two parties now. See http://seclists.org/oss-sec/2014/q2/681 http://seclists.org/oss-sec/2014/q2/681 and https://code.google.com/p/lz4/issues/detail?id=52&can=1 https://code.google.com/p/lz4/issues/detail?id=52&can=1 for more recent, measured responses.

aortega 12y ago

"besides TCP messages have a hard-code limit of 64KB."

hcles 12y ago
A legit criticism, but who would transmit an 8Mb datablock using LZ4 over a network? There are far better compression schemes whose lower performance tradeoffs become negligible when network latency is involved. I am using LZ4 currently for a project compressing Netflow-derived data for a major ISP, even then it's in 64Kb chunks for quick access and decompression.
- Xylakant 12y ago
  > but who would transmit an 8Mb datablock using LZ4 over a network Well, an attacker would. The question is whether the listener accepts that block, decodes it and gets pwned.
  - derefr 12y ago
    The question is whether every single router between the attacker and the destination accepts that block and passes it on. TCP window scaling will get an ISO across a building faster, or data from EC2 to S3 faster, but it won't get your 8MB block through the internet backbone. And, keeping in mind, that if you do manage to break something, it'll likely be the very first router in the chain, not the destination system. Exploiting said router in such a way that it will exactly rebroadcast the problematic packet (i.e. perform its intended function), from within your own shellcode--and then continuing that chain all the way to the destination--would be quite a feat.
    - jacquesm 12y ago
      Doesn't the compression happen at the frame level rather than at the connection level?
    - Xylakant 12y ago
      Depending on where the attacker and where the potential victim are located and depending the protocol used and on how the network is structured all of that may or may not be a problem for the attacker. Maybe he already owned the last router before the victim via a totally unrelated issue and it's just a single hop. I concur that this bug is unlikely to be exploited in the wild but I'm generally opposed to the statement "yes, that may be a critical problem, but who would do that?!" People that want to attack you do that. Today I read an article where the response to a critical flaw in a process was "but that's not a real world scenario, those actions would need to be done deliberately and that would require criminal energy." Phew, we're safe then. Only positive energy around, move along, nothing to see here.
      - derefr 12y ago
        I wasn't so much saying "phew, you can relax" as I was saying "this is why you haven't seen a new Internet worm based on this concept." It can still, of course, be done in one-off scenarios (attacking a peer on a guest wi-fi network is one easy possibility) but it's not Heartbleed-level scary, because you can't just "scan the Internet" for the vulnerability, and attack every vulnerable thing you find to useful result.
    - noselasd 12y ago
      That's not particularly relevant, you're thinking at the wrong layer. Seen from a program sitting on top of TCP, there's no such thing as a TCP message/block. You can very well grab a single 16MB chunk with 1 read() call, if the socket buffer is sized for that. Or you can append incoming data to a big internal buffer which you pass through decompression once that buffer fills up. You would have to look at protocols sitting above TCP that transfers compressed data, that typically would define a message based protocol, and how an implementation decompresses that data.
  - bleakcabal 12y ago
    Even then, like it's mentioned in the article you need a 16mb datablock.
  - mtdewcmu 12y ago
    What about Martians? Remember, the bug is on Mars.

thegeomaster 12y ago

There is a chance that e.g. some internal formats within companies use LZ4 in a way that's vulnerable. They have legacy code and legacy systems and who knows what's at play out there. Plugging this vuln would be a very good idea. The author hasn't convinced me completely that absolutely no one uses it that way.

hblanks 12y ago

Lobste.rs, of course, toplists the original citation in place of Yann's response. I find it better reading.

dang 12y ago

The original article was posted at https://news.ycombinator.com/item?id=7951324 https://news.ycombinator.com/item?id=7951324.

lostlogin 12y ago

And now its hit the front (Internet) page of a big paper. The Mars Rover hardly gets a mention, despite the headline. http://www.telegraph.co.uk/technology/news/10930090/Software-bug-found-on-Mars-Curiosity-rover.html http://www.telegraph.co.uk/technology/news/10930090/Software...