5 ms·

Hello tptacek, it is not very common with the normal use case to decompress things that you didn't compressed yourself, but I agree with you that is better to a

by antirez 18y ago

Hello tptacek, it is not very common with the normal use case to decompress things that you didn't compressed yourself, but I agree with you that is better to add this checks at the cost of some speed. Btw note that this is common in compression code that tries to do fast. in things like Fastlz and similar libs this is a compile time define, I may do the same, if SAFE_DECOMPRESSOR is defined it will do the check, otherwise will just skip the checking code. Also note that the access is read only, so basically it's not possible to use such thing to run unprivileged code. Also in moder operating systems pages are zeroed before to be used in a different process, so it is not possible to read part of memory about a different process. Thanks for the hint

mustpax 18y ago
This could be the backend for a cool stateless URL shortener. Not super useful, but stateless things are pretty nifty. In this case you'd be getting unsafe data for both compression and decompression. I really like your project though. Thanks for sharing!
- antirez 18y ago
  Thanks mustpax! I agree that stateless things are cool :) Btw optimizing the dictionary specifically for urls much better compression ratios are possible.
tptacek 18y ago
Three observations: (1) It is extremely common for network apps to decompress blobs they didn't generate; even in cases where those blobs aren't passed client/server, but are instead roundtripped through a database or shared persistence layer. (2) Whatever fastlz may do, the industry standard (in Win32 and Unix and in embedded devices) is zlib --- at least from what we've found on app pentests --- and zlib doesn't have this problem (now). (3) Zlib is itself a minefield of vulnerabilities, which supports the point that your compression codec is a security hotspot for an application. But my point isn't that your code sucks; it's nice and tight.
- antirez 18y ago
  1) ACK, you are right, actually I'm probably going to use in the future this lib for Redis (http://code.google.com/p/redis http://code.google.com/p/redis), and it is important that an untrusted DB will load correctly without memory corruptions or invalid accesses. 2) Right, but a define to don't this checks is useful for environments where data can be trusted but speed is very important. 3) Good point. I'll fix the code. Btw what is nice about that kind of things is that testing against random data helps a lot. Zlib should do that. For example, do you remember the old security bug of old bind implementations of using a 'ptr' opcode (used for compression) in DNS packets in order to create an infinite loop and mount a DoS? Years ago I tested how this bug were trivial to spot just processing random data. This is true for a lot of applications. If you know how the application works you can "tune" this random data with biases that will stimulate a lot more the code. This is a worthy testing tool that is underused IMHO. Thanks for your comment.
  - tptacek 18y ago
    The testing you're describing is called "fuzzing", and it's the dominant way researchers find vulnerabilities today (I didn't fuzz your code; I just read it, and then wrote a little test doohickey). David Molnar at Berkeley has done a ton of work on smart fuzzing: http://www.cs.berkeley.edu/~dmolnar/ http://www.cs.berkeley.edu/~dmolnar/ Ilja van Sprundel wrote what has got to be the most popular "dumb" fuzzer, mangle.c: http://www.digitaldwarf.be/products/mangle.c http://www.digitaldwarf.be/products/mangle.c Mangle routinely finds horrible, horrible things in modern code; Ilja deserves some kind of anti-Nobel Prize for it.

by antirez 18y ago

mustpax 18y ago

This could be the backend for a cool stateless URL shortener. Not super useful, but stateless things are pretty nifty. In this case you'd be getting unsafe data for both compression and decompression. I really like your project though. Thanks for sharing!

antirez 18y ago
Thanks mustpax! I agree that stateless things are cool :) Btw optimizing the dictionary specifically for urls much better compression ratios are possible.

tptacek 18y ago

Three observations: (1) It is extremely common for network apps to decompress blobs they didn't generate; even in cases where those blobs aren't passed client/server, but are instead roundtripped through a database or shared persistence layer. (2) Whatever fastlz may do, the industry standard (in Win32 and Unix and in embedded devices) is zlib --- at least from what we've found on app pentests --- and zlib doesn't have this problem (now). (3) Zlib is itself a minefield of vulnerabilities, which supports the point that your compression codec is a security hotspot for an application. But my point isn't that your code sucks; it's nice and tight.

antirez 18y ago
1) ACK, you are right, actually I'm probably going to use in the future this lib for Redis (http://code.google.com/p/redis http://code.google.com/p/redis), and it is important that an untrusted DB will load correctly without memory corruptions or invalid accesses. 2) Right, but a define to don't this checks is useful for environments where data can be trusted but speed is very important. 3) Good point. I'll fix the code. Btw what is nice about that kind of things is that testing against random data helps a lot. Zlib should do that. For example, do you remember the old security bug of old bind implementations of using a 'ptr' opcode (used for compression) in DNS packets in order to create an infinite loop and mount a DoS? Years ago I tested how this bug were trivial to spot just processing random data. This is true for a lot of applications. If you know how the application works you can "tune" this random data with biases that will stimulate a lot more the code. This is a worthy testing tool that is underused IMHO. Thanks for your comment.
- tptacek 18y ago
  The testing you're describing is called "fuzzing", and it's the dominant way researchers find vulnerabilities today (I didn't fuzz your code; I just read it, and then wrote a little test doohickey). David Molnar at Berkeley has done a ton of work on smart fuzzing: http://www.cs.berkeley.edu/~dmolnar/ http://www.cs.berkeley.edu/~dmolnar/ Ilja van Sprundel wrote what has got to be the most popular "dumb" fuzzer, mangle.c: http://www.digitaldwarf.be/products/mangle.c http://www.digitaldwarf.be/products/mangle.c Mangle routinely finds horrible, horrible things in modern code; Ilja deserves some kind of anti-Nobel Prize for it.