6 ms·

Compression schemes have to be predictable when run on the same data. Otherwise decompression wouldn't work. If signature detection actually is more difficult

by tsally 17y ago

Compression schemes have to be predictable when run on the same data. Otherwise decompression wouldn't work. If signature detection actually is more difficult on compressed data, it's not because the compression is unpredictable.

ryanpetrich 17y ago
Compression schemes have to decode predictably, but they don't necessarily generate the same compressed data. gzip -9 will usually use less space than gzip -1 and decompress to the same data. What's probably happening is virus scanners are taking the easy way out rather than implementing decompression algorithms in their scanning engines (it probably looks better on traditional benchmarks too)
- tsally 17y ago
  Even if different data are being generated, the compression is still predictable. There must be something in the gzip header indicating the level of compression (to ensure that the correct decompression algorithm is used). Point being, there's nothing stopping the AV program from figuring out which compressed signature is the correct one. It can even compress the raw signature on the fly. On your second point I agree. I'll refrain from ranting about AV programs here, but suffice to say there hasn't been enough innovation in the field because AV companies are able to sell substandard products and still make good money.
  - dmv 17y ago
    To be more specific, compression schemes decode predictably, but any given portion of the compression stream need not be independently predictable. Gzip (and DEFLATE in general) use adaptive compression approaches - the data determines the encoding. If you know your content of interest is in the beginning of the stream, pre-computing different compressions should work out. Otherwise...
  - ryanpetrich 17y ago
    You've got it backwards: it is the decompression that is predictable; compressors are free to transform the input in any way such that a compliant decompressor will extract the original file. The gzip format documentation is available here btw: http://www.gzip.org/zlib/rfc-gzip.html http://www.gzip.org/zlib/rfc-gzip.html
hristov 17y ago
Yes they are ultimately predictable, but they are not easily predictable. That is one usually has to go through all the steps of decompression to predict them there are no shortcuts. Imagine a data stream X that is compressed into another data stream Y. Imagine that a small portion of X is data stream x1 which is the portion of data used for a signature. That will get compressed into y1. Now lets define x2 as all the data in X that is not x1. Now if you are always guaranteed that the same x1 would get compressed into the same y1, then things would be easily predictable and you can just compare compressed signatures. But this is not the case. If x2 is different, then the same x1 can be compressed into a different string.

by tsally 17y ago

ryanpetrich 17y ago

Compression schemes have to decode predictably, but they don't necessarily generate the same compressed data. gzip -9 will usually use less space than gzip -1 and decompress to the same data. What's probably happening is virus scanners are taking the easy way out rather than implementing decompression algorithms in their scanning engines (it probably looks better on traditional benchmarks too)

tsally 17y ago
Even if different data are being generated, the compression is still predictable. There must be something in the gzip header indicating the level of compression (to ensure that the correct decompression algorithm is used). Point being, there's nothing stopping the AV program from figuring out which compressed signature is the correct one. It can even compress the raw signature on the fly. On your second point I agree. I'll refrain from ranting about AV programs here, but suffice to say there hasn't been enough innovation in the field because AV companies are able to sell substandard products and still make good money.
- dmv 17y ago
  To be more specific, compression schemes decode predictably, but any given portion of the compression stream need not be independently predictable. Gzip (and DEFLATE in general) use adaptive compression approaches - the data determines the encoding. If you know your content of interest is in the beginning of the stream, pre-computing different compressions should work out. Otherwise...
- ryanpetrich 17y ago
  You've got it backwards: it is the decompression that is predictable; compressors are free to transform the input in any way such that a compliant decompressor will extract the original file. The gzip format documentation is available here btw: http://www.gzip.org/zlib/rfc-gzip.html http://www.gzip.org/zlib/rfc-gzip.html

hristov 17y ago

Yes they are ultimately predictable, but they are not easily predictable. That is one usually has to go through all the steps of decompression to predict them there are no shortcuts. Imagine a data stream X that is compressed into another data stream Y. Imagine that a small portion of X is data stream x1 which is the portion of data used for a signature. That will get compressed into y1. Now lets define x2 as all the data in X that is not x1. Now if you are always guaranteed that the same x1 would get compressed into the same y1, then things would be easily predictable and you can just compare compressed signatures. But this is not the case. If x2 is different, then the same x1 can be compressed into a different string.