5 ms·

I haven't read the whole article yet, but gzip and bzip2 recover fairly well from errors. As they are divided into blocks, flipping a bit will make you lose, at

by lake99 10y ago

I haven't read the whole article yet, but gzip and bzip2 recover fairly well from errors. As they are divided into blocks, flipping a bit will make you lose, at most, one block. This is not related to error correction though, but a way of mitigating losses. >> Just one bit flip in the msb of any byte causes the remaining records to be read incorrectly. It also causes the size of the index to be calculated incorrectly, losing the position of the CRC32 and the stream footer. This sounds severe enough to me.

userbinator 10y ago
As they are divided into blocks, flipping a bit will make you lose, at most, one block. What happens if bits in the block header are corrupted? If it can't find the start of the next block, the same thing will happen. Also, breaking up the data into blocks will decrease compression, since each block starts with a fresh state. It is ultimately a tradeoff between compression ratio and error resistance.
- lake99 10y ago
  > If it can't find the start of the next block, the same thing will happen. This has been addressed in the article. "Bzip2 is affected by this defect to a lesser extent; it contains two unprotected length fields in each block header. Gzip may be considered free from this defect because its only top-level unprotected length field (XLEN) can be validated using the LEN fields in the extra subfields. Lzip is free from this defect." > Also, breaking up the data into blocks will decrease compression This has been tested very thoroughly. Larger block sizes give rapidly diminishing marginal returns (man bzip2). Now, the largest you can go with bzip2 is 900KB.
glandium 10y ago
gzip usually references the data from the previous window in the current window. If you lose one block, you're likely to lose the following one. And the one following it. etc. until the end of the compressed data.

by lake99 10y ago

userbinator 10y ago

As they are divided into blocks, flipping a bit will make you lose, at most, one block. What happens if bits in the block header are corrupted? If it can't find the start of the next block, the same thing will happen. Also, breaking up the data into blocks will decrease compression, since each block starts with a fresh state. It is ultimately a tradeoff between compression ratio and error resistance.

lake99 10y ago
> If it can't find the start of the next block, the same thing will happen. This has been addressed in the article. "Bzip2 is affected by this defect to a lesser extent; it contains two unprotected length fields in each block header. Gzip may be considered free from this defect because its only top-level unprotected length field (XLEN) can be validated using the LEN fields in the extra subfields. Lzip is free from this defect." > Also, breaking up the data into blocks will decrease compression This has been tested very thoroughly. Larger block sizes give rapidly diminishing marginal returns (man bzip2). Now, the largest you can go with bzip2 is 900KB.

glandium 10y ago

gzip usually references the data from the previous window in the current window. If you lose one block, you're likely to lose the following one. And the one following it. etc. until the end of the compressed data.