2 ms·
Depends a lot on your data. Our custom bit packed compression scheme is ten times smaller than protobufs using Google varint and about half the size of bzip2. W
by jrabone 14y ago
Depends a lot on your data. Our custom bit packed compression scheme is ten times smaller than protobufs using Google varint and about half the size of bzip2. We have a very specific, well-understood domain for these numbers and a lot of data to collect over limited bandwidth.
- barrkel 14y agoBit packing in the normal fashion as in the article - where you assume an exact bit length for all elements - isn't normally optimal. Arithmetic coding can use fractional bits, but bit backing forces you to a power of 2 domain.
- csense 14y agoA real compression algorithm experiences its gain due to low-entropy distribution of the input values. If your input is randomly distributed integers in the interval [0, 2^n), a bit-packed coding is the most efficient possible.