3 ms·
Slightly related: recently wrote a binary encoder/decoder that used bitpacking, delta-encoding, and other well-known 'tricks' to efficiently pack batches of 10k
by gbrits 7y ago
Slightly related: recently wrote a binary encoder/decoder that used bitpacking, delta-encoding, and other well-known 'tricks' to efficiently pack batches of 10k-100k events of the same uniform type (essentially encoding column by column and using similarities to my advantage). Nothing too fancy, but it was A) a huge success in terms of compresison-ratio and B) a hassle to write. Do any, more or less, turn-key solutions exist for this? Specifically targeting Node but a commandline util might work as well.
- jiggawatts 7y agoI had a play with something similar and I noticed that -- at least for my use case -- you can get ~90% of the gains through just a couple of simple tricks: 1) Identify medium-scale similarity boundaries in the data structures. E.g.: a sequence of messages in a protocol, such as a C "struct" with a bunch of fields. 2) Compute the binary difference between these structures so that most of the subsequent bytes after the first message are either zeroes or small numbers. Both the sender and receiver have to keep the previous message in a buffer to allow this. 3) Use a high-performance compression algorithm that supports "user provided dictionaries", such as Zstandard. Train it with sample data. This above is surprisingly straightforward because it doesn't require complex changes to the underlying data structures. You don't even necessarily need to be able to parse it at all, as long as it has large-scale repeating structures that you can identify.
- gbrits 7y agoAgreed. It really blows generic compression algos out of the water. Didn't know about Zstandard dictionary encoding. This might just be what I'm after. Thanks