3 ms·

What about using gzip on messages? I'd expect for it to yield similar savings while keeping message format simpler.

• masklinn 2y ago
  You'd never use gzip for that, it has two dozen bytes of overhead before you can even start building a deflate stream. Even a raw deflate stream might not recoup the overhead on values which are only a few dozen bytes at the most, and if you're not using a static huffman tree the overhead for the tree is going to end all possibility of a gain. Furthermore LZ constructions work off of redundancy, which is difficult to find in large amounts in such short strings. The overhead of framing will just increase the size of the payload after construction. You could define just a static huffman tree and make that part of your spec.
  • chaokunyang 2y ago
    Yes, totally agree. I tried gzip first, but it introduce about 10~20 bytes cost first. So I proposed such meta encoding here. If we have a much bigger string, gzip will be better, but all string we want to compress here are just classname/fieldname/packagename/namespace/path/modulename. We don't have enough repete pattern for gzip to work
    • Dylan16807 2y ago
      If you rip off the gzip headers and footers, DEFLATE applied to a small string only has 10 bits of overhead, 2 bytes with padding. The default Huffman table is not suited for this use case, and a Huffman table in general is a waste of space these days, but the overhead is very small and can be reduced to nothing if you know the original string length.
• chaokunyang 2y ago
  Using gzip to the whole stream will introduce extra cpu cost, we try to make out serialization implementation as fast as possible. If we apply gzip only on such small meta string, it emits bigger size. I belive some stats are written in the data.