Kafka has `zstd` encoding. Here is our version of the streaming decoder i wrote a while ago https://github.com/vectorizedio/redpanda/blob/dev/src/v/compression

Kafka has `zstd` encoding. Here is our version of the streaming decoder i wrote a while ago https://github.com/vectorizedio/redpanda/blob/dev/src/v/compression/stream_zstd.cc https://github.com/vectorizedio/redpanda/blob/dev/src/v/comp... that's our default for our internal RPC as well. in fact kafka protocol support lz4, zstd, snappy, gzip all of them. and you can change them per batch. compression is good w/ kafka.