3 ms·
Gzip will make most line protocols efficient enough that you can do away with needing to write a cryptic one that will just end up being friction every time som
by hinkley 9mo ago
Gzip will make most line protocols efficient enough that you can do away with needing to write a cryptic one that will just end up being friction every time someone has to triage a production issue. Zstd will do even better.
The real one-two punch is make your parser faster and then spend the CPU cycles on better compression.
- cb321 9mo agoDNA researchers developed a parallel format for gzip they call "bgzip" ( https://learngenomics.dev/docs/genomic-file-formats/compression-and-BGZF/ https://learngenomics.dev/docs/genomic-file-formats/compress... ) that makes data seem less trapped behind a decompression perf wall. Zstd is still a bit faster (but < ~2X) and also gets better compression ratios (https://forum.nim-lang.org/t/5103#32269 https://forum.nim-lang.org/t/5103#32269)