4 ms·
I generally use gzip for everything because it's everywhere and good enough, but xz and bzip also support streaming, in fact anything that tar compresses does a
by Tiksi 11y ago
I generally use gzip for everything because it's everywhere and good enough, but xz and bzip also support streaming, in fact anything that tar compresses does afaik.
> dd if=/dev/urandom bs=1M count=5 | gzip > test
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.438033 s, 12.0 MB/s
> dd if=/dev/urandom bs=1M count=5 | xz > test
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 1.52744 s, 3.4 MB/s
> dd if=/dev/urandom bs=1M count=5 | bzip2 > test
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.804324 s, 6.5 MB/s
- Ded7xSEoPKYNsDd 11y agoThe algorithms support streaming, but that doesn't mean implementations in (in this case) Python libraries still might not do so. Although I can't understand why they wouldn't, presumably they just wrap the same C libraries everybody else uses, and a streaming interface wrapping another stream (i.e. file, socket) should feel very natural and easy.