8 ms·
For me, as a Python user, I've found that gzip is currently the only compression format that allows streaming compression/decompression. I don't want to have to
by Phemist 11y ago
For me, as a Python user, I've found that gzip is currently the only compression format that allows streaming compression/decompression. I don't want to have to store hundreds of gigabytes of data and THEN compress it, rather than compressing it right during file generation. I haven't found any other compression lib that supports this out of the box.
- Tiksi 11y agoI generally use gzip for everything because it's everywhere and good enough, but xz and bzip also support streaming, in fact anything that tar compresses does afaik. > dd if=/dev/urandom bs=1M count=5 | gzip > test 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.438033 s, 12.0 MB/s > dd if=/dev/urandom bs=1M count=5 | xz > test 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 1.52744 s, 3.4 MB/s > dd if=/dev/urandom bs=1M count=5 | bzip2 > test 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.804324 s, 6.5 MB/s
- Ded7xSEoPKYNsDd 11y agoThe algorithms support streaming, but that doesn't mean implementations in (in this case) Python libraries still might not do so. Although I can't understand why they wouldn't, presumably they just wrap the same C libraries everybody else uses, and a streaming interface wrapping another stream (i.e. file, socket) should feel very natural and easy.