5 ms·
Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code undern
by da_big_ghey 6y ago
Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code underneath.
- jzwinck 6y agozipfile uses zlib which is C. But it's even better than that: it releases the GIL, so it gives linear speedup with multiple threads. If you need to (de)compress a bunch of files, you can do them all at once quite easily using e.g. concurrent.futures. If you want that speedup on the command line without Python, check out pigz. It's gzip with parallelism. Easy 10-20x speedup for some jobs.
- AzN1337c0d3r 6y agoNote that pigz only parallelizes compression, not decompression.