3 ms·
I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/
by userbinator 6y ago
I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
- icegreentea2 6y agoI think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk. The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to writing the whole file to memory before serving (thus memory inflation). This library just looks like a pretty straightforward way to implement the same idea, but with chunking to bound memory usage. At the bottom, it's doing the same thing, but using generators to yield chunks at a time. It's a useful utility for that context. Nothing groundbreaking, it's something that most intermediate and higher developers could stitch together in probably a few days (especially if they had to brush on on DEFLATE and generator protocol), but it's nice to not have to.
- thewakalix 6y agoAh, so it's like Haskell's streaming (de)compression functions? Examples: * https://hackage.haskell.org/package/conduit-extra-1.1.7.3/docs/Data-Conduit-Zlib.html https://hackage.haskell.org/package/conduit-extra-1.1.7.3/do... * http://hackage.haskell.org/package/streaming-utils-0.2.0.0/docs/Streaming-Zip.html http://hackage.haskell.org/package/streaming-utils-0.2.0.0/d...
- CodesInChaos 6y agoThe functions you link look like they're for simple deflate streams (i.e. a single file), while the OP appears to be about streaming zip archives which can contain multiple files with metadata.
- harel 6y ago"without writing to disk" This is a valid concern and a good enough reason to have such a library. I've written a similar thing for uploading large files to S3 (via Django) by streaming them without the file ever touching the file system (S3ChunkUploader). The reason was the large files were deemed security-sensitive and the containers were limited to 2GB in disk space. Just uploading 4 500MB files at the same time would be an attack vector.
- tyingq 6y agoI believe the comparison is just to the bundled zipfile module and BytesIO, which would be the quick and dirty way to make a zipfile without creating actual files, but would be memory intensive.
- cozzyd 6y agonot a safe assumption with Python packages!