4 ms·
I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve fil
by icegreentea2 6y ago
I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk.
The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to writing the whole file to memory before serving (thus memory inflation).
This library just looks like a pretty straightforward way to implement the same idea, but with chunking to bound memory usage. At the bottom, it's doing the same thing, but using generators to yield chunks at a time.
It's a useful utility for that context. Nothing groundbreaking, it's something that most intermediate and higher developers could stitch together in probably a few days (especially if they had to brush on on DEFLATE and generator protocol), but it's nice to not have to.
- thewakalix 6y agoAh, so it's like Haskell's streaming (de)compression functions? Examples: * https://hackage.haskell.org/package/conduit-extra-1.1.7.3/docs/Data-Conduit-Zlib.html https://hackage.haskell.org/package/conduit-extra-1.1.7.3/do... * http://hackage.haskell.org/package/streaming-utils-0.2.0.0/docs/Streaming-Zip.html http://hackage.haskell.org/package/streaming-utils-0.2.0.0/d...
- CodesInChaos 6y agoThe functions you link look like they're for simple deflate streams (i.e. a single file), while the OP appears to be about streaming zip archives which can contain multiple files with metadata.
- harel 6y ago"without writing to disk" This is a valid concern and a good enough reason to have such a library. I've written a similar thing for uploading large files to S3 (via Django) by streaming them without the file ever touching the file system (S3ChunkUploader). The reason was the large files were deemed security-sensitive and the containers were limited to 2GB in disk space. Just uploading 4 500MB files at the same time would be an attack vector.