3 ms·

That can't be the case anymore because I implemented a streaming function a few years back that sends an unlimited number of files as a zip, but builds the zip

That can't be the case anymore because I implemented a streaming function a few years back that sends an unlimited number of files as a zip, but builds the zip on the fly streaming it to the client. Maybe it works because I am not using compression (files in my case are all JPG and/or compressed video so there is little benefit). But my process definitely starts streaming the zip right away and has to pull URLs on the fly to create the zip file.

kbenson 10y ago
I'm pretty sure you were using the deflate stream format. There may be other formats (maybe with different magic numbers in the header) that have slightly different capabilities and needs.
TimMeade 10y ago
This cannot be a 'true' zip file. The central directory is at the end of the zip file. It's made up of information that is needed for the decompression. So if you are streaming it, you cannot decompress until the entire file is reassembled.
- pixelglow 10y ago
  You can definitely compose a zip file "on the fly" and stream it out. The central directory at the end of the zip file can be determined from all the content already streamed. The only wrinkle is that each entry has a header which typically states the compressed size and checksum. Either you have to compress each entry content in some temp buffer or file to figure out the compressed size and checksum, then write out the header and compressed content. Or you write out the header with these fields zeroed, then compress the content on the fly and write it out, then write out a data descriptor with the compressed size and checksum.
  - derefr 10y ago
    What the parent is saying is that the client has to buffer the entire zip file before they receive the index and can begin decompressing it. "Streaming compression" usually implies streaming decompression as well—importantly because a non-negligible use case for streaming compression is sending streams of compressed data that won't fit on the destination together with the decompressed copy; or sending continuous streams of compressed data that could never reach the end to be decompressed. What the parent is saying is that zip cannot work for these use-cases.

kbenson 10y ago

I'm pretty sure you were using the deflate stream format. There may be other formats (maybe with different magic numbers in the header) that have slightly different capabilities and needs.

TimMeade 10y ago

This cannot be a 'true' zip file. The central directory is at the end of the zip file. It's made up of information that is needed for the decompression. So if you are streaming it, you cannot decompress until the entire file is reassembled.

pixelglow 10y ago
You can definitely compose a zip file "on the fly" and stream it out. The central directory at the end of the zip file can be determined from all the content already streamed. The only wrinkle is that each entry has a header which typically states the compressed size and checksum. Either you have to compress each entry content in some temp buffer or file to figure out the compressed size and checksum, then write out the header and compressed content. Or you write out the header with these fields zeroed, then compress the content on the fly and write it out, then write out a data descriptor with the compressed size and checksum.
- derefr 10y ago
  What the parent is saying is that the client has to buffer the entire zip file before they receive the index and can begin decompressing it. "Streaming compression" usually implies streaming decompression as well—importantly because a non-negligible use case for streaming compression is sending streams of compressed data that won't fit on the destination together with the decompressed copy; or sending continuous streams of compressed data that could never reach the end to be decompressed. What the parent is saying is that zip cannot work for these use-cases.