6 ms·

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambd

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambda environments (at least for Node) do not have the `zip` executable available which means that to zip/unzip something inside Lambda I need one written in javascript which has to be included in the zip file I upload to AWS instead of being able to use the native one.

skybrian 9y ago
You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.
- fulafel 9y ago
  But this does not work for streaming since the directory is at the end. So you have to temporarily store the whole zip file.
  - skybrian 9y ago
    Yes, you need random-access file I/O. But if the zip file isn't stored locally, sending the entire thing over the network to access one subfile is very inefficient. Instead you'd want a network protocol where you request the files you actually want and the remote server just sends those.
- sheetjs 9y ago
  The table of contents in a zip file is at the end, not the beginning, so you need random access to the full file if you want to properly jump to the right location. Some zip files use the length feature of local file headers so you can jump ahead in the file without processing the data, but many skip the length so you have to work through the DEFLATE algorithm to find the end of the first file (to find where the second file starts)
  - skybrian 9y ago
    Yes, you can and should use random file access to load things from a zip file. For example, Go's zip package takes an io.ReaderAt interface, not an io.Reader. Using sequential I/O for this would be inefficient.
    - jlouis 9y ago
      The problem here is that you mostly have a streaming interface In s3.
      - skybrian 9y ago
        I'm not familiar with s3, but it seems like if you want random access for a key-value database, you should store each file under a separate key? Storing an entire file archive as a blob (in any format) and downloading the whole thing just to access one file seems like a weird way to do it. Apparently there is "s3 select" in preview [1], but it only supports gzipped CSV or JSON. [1] https://aws.amazon.com/blogs/aws/s3-glacier-select/ https://aws.amazon.com/blogs/aws/s3-glacier-select/
      - etaioinshrdlu 9y ago
        S3 supports reading ranges of bytes. It might not always be a great idea for performance.
FLUX-YOU 9y ago
.tar.gz sucks on Windows though (even 7-zip is a two step process).
- braderhart 9y ago
  The problem though is that Windows sucks, not 7-zip. You are just using a crappy closed-source spyware OS.
userbinator 9y ago
Zip files cannot be unzipped without having the whole thing in memory. Where/how did you come upon that piece of misinformation? I can still remember using PKZIP to extract archives of over 100MB, on a machine with only 4MB of RAM, many years ago.

skybrian 9y ago

You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.

fulafel 9y ago
But this does not work for streaming since the directory is at the end. So you have to temporarily store the whole zip file.
- skybrian 9y ago
  Yes, you need random-access file I/O. But if the zip file isn't stored locally, sending the entire thing over the network to access one subfile is very inefficient. Instead you'd want a network protocol where you request the files you actually want and the remote server just sends those.
sheetjs 9y ago
The table of contents in a zip file is at the end, not the beginning, so you need random access to the full file if you want to properly jump to the right location. Some zip files use the length feature of local file headers so you can jump ahead in the file without processing the data, but many skip the length so you have to work through the DEFLATE algorithm to find the end of the first file (to find where the second file starts)
- skybrian 9y ago
  Yes, you can and should use random file access to load things from a zip file. For example, Go's zip package takes an io.ReaderAt interface, not an io.Reader. Using sequential I/O for this would be inefficient.
  - jlouis 9y ago
    The problem here is that you mostly have a streaming interface In s3.
    - skybrian 9y ago
      I'm not familiar with s3, but it seems like if you want random access for a key-value database, you should store each file under a separate key? Storing an entire file archive as a blob (in any format) and downloading the whole thing just to access one file seems like a weird way to do it. Apparently there is "s3 select" in preview [1], but it only supports gzipped CSV or JSON. [1] https://aws.amazon.com/blogs/aws/s3-glacier-select/ https://aws.amazon.com/blogs/aws/s3-glacier-select/
    - etaioinshrdlu 9y ago
      S3 supports reading ranges of bytes. It might not always be a great idea for performance.

FLUX-YOU 9y ago

.tar.gz sucks on Windows though (even 7-zip is a two step process).

braderhart 9y ago
The problem though is that Windows sucks, not 7-zip. You are just using a crappy closed-source spyware OS.

userbinator 9y ago

Zip files cannot be unzipped without having the whole thing in memory. Where/how did you come upon that piece of misinformation? I can still remember using PKZIP to extract archives of over 100MB, on a machine with only 4MB of RAM, many years ago.