4 ms·

Unrelated to the issue in the post, but a few years back we ran into an annoying oddity in the Python zipfile module [1]. By default, when you create a new zip

Unrelated to the issue in the post, but a few years back we ran into an annoying oddity in the Python zipfile module [1]. By default, when you create a new zipfile object to create a .zip archive with, it stores the data uncompressed. Only noticed the issue when a 100MB zip file suddenly became 3MB after I had edited one of the files and re-saved the archive. [1] https://docs.python.org/3/library/zipfile.html https://docs.python.org/3/library/zipfile.html

Jiocus 6y ago
This isn't an oddity or issue with Python zip creation, it's expected behaviour as per ISO/IEC WD 21320-1[1]. > The compression method shall be either 0 (“stored”) or 8(“deflated”). The format does not prescribe a compression algorithm (many could be available for use). It's merely a container. [1] ISO/IEC WD 21320-1, Document Container File - Part 1: Core - https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.7181&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.39...
- bawolff 6y ago
  What defaults python chooses is an oddity within python.
  - Jiocus 6y ago
    Using defaults aimed at compatability with other zip archivers, today and tomorrow, doesn't strike me as odd.
    - dylan604 6y ago
      If you can't deflate existing ZIP files because no libzip, then you're not maintaining compatibility with anything really
masklinn 6y ago
Yes, it's clearly documented but very much unexpected the first time 'round. The reason for the default is probably that python can be built with none of zlib, bzip2 or lzma support: > If ZIP_DEFLATED, ZIP_BZIP2 or ZIP_LZMA is specified but the corresponding module (zlib, bz2 or lzma) is not available, RuntimeError is raised. in which case only STORED is available, therefore that's the only possible default, any other would lead to zipfile raising errors out of the box for some people. It would probably have been a better idea to just not use a default, and require the compression method to always be specified, at least when opening zipfiles with modes other than "r".
- duskwuff 6y ago
  TBH, Python's zipfile should probably just require zlib. Without that module, it's basically useless (can't open most ZIP files, can't emit compressed ZIP files).
  - Aachen 6y ago
    Actually, I've argued the opposite in the past. Any file that is of any significant size is already compressed: images, audio, video, gifs, PDFs, game map/save data... even documents (and by extension, spreadsheets and presentations) are compressed these days (because they're... zips! It's zips all the way down). Compression in your zip archive implementation is unnecessary complexity at this point. Compressing archives most only makes sense if you have some huge (many many millions of lines) set of source code because that's a lot of data, more than one file that you want to compress, and low entropy in those files. Only in that combination does zip with compression make sense over uncompressed archives or compression without archive format. Of course, zip can shoehorn both functions that you usually don't need together into one format and we're usually not CPU-bound anyway so I totally see why you might as well always apply a small amount of compression, but I would beg to differ if you say "it's basically useless to have zip without compression support". There are more cases where zip with compression is completely redundant than cases where it helps you.
    - duskwuff 6y ago

Jiocus 6y ago

This isn't an oddity or issue with Python zip creation, it's expected behaviour as per ISO/IEC WD 21320-1[1]. > The compression method shall be either 0 (“stored”) or 8(“deflated”). The format does not prescribe a compression algorithm (many could be available for use). It's merely a container. [1] ISO/IEC WD 21320-1, Document Container File - Part 1: Core - https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.7181&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.39...

bawolff 6y ago
What defaults python chooses is an oddity within python.
- Jiocus 6y ago
  Using defaults aimed at compatability with other zip archivers, today and tomorrow, doesn't strike me as odd.
  - dylan604 6y ago
    If you can't deflate existing ZIP files because no libzip, then you're not maintaining compatibility with anything really

masklinn 6y ago

Yes, it's clearly documented but very much unexpected the first time 'round. The reason for the default is probably that python can be built with none of zlib, bzip2 or lzma support: > If ZIP_DEFLATED, ZIP_BZIP2 or ZIP_LZMA is specified but the corresponding module (zlib, bz2 or lzma) is not available, RuntimeError is raised. in which case only STORED is available, therefore that's the only possible default, any other would lead to zipfile raising errors out of the box for some people. It would probably have been a better idea to just not use a default, and require the compression method to always be specified, at least when opening zipfiles with modes other than "r".

duskwuff 6y ago
TBH, Python's zipfile should probably just require zlib. Without that module, it's basically useless (can't open most ZIP files, can't emit compressed ZIP files).
- Aachen 6y ago
  Actually, I've argued the opposite in the past. Any file that is of any significant size is already compressed: images, audio, video, gifs, PDFs, game map/save data... even documents (and by extension, spreadsheets and presentations) are compressed these days (because they're... zips! It's zips all the way down). Compression in your zip archive implementation is unnecessary complexity at this point. Compressing archives most only makes sense if you have some huge (many many millions of lines) set of source code because that's a lot of data, more than one file that you want to compress, and low entropy in those files. Only in that combination does zip with compression make sense over uncompressed archives or compression without archive format. Of course, zip can shoehorn both functions that you usually don't need together into one format and we're usually not CPU-bound anyway so I totally see why you might as well always apply a small amount of compression, but I would beg to differ if you say "it's basically useless to have zip without compression support". There are more cases where zip with compression is completely redundant than cases where it helps you.
  - duskwuff 6y ago