9 ms·

Firefox's Optimized Zip Format

281 points by throw0101a 5y ago

eatonphil 5y ago
The oddest part of the zip format to me is the msdos time encoding. It is pretty unpopular today though (at least on Google) because it took me a while to find an explanation of it with code [0] when I was doing an educational project recently on building a zip reader from scratch. But I assume it was a more popular encoding at the time zip was designed. [0] https://groups.google.com/g/comp.os.msdos.programmer/c/ffAVUFN2NbA https://groups.google.com/g/comp.os.msdos.programmer/c/ffAVU...
- donio 5y ago
  It was the easiest to do since that's the format used by the DOS API calls for getting and setting the mtime.
- barosl 5y ago
  > Since the lower four bits are /2 Wow, so the second field of every timestamp in a ZIP file must be an even number? That's shocking to me! It seems that RAR and FAT32 share this problem. How could I not know this?
  - mappu 5y ago
    FAT32 timestamps are also stored as local time instead of UTC, so there's no way to know a real mtime at all, unless you have out-of-band knowledge. Just nonsense all round - - by modern standards. The really impressive thing is that our industry has managed to paper over most relics from The Olden Days to the point where this discovery is surprising!
  - aidenn0 5y ago
    You've clearly never rsync'd data to FAT before...
  - lifthrasiir 5y ago
    Because there are several extensions (extra fields) to ZIP that record the full modified date in addition to creation and access date if the OS supports them.
- palsecam 5y ago
  Another explication with (JS) code of the MS-DOS datetime encoding is at https://github.com/PaulCapron/pwa2uwp/blob/79ae42ea43e98eaf9db6ad4bc4059e8c0f31b18d/src/zip.js#L52 https://github.com/PaulCapron/pwa2uwp/blob/79ae42ea43e98eaf9... https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js in its entirety provides a simple intro, with code, to the ZIP format (archiving only, with no compression)
The_rationalist 5y ago
how about this -> https://github.com/ebiggers/libdeflate https://github.com/ebiggers/libdeflate https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-zstd/ https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-z...
jandrese 5y ago
> At the time optimized jar change broke antivirus scanners, which further sped up Firefox startup :) I'm pretty sure that counts as cheating. AV overhead is a real issue though. I was recently copying a directory full of small files from one fast SSD to another (larger) one. I started the transfer and Windows estimated about 8 hours to finish the transfer, and the overall progress seemed to be roughly on track. On a whim I tried temporarily turning off Microsoft's built-in antivirus scanning. Total transfer time dropped to about 40 minutes.
- cl0ckt0wer 5y ago
  the built in AV scans on close of file handle. moving this to a background thread will really speed things up. https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5T4/m/WAMC_Q-DCAAJ https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5...
  - cl0ckt0wer 5y ago
    Found the article that goes into depth on this issue https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ and the bug link: https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c94393dbf45ad417da257fe0 https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c9...
- jazzyjackson 5y ago
  Wild that windows feels it important to scan every file going from one local drive to another. my laptop has gotten a lot quieter after turning off virus protection, I wonder how much electricity is burned just idley checking for viruses. Edit: thanks for the replies, on second thought it is pretty tricky to know a file you’re writing already exists without trusting the process doing the copying
  - passerby1 5y ago
    Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something. Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks. But anyway, av-scanning is doing much good, but can be socially out-engineered in a single click, making it useless in most critical cases.
throw0101a 5y ago
For anyone wondering about the official documentation of the file format, PKWARE has: * https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT * Via: https://support.pkware.com/home/pkzip/developer-tools/appnote https://support.pkware.com/home/pkzip/developer-tools/appnot... 6.3.9 was released July 15, 2020.
- lulalalalala 5y ago
  And for design spec ambiguities read https://web.archive.org/web/20210722231119/https://games.greggman.com/game/zip-rant/ https://web.archive.org/web/20210722231119/https://games.gre...
- MontagFTB 5y ago
  I recently made a visualization template of ZIP file contents for HexFiend. The format is certainly wonky, and some writers (I’m looking at you, macOS) don’t get it quite right: https://github.com/HexFiend/HexFiend/blob/master/templates/Archives/ZIP.tcl https://github.com/HexFiend/HexFiend/blob/master/templates/A...
floatboth 5y ago
> renamed .jar files .ja so Microsoft System restore wouldn’t corrupt Firefox OHH that's why it's called omni.ja.. But why was it "jar" in the first place, for a thing that's not Java?
- ronsor 5y ago
  It's simply a jar of files, not a JAR (JAva aRchive) of files.
  - koolba 5y ago
    Though even the Java JAR file is itself just a zip file.
    - js2 5y ago
      As are apk, aar and ipa files among others. However, they aren’t just zip files as they must be in a specific layout, so they are a subset of the zip format.
      - easrng 5y ago
        You can make a valid APK/IPA/JAR polyglot, just put all the files in one folder, and use the jar tool to make it back into a zip, then rerun jarsigner because the APK has to be resigned after adding the Java and iOS files.
    - tadfisher 5y ago
      I believe the format specifies the META-INF/MANIFEST.MF path, but in practice the classloader and jar tools can deal with a missing manifest. Otherwise it is a zip, with all of its warts and non-reproducible default behavior. Did you know you can have multiple zip entries with the same path? Ask me how I found out.
      - koolba 5y ago
        That sounds like a recipe for a good time with ClassLoaders.
userbinator 5y ago
...and the reason they had to go and optimise it in the first place is because of how much of Firefox is actually not native code, but written in JavaScript! I discovered that the first time I wanted to modify something, and was also rather surprised that it's not been minified either.
- db48x 5y ago
  Minifying the javascript wouldn’t help anything, as it all gets byte–compiled and then cached. It wouldn’t even make the download smaller, since the download is compressed.
  - Jyaif 5y ago
    It would make decompression faster, and in practice it would also make the download smaller. The reason they were not minifying the JS is probably that at the time it made debugging and/or analysing stack traces easier.
pvorb 5y ago
It's a nitpick but did anyone else stumble over the term "pre-pandemic 2010"? What audience is the author writing for? Historians in a hundred years?
greggman3 5y ago
Making a file with that format would work but it would be easy to create a valid zip file that a reader that assumes that format would fail to read. You can easily have a Central Directory near the beginning of the file that the End Of Central Directory does not point to.
- usrusr 5y ago
  Or a zip file that would work for both format assumptions but has different content in each. Since both central directories could very well share file blocks you could be quite subtle with intended differences!
ksec 5y ago
Achievement unlocked! A submission I made on HN leads to a blog post from Taras Glek! On an optimisation that I actually remember when he was still at Mozilla. For those who dont know Taras was part of Project MemShrink and Project Snappy and lots of other pref issues. ( I think he also worked on profiling? ) A lot of those were what made Firefox useable before e10s and Quantum landed.
molszanski 5y ago
Pixar USDZ (fancy ZIP) files also have some tricks to improve performance of ZIP files https://graphics.pixar.com/usd/release/wp_usdz.html#layout https://graphics.pixar.com/usd/release/wp_usdz.html#layout Basically, important stuff first, no compression and 64 byte alignment

281 points by throw0101a 5y ago

The oddest part of the zip format to me is the msdos time encoding. It is pretty unpopular today though (at least on Google) because it took me a while to find an explanation of it with code [0] when I was doing an educational project recently on building a zip reader from scratch. But I assume it was a more popular encoding at the time zip was designed. [0] https://groups.google.com/g/comp.os.msdos.programmer/c/ffAVUFN2NbA https://groups.google.com/g/comp.os.msdos.programmer/c/ffAVU...

donio 5y ago
It was the easiest to do since that's the format used by the DOS API calls for getting and setting the mtime.
barosl 5y ago
> Since the lower four bits are /2 Wow, so the second field of every timestamp in a ZIP file must be an even number? That's shocking to me! It seems that RAR and FAT32 share this problem. How could I not know this?
- mappu 5y ago
  FAT32 timestamps are also stored as local time instead of UTC, so there's no way to know a real mtime at all, unless you have out-of-band knowledge. Just nonsense all round - - by modern standards. The really impressive thing is that our industry has managed to paper over most relics from The Olden Days to the point where this discovery is surprising!
- aidenn0 5y ago
  You've clearly never rsync'd data to FAT before...
- lifthrasiir 5y ago
  Because there are several extensions (extra fields) to ZIP that record the full modified date in addition to creation and access date if the OS supports them.
palsecam 5y ago
Another explication with (JS) code of the MS-DOS datetime encoding is at https://github.com/PaulCapron/pwa2uwp/blob/79ae42ea43e98eaf9db6ad4bc4059e8c0f31b18d/src/zip.js#L52 https://github.com/PaulCapron/pwa2uwp/blob/79ae42ea43e98eaf9... https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js in its entirety provides a simple intro, with code, to the ZIP format (archiving only, with no compression)

The_rationalist 5y ago

how about this -> https://github.com/ebiggers/libdeflate https://github.com/ebiggers/libdeflate https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-zstd/ https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-z...

jandrese 5y ago

> At the time optimized jar change broke antivirus scanners, which further sped up Firefox startup :) I'm pretty sure that counts as cheating. AV overhead is a real issue though. I was recently copying a directory full of small files from one fast SSD to another (larger) one. I started the transfer and Windows estimated about 8 hours to finish the transfer, and the overall progress seemed to be roughly on track. On a whim I tried temporarily turning off Microsoft's built-in antivirus scanning. Total transfer time dropped to about 40 minutes.

cl0ckt0wer 5y ago
the built in AV scans on close of file handle. moving this to a background thread will really speed things up. https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5T4/m/WAMC_Q-DCAAJ https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5...
- cl0ckt0wer 5y ago
  Found the article that goes into depth on this issue https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ and the bug link: https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c94393dbf45ad417da257fe0 https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c9...
jazzyjackson 5y ago
Wild that windows feels it important to scan every file going from one local drive to another. my laptop has gotten a lot quieter after turning off virus protection, I wonder how much electricity is burned just idley checking for viruses. Edit: thanks for the replies, on second thought it is pretty tricky to know a file you’re writing already exists without trusting the process doing the copying
- passerby1 5y ago
  Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something. Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks. But anyway, av-scanning is doing much good, but can be socially out-engineered in a single click, making it useless in most critical cases.

throw0101a 5y ago

For anyone wondering about the official documentation of the file format, PKWARE has: * https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT * Via: https://support.pkware.com/home/pkzip/developer-tools/appnote https://support.pkware.com/home/pkzip/developer-tools/appnot... 6.3.9 was released July 15, 2020.

lulalalalala 5y ago
And for design spec ambiguities read https://web.archive.org/web/20210722231119/https://games.greggman.com/game/zip-rant/ https://web.archive.org/web/20210722231119/https://games.gre...
MontagFTB 5y ago
I recently made a visualization template of ZIP file contents for HexFiend. The format is certainly wonky, and some writers (I’m looking at you, macOS) don’t get it quite right: https://github.com/HexFiend/HexFiend/blob/master/templates/Archives/ZIP.tcl https://github.com/HexFiend/HexFiend/blob/master/templates/A...

floatboth 5y ago

> renamed .jar files .ja so Microsoft System restore wouldn’t corrupt Firefox OHH that's why it's called omni.ja.. But why was it "jar" in the first place, for a thing that's not Java?

ronsor 5y ago
It's simply a jar of files, not a JAR (JAva aRchive) of files.
- koolba 5y ago
  Though even the Java JAR file is itself just a zip file.
  - js2 5y ago
    As are apk, aar and ipa files among others. However, they aren’t just zip files as they must be in a specific layout, so they are a subset of the zip format.
    - easrng 5y ago
      You can make a valid APK/IPA/JAR polyglot, just put all the files in one folder, and use the jar tool to make it back into a zip, then rerun jarsigner because the APK has to be resigned after adding the Java and iOS files.
  - tadfisher 5y ago
    I believe the format specifies the META-INF/MANIFEST.MF path, but in practice the classloader and jar tools can deal with a missing manifest. Otherwise it is a zip, with all of its warts and non-reproducible default behavior. Did you know you can have multiple zip entries with the same path? Ask me how I found out.
    - koolba 5y ago
      That sounds like a recipe for a good time with ClassLoaders.

userbinator 5y ago

...and the reason they had to go and optimise it in the first place is because of how much of Firefox is actually not native code, but written in JavaScript! I discovered that the first time I wanted to modify something, and was also rather surprised that it's not been minified either.

db48x 5y ago
Minifying the javascript wouldn’t help anything, as it all gets byte–compiled and then cached. It wouldn’t even make the download smaller, since the download is compressed.
- Jyaif 5y ago
  It would make decompression faster, and in practice it would also make the download smaller. The reason they were not minifying the JS is probably that at the time it made debugging and/or analysing stack traces easier.

pvorb 5y ago

It's a nitpick but did anyone else stumble over the term "pre-pandemic 2010"? What audience is the author writing for? Historians in a hundred years?

greggman3 5y ago

Making a file with that format would work but it would be easy to create a valid zip file that a reader that assumes that format would fail to read. You can easily have a Central Directory near the beginning of the file that the End Of Central Directory does not point to.

usrusr 5y ago
Or a zip file that would work for both format assumptions but has different content in each. Since both central directories could very well share file blocks you could be quite subtle with intended differences!

ksec 5y ago

Achievement unlocked! A submission I made on HN leads to a blog post from Taras Glek! On an optimisation that I actually remember when he was still at Mozilla. For those who dont know Taras was part of Project MemShrink and Project Snappy and lots of other pref issues. ( I think he also worked on profiling? ) A lot of those were what made Firefox useable before e10s and Quantum landed.

molszanski 5y ago

Pixar USDZ (fancy ZIP) files also have some tricks to improve performance of ZIP files https://graphics.pixar.com/usd/release/wp_usdz.html#layout https://graphics.pixar.com/usd/release/wp_usdz.html#layout Basically, important stuff first, no compression and 64 byte alignment