4 ms·

A few years ago I was working on optimizing scientific data delivery by switching our compression algorithms to use bzip2 instead of deflate64 for our zip files

A few years ago I was working on optimizing scientific data delivery by switching our compression algorithms to use bzip2 instead of deflate64 for our zip files. I checked the zip standard and saw that bzip2 compression was added in 2001 so clients have had 20 years to implement it. After changing our data delivery pipeline to bzip2 I patted myself on the back for finding an easy win which significantly reduced data size and increased compression speed. Fast forward a month or two later I get word that one of our clients is having problems uncompressing the data we sent them. It turns out that there were still windows 10 versions out in the wild that don’t support bzip2 zip files. I begrudgingly cursed windows and reverted my change, robbed of my small victory. It truly is amazing how terrible compression support has been in windows. I could compress a +100GB data archive in under a minute using pigz, but since we had to support windows it would take over 30 minutes to compress the file using deflate. I calculated that with the space savings of gzip we could ship a gzip binary embedded with the data in a smaller size than the deflate data on its own. Unfortunately it was a no-go to ship decompression binaries with embedded data, but I thought it would be a cool idea to use cosmopolitan libc to create a platform agnostic binary which when run uses gzip/bzip/lz4 to quickly decompress the data in a platform agnostic way. I imagine such binaries would be picked up by anti-virus but it would be cool to be able to compress data with the latest algorithms without needing the client to have the matching decompression libraries.

kwanbix 3y ago
Wouldn't have been easier to ask the partner to install 7zip?
- jonnycomputer 3y ago
  Ha! No, probably not. I don't mean to be flippant. But I recently had to convert Phillips proprietary DS2 audio format file to MP3, and purchased Phillips software to do that. And then financials came down hard because it wasn't on the approved software list.
  - kwanbix 3y ago
    I know what you mean, but still, I find weird that any company worth 2 cents doesn't have an official compression app. In my experience lots of them use 7zip (free/open source), WinRAR (paid) or even WinZIP (also paid). But I do know what a pain some IT shops can be.
- thecopy 3y ago
  Why on earth would you think that? Have you never worked with "partners" before?:)
- whywhywhywhy 3y ago
  The idea of talking someone through using 7zip gives me nightmares. The fact it's multiple programs, has it's own file managers etc. Nah not worth trying to explain how to use it to a non-tech person. I know it's a favorite program of a lot of people but personally I just think that's testament to how bad the situation is on windows.
  - BizarroLand 3y ago
    Multiple programs and has its own file managers? I've not seen this in practice. Sure, it has a clunky gui, but outside of that 7-zip has excellent windows 10 or less integration and works fine in windows 11 even though the jerkwad potentates at M$ decided to hide the usual explorer options behind a second panel.
    - KOTYAR 3y ago
      Sounds like you didn't have to explain to someone after 2020 where X button to close the window is located. I work with about 100 women on my job, only a few of them younger than 40.
      - BizarroLand 3y ago
        Do you typically say ageist and sexist things online? If you work IT support in any capacity, you should accept that your job is to solve problems for people who are doing things you either can't do or don't want to do. Their experience is outside of your wheelhouse, and you are there to make the technology transparent to their job function. If you can't do your job with a glad heart then find a different one. Have some humility.
- __turbobrew__ 3y ago
  It is easy to ask, it is nearly impossible for every partner to actually install it. Many were working in locked down environments where they would need to get some very unhelpful central IT team to approve new software and install it on workstations. Even if they could convince the central IT team to install 7zip they probably weren’t using configuration management to install it so there was a chance someone would forget to install 7zip while provisioning new workstations. Unfortunately, when you do not know who is going to consume the data you need to use the most common denominator which is still zip with deflate64.
  - arthur2e5 3y ago
    7z does have two advantages. The main one is that lack of support is obvious, so nobody wastes time thinking the file is broken. The other one is that you don't use bzip2, which compresses worse than lzma and decompresses slower.

kwanbix 3y ago

Wouldn't have been easier to ask the partner to install 7zip?

jonnycomputer 3y ago
Ha! No, probably not. I don't mean to be flippant. But I recently had to convert Phillips proprietary DS2 audio format file to MP3, and purchased Phillips software to do that. And then financials came down hard because it wasn't on the approved software list.
- kwanbix 3y ago
  I know what you mean, but still, I find weird that any company worth 2 cents doesn't have an official compression app. In my experience lots of them use 7zip (free/open source), WinRAR (paid) or even WinZIP (also paid). But I do know what a pain some IT shops can be.
thecopy 3y ago
Why on earth would you think that? Have you never worked with "partners" before?:)
whywhywhywhy 3y ago
The idea of talking someone through using 7zip gives me nightmares. The fact it's multiple programs, has it's own file managers etc. Nah not worth trying to explain how to use it to a non-tech person. I know it's a favorite program of a lot of people but personally I just think that's testament to how bad the situation is on windows.
- BizarroLand 3y ago
  Multiple programs and has its own file managers? I've not seen this in practice. Sure, it has a clunky gui, but outside of that 7-zip has excellent windows 10 or less integration and works fine in windows 11 even though the jerkwad potentates at M$ decided to hide the usual explorer options behind a second panel.
  - KOTYAR 3y ago
    Sounds like you didn't have to explain to someone after 2020 where X button to close the window is located. I work with about 100 women on my job, only a few of them younger than 40.
    - BizarroLand 3y ago
      Do you typically say ageist and sexist things online? If you work IT support in any capacity, you should accept that your job is to solve problems for people who are doing things you either can't do or don't want to do. Their experience is outside of your wheelhouse, and you are there to make the technology transparent to their job function. If you can't do your job with a glad heart then find a different one. Have some humility.
__turbobrew__ 3y ago
It is easy to ask, it is nearly impossible for every partner to actually install it. Many were working in locked down environments where they would need to get some very unhelpful central IT team to approve new software and install it on workstations. Even if they could convince the central IT team to install 7zip they probably weren’t using configuration management to install it so there was a chance someone would forget to install 7zip while provisioning new workstations. Unfortunately, when you do not know who is going to consume the data you need to use the most common denominator which is still zip with deflate64.
- arthur2e5 3y ago
  7z does have two advantages. The main one is that lack of support is obvious, so nobody wastes time thinking the file is broken. The other one is that you don't use bzip2, which compresses worse than lzma and decompresses slower.