6 ms·

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768?t=flIt5lJb6xXTWgLvoEVzAg&s=19 https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

gpderetta 3y ago
But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)
- viraptor 3y ago
  > compressing before tarring is probably better Not if the files are similar. If you're compressing the files separately you'll start with a clean state rather than reusing previous fragments. Compressing a BMP after a TXT may not be beneficial, but compressing 3 tar'ed TXTs is definitely better than doing them separately.
  - zweifuss 3y ago
    That is true and seems easily half the total size with small and similar files, but it also means you have to unpack the whole archive when you need the last file in a tarball. AFAIK the gzip command still cannot compress directory information and therefore needs tar in front of it if you want to retain a folder structure.
- cout 3y ago
  For many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio. Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compression settings with solid archive turned off). Tarring then compressing means small files can be aggregated into a single block, giving both good speed and compression ratio.
  - rhinoceraptor 3y ago
    Zstandard has a dictionary functionality which allows you to pre-train on sample data to achieve higher compression ratios and faster compression on large numbers of small files.
    - cout 3y ago
      Interesting - does it fit that automatically or is there a manual step?
    - sgtnoodle 3y ago
      Don't you then need to store that dictionary somewhere out of band? It seems like you would still need a tar-like format to manage that, at which point as an archive format it seems more complicated with a worse separation of concerns.
- ElectricalUnion 3y ago
  Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs). TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs
  - account42 3y ago
    The only point of using .zip is for maximum compatibility and you loose that if you use anything other than deflate or no compression. If you are going to use something else you might as well use a less wasteful and better defined archive container - or something entirely custom like most games do.
  - cout 3y ago
    93?
    - zweifuss 3y ago
      See 4.4.5. in https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.8.TXT https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.8.TX... and here: http://facebook.github.io/zstd/ http://facebook.github.io/zstd/
aigoochamna 3y ago
How big is that file... I have 2TB files compressed down to ~300GB and gunzip'ing them takes ~2-3 hours. Granted, that's still a long ass time, but not 2-3 days. If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P
- lathiat 3y ago
  bzip2 is much slower than gzip.
  - mobilio 3y ago
    Very bzip2 problem is that even decompression is slower.
  - aidenn0 3y ago
    bzip2 is not much slower at compression than gzip; gzip is far faster at decompression though. Either way since bzip2 is a block based compressor, parallelization is trivial, and parallel implementations started appearing about 20 years ago; pbzip2 is almost certainly in whichever package manager is in use for TFA.
Aachen 3y ago
I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.
- masklinn 3y ago
  With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.
  - dgacmu 3y ago
    You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.
    - Aachen 3y ago
      I've also noticed for some text documents (was it json? I don't remember) that bzip compresses significantly better than xz (and of course gzip/pigz). Not sure if I tested zstd with high/extreme settings at that time.
      - elteto 3y ago
        For some reason, bzip compresses text incredibly well. And it has for years, I remember noticing this almost 20 years ago.
        tehjoker 3y ago
        It uses the Burrows-Wheeler transform to place similar strings next to each other before using other compression tricks, so it usually does a bit better.
    - krzyk 3y ago
      Oh, that's interesting, I stopped using bzip2 at the time kernel sources started shipping in xz. Do you know if there are any tests showing which compressor is better (compression wise) for which data?
boomboomsubban 3y ago
Is the .07 of a core a margin of error or some kind of status report done on a different core?
- lionkor 3y ago
  iirc its because core usage is an average over time, and you can get weird situations where switching between cores (which happens frequently depending on the scheduler) causes the average usage of one core by the program to not be zero yet, and the usage on another core to already be 100. This is purely speculation though
sllabres 3y ago
There is a _parallel_ bzip2: http://compression.great-site.net/pbzip2/ http://compression.great-site.net/pbzip2/ which should solve the 'my cores are idle' issue.
gruturo 3y ago
With all respect to John Carmack (and it is really a lot of respect!) I'm surprised he seems unaware of pbzip2? It's a parallel implementation scales almost linearly with the amount of cores, and has been around since ~2010, so it's not yet old enough to drive, but anyone dealing with bzip2'ing large amounts of data should have discovered it long ago. And yes, use zstandard (or xz, where the default binary in your distro is already multithreaded) where you can.
- account42 3y ago
  > anyone dealing with bzip2'ing large amounts of data ... should really reconsider their choice of compression algorithm at this point.
jandrese 3y ago
Shame he didn't discover pbzip2 before starting that job.
- zargon 3y ago
  If it's less than 98% complete he could still stop it, start over and still finish sooner.
  - jandrese 3y ago
    I'd bet tar becomes the bottleneck before pbzip2 does on that multicore monster. It can be surprisingly slow and I don't think any version of tar uses more than one core.
res0nat0r 3y ago
He should just be using pbzip2 :) https://linux.die.net/man/1/pbzip2 https://linux.die.net/man/1/pbzip2
h2odragon 3y ago
Of course the problem there is that `tar` is outputting a single stream. You might, in similar situations, start multiple `tar` running on subsets of the input, which pipelines then become fully parallel again.
deleted 3y ago
[deleted]
stjohnswarts 3y ago
There are several programs to solve this problem. He just wanted to complain and write something built around that.

gpderetta 3y ago

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

viraptor 3y ago
> compressing before tarring is probably better Not if the files are similar. If you're compressing the files separately you'll start with a clean state rather than reusing previous fragments. Compressing a BMP after a TXT may not be beneficial, but compressing 3 tar'ed TXTs is definitely better than doing them separately.
- zweifuss 3y ago
  That is true and seems easily half the total size with small and similar files, but it also means you have to unpack the whole archive when you need the last file in a tarball. AFAIK the gzip command still cannot compress directory information and therefore needs tar in front of it if you want to retain a folder structure.
cout 3y ago
For many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio. Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compression settings with solid archive turned off). Tarring then compressing means small files can be aggregated into a single block, giving both good speed and compression ratio.
- rhinoceraptor 3y ago
  Zstandard has a dictionary functionality which allows you to pre-train on sample data to achieve higher compression ratios and faster compression on large numbers of small files.
  - cout 3y ago
    Interesting - does it fit that automatically or is there a manual step?
  - sgtnoodle 3y ago
    Don't you then need to store that dictionary somewhere out of band? It seems like you would still need a tar-like format to manage that, at which point as an archive format it seems more complicated with a worse separation of concerns.
ElectricalUnion 3y ago
Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs). TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs
- account42 3y ago
  The only point of using .zip is for maximum compatibility and you loose that if you use anything other than deflate or no compression. If you are going to use something else you might as well use a less wasteful and better defined archive container - or something entirely custom like most games do.
- cout 3y ago
  93?
  - zweifuss 3y ago
    See 4.4.5. in https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.8.TXT https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.8.TX... and here: http://facebook.github.io/zstd/ http://facebook.github.io/zstd/

aigoochamna 3y ago

How big is that file... I have 2TB files compressed down to ~300GB and gunzip'ing them takes ~2-3 hours. Granted, that's still a long ass time, but not 2-3 days. If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P

lathiat 3y ago
bzip2 is much slower than gzip.
- mobilio 3y ago
  Very bzip2 problem is that even decompression is slower.
- aidenn0 3y ago
  bzip2 is not much slower at compression than gzip; gzip is far faster at decompression though. Either way since bzip2 is a block based compressor, parallelization is trivial, and parallel implementations started appearing about 20 years ago; pbzip2 is almost certainly in whichever package manager is in use for TFA.

Aachen 3y ago

I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.

masklinn 3y ago
With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.
- dgacmu 3y ago
  You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.
  - Aachen 3y ago
    I've also noticed for some text documents (was it json? I don't remember) that bzip compresses significantly better than xz (and of course gzip/pigz). Not sure if I tested zstd with high/extreme settings at that time.
    - elteto 3y ago
      For some reason, bzip compresses text incredibly well. And it has for years, I remember noticing this almost 20 years ago.
      - tehjoker 3y ago
        It uses the Burrows-Wheeler transform to place similar strings next to each other before using other compression tricks, so it usually does a bit better.
  - krzyk 3y ago
    Oh, that's interesting, I stopped using bzip2 at the time kernel sources started shipping in xz. Do you know if there are any tests showing which compressor is better (compression wise) for which data?

boomboomsubban 3y ago

Is the .07 of a core a margin of error or some kind of status report done on a different core?

lionkor 3y ago
iirc its because core usage is an average over time, and you can get weird situations where switching between cores (which happens frequently depending on the scheduler) causes the average usage of one core by the program to not be zero yet, and the usage on another core to already be 100. This is purely speculation though

sllabres 3y ago

There is a _parallel_ bzip2: http://compression.great-site.net/pbzip2/ http://compression.great-site.net/pbzip2/ which should solve the 'my cores are idle' issue.

gruturo 3y ago

With all respect to John Carmack (and it is really a lot of respect!) I'm surprised he seems unaware of pbzip2? It's a parallel implementation scales almost linearly with the amount of cores, and has been around since ~2010, so it's not yet old enough to drive, but anyone dealing with bzip2'ing large amounts of data should have discovered it long ago. And yes, use zstandard (or xz, where the default binary in your distro is already multithreaded) where you can.

account42 3y ago
> anyone dealing with bzip2'ing large amounts of data ... should really reconsider their choice of compression algorithm at this point.

jandrese 3y ago

Shame he didn't discover pbzip2 before starting that job.

zargon 3y ago
If it's less than 98% complete he could still stop it, start over and still finish sooner.
- jandrese 3y ago
  I'd bet tar becomes the bottleneck before pbzip2 does on that multicore monster. It can be surprisingly slow and I don't think any version of tar uses more than one core.

res0nat0r 3y ago

He should just be using pbzip2 :) https://linux.die.net/man/1/pbzip2 https://linux.die.net/man/1/pbzip2

h2odragon 3y ago

Of course the problem there is that `tar` is outputting a single stream. You might, in similar situations, start multiple `tar` running on subsets of the input, which pipelines then become fully parallel again.

deleted 3y ago

[deleted]

stjohnswarts 3y ago

There are several programs to solve this problem. He just wanted to complain and write something built around that.