3 ms·

>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Look this up: gnu parallel Also, with building on wh

>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Look this up: gnu parallel Also, with building on what the top reply to what you said, if "uncompressing is a bottleneck" and you didn't know you could uncompress multiple files at once suggests you should spend time learning about tools that exist before you jump and try to force your colleagues (who likely have much more experience than you) to adopt "new practices" that the js community will move on from in 6 months.

Symbiote 4y ago
TAR files can't be uncompressed in parallel. ZIP archives containing a single large file can't be uncompressed in parallel. The resulting CSV file can't be processed in parallel, as there might be quoted record delimiters. There are compressed file formats where a single file can be processed in parallel. Apache Avro and Parquet are examples I'm familiar with, but these handle columnar data (i.e. replacing CSV), not large numeric matrices etc.
- noobermin 4y ago
  Of course...I'm assuming (perhaps incorrectly) that they are decompressing many files sequentially without knowing that you can easily run gunzip in parallel using tools like gnu parallel. The OP says they get to "ten gigabytes in size." It does not take 15 minutes to decompress a 10GB file on a modern workstation, as they complain in a child comment, unless you decompress multiple files sequentially. I've routinely compressed and decompressed 100GB+ files on 5 year old workstation class machines and it takes at most 5-ish minutes for one direction, (stress on "at most").