4 ms·

You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.

You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...

ncmncm 7y ago
Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.
- grizzles 7y ago
  Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.
  - ncmncm 7y ago
    OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.
  - etaioinshrdlu 7y ago
    I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.
    - skylanh 7y ago
      Version 6.3.3 from the Library of Congress - https://www.loc.gov/preservation/digital/formats/digformatspecs/APPNOTE%2820120901%29_Version_6.3.3.txt https://www.loc.gov/preservation/digital/formats/digformatsp... Writeup on zip file Appnotes: https://entropymine.wordpress.com/2019/08/22/survey-of-zip-appnote-documents/ https://entropymine.wordpress.com/2019/08/22/survey-of-zip-a... Actual PKWare AppNotes: https://support.pkware.com/display/PKZIP/Application+Note+Archives https://support.pkware.com/display/PKZIP/Application+Note+Ar... Section 4.4.5 is "compression method"
- anchpop 7y ago
  With zstd's training mode, why is there a need for custom compression algorithms for genomics? Are they able to compress more or compress faster?
- pnako 7y ago
  Nitpicking a bit, but a format that is still used ~90% of the time is not obsolete. Not optimal (anymore) is what you meant, I think.
  - ncmncm 7y ago
    I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator. It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?
    - ncmncm 7y ago
      *... with a gzip inflater.
- errantspark 7y ago
  You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction. From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?

ncmncm 7y ago

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

grizzles 7y ago
Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.
- ncmncm 7y ago
  OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.
- etaioinshrdlu 7y ago
  I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.
  - skylanh 7y ago
    Version 6.3.3 from the Library of Congress - https://www.loc.gov/preservation/digital/formats/digformatspecs/APPNOTE%2820120901%29_Version_6.3.3.txt https://www.loc.gov/preservation/digital/formats/digformatsp... Writeup on zip file Appnotes: https://entropymine.wordpress.com/2019/08/22/survey-of-zip-appnote-documents/ https://entropymine.wordpress.com/2019/08/22/survey-of-zip-a... Actual PKWare AppNotes: https://support.pkware.com/display/PKZIP/Application+Note+Archives https://support.pkware.com/display/PKZIP/Application+Note+Ar... Section 4.4.5 is "compression method"
anchpop 7y ago
With zstd's training mode, why is there a need for custom compression algorithms for genomics? Are they able to compress more or compress faster?
pnako 7y ago
Nitpicking a bit, but a format that is still used ~90% of the time is not obsolete. Not optimal (anymore) is what you meant, I think.
- ncmncm 7y ago
  I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator. It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?
  - ncmncm 7y ago
    *... with a gzip inflater.
errantspark 7y ago
You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction. From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?