7 ms·
Yep, I've been advocating zstd since I started working in the field. It compresses and decompresses so much faster than xz and is much much more compact than gz
by croemer 4y ago
Yep, I've been advocating zstd since I started working in the field. It compresses and decompresses so much faster than xz and is much much more compact than gzip.
All public SARS-CoV-2 consensus sequences are ~300GB uncompressed. If you compress using gzip you end up with ~30GB. With xz/zstd you can get it down to ~2GB. However xz takes ~40min to uncompress, whereas zstd can do it in ~8min.
- mort96 4y ago[flagged]
- Underphil 4y agoWeird take seeing as it's licensed under GPL.
- mort96 4y ago[flagged]
- chungy 4y agoYou're free to march on with your weird bias. The rest of us can benefit.
- mort96 4y agoCan't "the rest of us" make other good compression formats?
- mikepurvis 4y agoBecause large companies are willing to pay people market salaries to work on this stuff, and then give it away? Isn't that what we want, a model where free software development is properly compensated?
- mort96 4y agoFOSS funding is a major problem. "Get a job at Facebook and make products for them" isn't a great solution to the FOSS funding problem.
- chungy 4y agoIt sounds like a perfect solution if you ask me. Free software gets made, developer gets paid.
- mort96 4y agoFree software gets made, to the degree that it benefits Facebook, in the ways which benefit Facebook. Software development in areas which don't benefit Facebook suffers. Facebook gets people to use their software, and therefore gains influence in places where they don't belong. Yes, the developer gets paid, but it's not a good solution that's healthy for an independent free software movement.
- chungy 4y agoIt should be no surprise that Facebook works on projects that benefit Facebook. By making it free software, it also happens to benefit the wider world at the same time. Win-win. Is your problem with the idea that developers are getting paid at all, or just that this instance is at Facebook?
- mort96 4y agoMy main issue is that I would rather not use Facebook products, and I would rather we don't create standards which rely on Facebook products. The FOSS funding issue I more of a side-point which I didn't even bring up, but I don't see "get hired by Facebook if you want to work on FOSS" to be a healthy solution to the FOSS funding problem, we need a way to ensure funding for FOSS developers who don't work for Facebook. But that's not strictly related to the zstd thing specifically.
- cjbgkagh 4y agoA lot of consumer surplus is derived from companies trying to undermine their competitors by commoditizing non-core competency. To not take advantage of that is to have to bad parts of capitalism without the good parts.
- mort96 4y agoStandardising on Facebook products is giving Facebook an undue influence in areas they don't belong. That's not one of the good parts of capitalism.
- sebzim4500 4y agoI guess, but when there is already a great format which is GPL licenced what would be the point?
- nibbleshifter 4y agoMake one that matches zstd performance then, and get back to us. Doing this kind of thing requires a shitload of time and money.
- felixhandte 4y agoIndeed--at this point zstd is the product of something like 25 man-years of improvements and optimizations.
- unethical_ban 4y agoIt's completely open-source, general purpose software that is essentially static (lower chance of drama or ideological/management concerns ala golang). It would be like avoiding C because it was developed by ATT.
- mort96 4y agoIf AT&T still controlled C compilers and the C standard and and contributions to the C compiler everyone uses was gated behind AT&T I might strongly have considered avoiding C. If a good non-Facebook implementation of ZSTD shows up, I might strongly consider using (that implementation of) ZSTD.
- unethical_ban 4y agoI simply don't think it matters. This isn't golang/react. Features, bugs and upgrades don't move with nearly the velocity of something like a large, opinionated library or language. But I rest my case.
- felixhandte 4y ago1. There are several implementations of Zstandard other than the "reference" libzstd that we maintain, tagged as "Ports" in the table here [0]. You are of course welcome to create or fork your own. We will happily take a PR adding yours to the table. 2. The format spec is an RFC [1]. It's frozen. We're not going to pull the rug out from under you somehow (and why would we??). 3. I think we do a pretty good job accepting contributions. But again, if we aren't, you can always fork it. All in all, we've worked pretty hard to make zstd universal--portable, stable, flexible, robust, etc. I guess I don't really have a point... I guess I hope you reconsider your rejection of zstd? It's pretty cool. [0] https://facebook.github.io/zstd/ https://facebook.github.io/zstd/ [1] https://www.rfc-editor.org/rfc/rfc8878.txt https://www.rfc-editor.org/rfc/rfc8878.txt
- mort96 4y agoThis is going to sound like I'm moving the goal posts, because, well, I am. I apologize. But what I notice with all those ports are that they're all in languages other than C (or C++), which means they're pretty much confined to their language's ecosystem; there is no port that's prepared to take the place of the Facebook zstd library. If I want to write, say, Common Lisp bindings, I'm not going to write them against the Java port. The one which comes the closest is probably the Rust port, since Rust is establishing itself as a core low-level language which people write bindings to, but the Rust port is only a decompressor. I'm also concerned with the term "port". I don't know exactly what you put in the term, but to me, it sounds like someone basically translated the C code directly to other languages. What I would be looking for is more like a "clean re-implementation" by an independent group of people who properly understand all the algorithms which are used and where all the code is written from scratch without the Facebook code as a "crutch". This would ensure that whatever Facebook does, there will be a group of people that is capable of fixing tricky bugs in the algorithms or implementing further optimizations. Basically, I would want the alternative implementation to be as unrelated to Facebook's libzstd as Clang is to GCC. All of that said though, from a purely technical perspective, zstd seems like a truly great piece of engineering. Kudos. Though I will never understand what drives great developers to dedicate their life to producing value for Facebook.
- nmlt 4y agoZstd isn’t a product. Facebook isn’t getting anything out of anybody using it. So it isn’t a product.
- belthesar 4y agoI have some internal conflict about zstd. On the one hand, it's truly become the superior general purpose compression codec, and I hate to look a gift horse in the mouth there. Using it for everything from data-at-rest compression on filesystems and zswap to channel compression with rsync, it truly works wonders. And yet, I can't really deny how absolutely at odds I am with the business model and actions of the company that has sponsored its development. Personally, I've resolved that we should extract as much good from things I perceive as evil. I've stopped using Facebook platforms myself, so I'm doing as much as I can to reduce the amount of money I generate for the platform as I can. That's about the most "voting with my dollars" I can do. However, because zstd has been such a net good, I will continue to use it in projects going forward for compression needs.
- FullyFunctional 4y agoPetty perhaps, but I'm really annoyed that he didn't follow the celebrated command line tool tradition of compress, gzip, xz, lzip, bzip2, etc. I always have to remember that zstd behaves differently and you have to take extra steps to achieve the same thing.
- felixhandte 4y agoWhat differences do you have in mind? Zstd is largely compatible as a drop-in replacement for those commands. There are differences, but it's our goal to minimize them.
- FullyFunctional 4y agoDid you try it? compress/gzip/bzip2/xz deletes the source file whereas zstd leaves it behind. This applies both to compression and uncompression.
- coldpie 4y agoI'm a Facebook hater too, but I can't think of any even theoretical concerns around zstd. It's freely licensed, no patent concerns, it's straight up free software. You're not giving them money or supporting their ad model by using it. It's not like Chrome where Google has an interest in making web browsing suck. What theoretical path to harm is there to adopting it more widely?
- theandrewbailey 4y agoZstd doesn't use a login, it doesn't track you across the internet, it doesn't make shadow profiles on you, and it doesn't have ads or telemetry. Zstd is even part of the Linux kernel at this point[0], so it's safe to say that it isn't a Facebook product anymore. They can't pull the rug on zstd, since it's BSD/GPL licensed. Unless I'm missing something, nobody is feeding the beast by using zstd. There's lots of similar cool technologies made by other "evil" companies. I don't use ZFS, but its origins in Sun/Oracle isn't the reason I don't. [0] https://lore.kernel.org/lkml/20181109190304.8573-1-kilobyte@angband.pl/ https://lore.kernel.org/lkml/20181109190304.8573-1-kilobyte@...
- mort96 4y agoI don't use ZFS, and the Oracle origins is the reason why. I'm very worried that the best filesystem available is forever stuck with a hostile license, and that so many of the good filesystem engineers are contributing to Oracle's product rather than to a non-Oracle alternative. (EDIT: I know zstd is under a good license, so its situation isn't the same a ZFS's, I just commented on ZFS since you brought it up.) Anyways, I prefer to not use Facebook products.
- yjftsjthsd-h 4y ago> that so many of the good filesystem engineers are contributing to Oracle's product rather than to a non-Oracle alternative. OpenZFS isn't Oracle's product; it certainly includes a great deal of Sun/Oracle code, but Oracle themselves can't use the OpenZFS commits without honoring the CDDL license, which to my knowledge they've not done.
- mort96 4y agoFair, but all the Oracle code in OpenZFS precludes OpenZFS from ever re-licensing under a less hostile license.
- Annatar 4y agoZFS was made by Sun Microsystems, which was a dear, awesome company that educated, fed, clothed and put bread on the table for an entire generation of computer professionals outside of Sun Microsystems, not to mention paid millions out of their own coffers to open source a reference implementation of a System V Release 4.0 UNIX. And to top it all off, Sun Microsystems made awesome hardware. Sun kicked butt had fun didn't cheat loved their customers changed computing forever.
- 1vuio0pswjnm7 4y agoAs the copyright holder Meta, its affiliates (e.g., Facebook) or their successors, can change the license for zstd at any time. It is a BSD license, not GPL. https://raw.githubusercontent.com/facebook/zstd/dev/LICENSE https://raw.githubusercontent.com/facebook/zstd/dev/LICENSE
- croemer 4y ago> As the copyright holder Facebook, or its successors, can change the license for zstd at any time. This is wrong? Once something is GPL, it's always GPL. https://softwareengineering.stackexchange.com/a/98777/347066 https://softwareengineering.stackexchange.com/a/98777/347066
- mort96 4y agoIt's a bit more complicated than that. Yes, the code that's out right now under the GPL will always be licensed under the GPL. But Facebook can at any time change the license of the zstd project, meaning all future releases will be under a different license. There will be an opportunity to fork the code base pre license change, but those forks are always risky and there's a very large change that the ecosystem won't end up with a de facto standard zstd fork.
- chungy 4y agoI wonder how well a zstd dictionary will work against these kinds of data sets.
- nullc 4y agoThey're big enough that it shouldn't help. Dictionaries are useful when you're compressing tiny files such that the overhead of the compressor learning the statistics from the input is significant. Basically you hope to save a fraction of the size of the dictionary from each file. From a 2GB file that's negligible. From a 1000 byte file it could be gigantic.
- croemer 4y agoUsing a custom dictionary works well if you want to compress single consensus sequences (30kbp), say before putting them into an sqlite db. Then you can get each sequence down to ~200 bytes. If you compress a large dataset, there's no point in using a custom dictionary, the auto-embedded one works just fine.
- TheGoodBarn 4y agoCurious, how long would Gzip take to uncompress? Is it faster since it is less compact / efficient on the compression side?
- wongarsu 4y agoDecompression speed is one of zstd's biggest advantages (in its class, lz4 is obviously faster in return for worse compression ratios). Zstd decompression is generally at least factor 3 faster than gzip.
- pletnes 4y agoSince the gzip’ed file is larger, it might well be slower since more IO is required. Depends on the case in question, obviously.
- andrewf 4y agoThe bitstream format for Gzip files (PKZIP's "deflate") was established in 1993. In comparison, the much newer Zstd doesn't represent a different tradeoff between compression time / decompression time / size; it's just better overall. There's a nice graph of the different time/compression tradeoffs both can achieve about 1/3rd of the way into https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/ https://engineering.fb.com/2016/08/31/core-data/smaller-and-...
- actually_a_dog 4y agoTo put that in perspective, consider that a large portion of the file is literally a nucleotide sequence, e.g. "GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACC" and that the rest of the file format is also fairly repetitive. If we just imagine only the nucleotide sequence, one way to compress that would be to take advantage of going from an 8 bit alphabet (ASCII encoding) to a 4 bit alphabet (CGAT). Naively, you'd expect ~50% compression from that alone, and be left with a bit stream that's still compressible.
- bombcar 4y agoYeah, it would seem to me (as an uneducated moron looking from the outside) that if you have domain knowledge about what is being compressed, a specialized compression algorithm would be much better overall than a generic one. Especially if major portions of the files are repeated.
- robert-boehnke 4y agoWouldn't you only need two bits to encode C, G, A, and T?
- dekhn 4y agoYes, two bits (assuming you only need to encode those four bases; unfortunately, biological reality requires FASTQ to encode other nucleotide values). There's even a format from UCSC called "2bit" (https://genome.ucsc.edu/goldenPath/help/twoBit.html https://genome.ucsc.edu/goldenPath/help/twoBit.html)
- kkielhofner 4y agoSame. If nothing else zstd universally supports compression and decompression with threading across any number of cores. It will even adjust threads dynamically based on the next limiting factor (I/O speed). Meanwhile it's also generally faster and provides significantly better compression ratios (as you describe). In my mind "always threading, everywhere" is reason enough to always use zstd. Sure there are implementations for other compression algorithms that support threading but the reference zstd implementation from Facebook always does, everywhere.
- Annatar 4y ago[dead]
- codesnik 4y agohm. Surely SARS-CoV-2 RNA sequence for one virus itself isn't 300gb? does it contain multiple variants or unprocessed and unjoined fragments?
- jacquesm 4y ago"All public SARS-CoV-2 consensus sequences" That seems clear enough.
- xboxnolifes 4y agoIt's actually ambiguous. It could mean that each sequence is 300GB, or that combined they total 300GB.
- thamer 4y agoThe 300GB is for "all public sequences", as explained. But to answer your question about "one virus", the initial isolate of SARS-CoV-2 from Wuhan, China has a genome containing 29,903 nucleotides. You can see the full raw sequence by clicking on "FASTA" on this page: https://www.ncbi.nlm.nih.gov/nuccore/1798174254 https://www.ncbi.nlm.nih.gov/nuccore/1798174254
- attractivechaos 4y agoFor compressing similar genomes, specialized tools will do better. AGC [1] can compress 620k SARS-CoV-2 genomes (19 GB uncompressed) into tens of MB. More importantly, AGC allows you to extract individual sequences without decompressing the entire file. In addition, zstd/xz wouldn't have a big enough dictionary to compress multiple human genomes efficiently. AGC can compress 300 GB human assemblies to 1.5 GB, a 200-fold compression ratio. I agree for sequence reads, zstd is better than gzip in almost every aspect. The only problem is that zstd is not as widely available. Many Linux distros don't ship zstd by default but on managed clusters, common users don't have the root permission to install zstd system-wide. It is worth mentioning that many mainstream bioinformatics tools can read FASTQ over pipes. You can use something like mytool <(zstd -dc in.fastq.zst) > output.txt # bash; also works on multiple zstd input files zstd -dc in.fastq.zst | mytool - > output.txt # general; for multiple input files, use mkfifo Those who prefer zstd can use it now. [1] https://www.biorxiv.org/content/10.1101/2022.04.07.487441v1 https://www.biorxiv.org/content/10.1101/2022.04.07.487441v1
- michaelbarton 4y agoUnfortunately some bioinformatics tools require multiple passes over the FASTQ. Spades is an example of this.
- attractivechaos 4y agoFair enough. For the relatively few tools that don’t support streaming, you may generate temporary plain files in your pipeline. Spades itself is a pipeline and writes temporary files. It is anyway not realistic to expect wide adoption of zstd in the near future. If you want to save storage cost now, you have to take some trade off.
- CreRecombinase 4y agoI've never seen a 15x difference in density between gzip and zstd on genomic data in an apples-to-apples comparison; I'd be really curious to learn more about how you achieved that.