5 ms·
I poke around in this space periodically, but I've never found a compelling reason to move away from gzip.
by jgalt212 2y ago
I poke around in this space periodically, but I've never found a compelling reason to move away from gzip.
- loeg 2y agozstd is faster and provides better compression than gzip at every point on the curve. There is no reason to use gzip these days other than backwards compatibility.
- idoubtit 2y agoI've reached the same conclusion, and I've been using zstd unless I want extremely fast (de)compression with lz4. And even when I still have to use gzip (the format), the executables of libdeflate are fully compatible and noticeably faster than gzip/gunzip (the tool).
- jrockway 2y agoThis was on HN a while ago https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ and reaches the same conclusion. I am inclined to use Zstd even in cases where people would use lz4. Zstd is very fast.
- ThatGuyRaion 2y agozstd is a great setup, for sure, but the build system they use is patently awful. Can someone make an autoconf or hell, even CMake build system for them pleasee???
- loeg 2y agoJust install the binary package from whatever distro you use? Why do you need to build it? But if it matters, FreeBSD has a pretty trivial BSDmake build of it: https://github.com/freebsd/freebsd-src/blob/main/lib/libzstd/Makefile https://github.com/freebsd/freebsd-src/blob/main/lib/libzstd... https://github.com/freebsd/freebsd-src/blob/main/usr.bin/zstd/Makefile https://github.com/freebsd/freebsd-src/blob/main/usr.bin/zst... You could easily do something similar in GNU make or whatever without the dependencies on the FBSD build system. It's basically just the classic "cc -c a bunch of C files, then link them."
- jrockway 2y agoI suppose the point of open source software is to edit the source code and make your own version. If you can't build it, you can't do that.
- loeg 2y agoZstd has lots of contributors who have all figured out how to edit and build it.
- ThatGuyRaion 2y agoIt's basically a giant GCCism. Don't be condescending.
- loeg 2y agoI have no idea what you're talking about at this point.
- jrockway 2y agoYeah, not sure what the original commenter is complaining about. I did "git clone https://github.com/facebook/zstd https://github.com/facebook/zstd", "cd zstd", "make" and have a working binary. Doesn't get easier than that.
- ThatGuyRaion 2y agoNot everyone is using a standard tool chain or anything like that and oftentimes when I find projects with atypical build systems that give me trouble I tend to just move on to something else
- ThatGuyRaion 2y ago> Just install the binary package from whatever distro you use? Why do you need to build it? Cuz I'm actually a packaging maintainer for a couple different old operating systems. Regardless I thank you for being thankful even though I feel like you were being backhanded and demeaning on a number of levels.
- MawKKe 2y agoQuick glance of zstd github repo shows they do provide CMake build scripts?
- ThatGuyRaion 2y agoLast I touched it they only had glitchy make files and I don't like cmake particularly, I was just like "bespoke make files are fine if they're posix compliant, but these aren't"
- hinkley 2y agoDo zless, zcat and zgrep support zstd everywhere? And I mean everywhere? VMs? Alpine? FreeBSD? OSX? Openwrt? Nothing is shittier than sshing into a box that doesn’t understand half of your command line tricks. Or the clever shell script you just tested six ways to Sunday. It’s like fighting with your hands tied behind your back.
- sophiebits 2y ago> other than backwards compatibility
- hinkley 2y agoOther than that, Mrs Lincoln, how was the play?
- adastra22 2y agoYes, it's supported on all those platforms.
- skrebbel 2y ago> There is no reason to use gzip these days other than backwards compatibility And forward compatibility. The Lindy effect says gzip is likelier to be widely available across platforms and tools in the long term than zstd.
- t43562 2y ago> bzip2 -9 -k all.tar 981.78s user 9.77s system 95% cpu 8M memory 17:16.64 total > bzip3 -e -b 256 -j 12 all.tar 2713.81s user 16.28s system 634% cpu 18301M memory 7:10.10 total The memory usage is one reason: 8M vs 18301M
- Borg3 2y agoSame here. Storage is not that expensive so I do NOT care to squize every byte out of archive. Also, im more into retro, so portability and memory usage is more importand for me :)
- palaiologos 2y agoHi, tool author here! Regarding your first remark: high ratio data compression has its time and place, and I personally understand that to many people it is not very desirable. In a lot of scenarios something as plain and simple as LZ4 generally suffices. On the other hand, there is an unofficial (= unsupported) port of bzip3 to older (386+) machines that run MS-DOS6.22. I have prepared it for a retrocomputing meeting in Ontario that I attended a while back. Let me know what you think :). https://github.com/kspalaiologos/dev-urandom/blob/main/dos/BZIP3.EXE https://github.com/kspalaiologos/dev-urandom/blob/main/dos/B...
- Borg3 2y agoOh nice. Ill will look at it at some point to check things out and to see if I can compile it here and there out of the box :)
- alkh 2y agoI am a huge proponent of zstd after I learned about it on HN. I've recently had to compress a 12 Gb csv file. zstd took ~3 sec for compression and ~11 sec for decompression and got the file to ~1.1 Gb. Xz took ~3.5 min for compression(!) and the resulting file was ~740 Mb(I didn't measure the decompression time). I just realized that in most cases it's more efficient to use zstd, especially for file transfer. The majority of OS that people use either have it installed or can be trivially downloaded to, so I see no point in using gzip nowadays, unless it is mandated by API
- eqvinox 2y agoDid you confuse gzip and xz? You mention numbers from xz and then suddenly talk about gzip? These two are not related…
- alkh 2y agoHey, I didn't confuse them but I guess I should have been more specific. I've addressed 2 main points in my text. 1) In my opinion, for the vast majority of cases you don't need to be worried about comparability, as you can easily install better alternatives to gzip for platforms a lot of people use. 2) I think zstd can become a primary replacement for gzip due to its high speed of compression and good compression ratio, even when compared to such great algorithms like xz/lzma. Sacrificing some compression ratio for (de)compression speed is worth it, for me at least
- dist-epoch 2y agozstandard at level 21 with some tweaked parameters (larger dictionaries) will give you xz/lzma compression ratios, while still being faster.
- jgalt212 2y agoAfter reading this, I see lots of zstd fans around here, and with good reason. That being said, I think our shop will stick with gzip until zstd arrives in the Python Standard Library.