7 ms·
Zstandard – Fast real-time compression algorithm
- jtakkala 4y agoHere’s a recent talk by Yann Collet at Stanford which leads into an interesting Q&A about his development of Zstandard: https://youtu.be/gZikN5hhlxA https://youtu.be/gZikN5hhlxA
- renewiltord 4y agoWe have a delta-encoded binary format for trade logs that we use and zstd still does a pretty good job on them for the last bit. It's just a pity that storage providers won't implement transparent compression. We store our logs on S3 and on-prem and it's just annoying that we have to store them compressed on S3. Would be cool to just store without the compression stage and have S3 do that bit. I haven't tried btrfs with the transparent zstd compression, but perhaps worth a shot.
- deleted 4y ago[deleted]
- ericpauley 4y agoOne upshot of compressing outside s3 is that you save on data transfer. In theory I suppose the compressed data could be sent over to the client and decompressed but then it’s basically the same thing.
- skrtskrt 4y agoit’s also a conflation of two things that are each much more sustainable if kept independent. You don’t want a particular compression pattern becoming part of the S3 standard IMO
- fnordpiglet 4y agoYeah I’ve suggested to the s3 leadery folks many times to provide some sort of managed compression and other sorts of codec interfaces. They could do other things like convert between structured encodings as well. Unfortunately it’s never gone anywhere.
- coredog64 4y agoIsn’t that what S3 object lambda access points are for? https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-create.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-c...
- fnordpiglet 4y agoIt’s too expensive. The thing is s3 has a ton of compute sitting around. The disks have a full system behind them on the storage nodes. A big question always was how can we do more with the edge storage? S3 select was one effort (although I don’t know if ever got pushed to the edge, it originally ran in the front end servers). But I always figured compression was a real obvious s3 edge storage operation, along with as I said format conversions.
- abotsis 4y agoI still wish s3 object select supported zstd on json.
- acdha 4y agoCould you use a Lambda to compress with zstd on write and S3 Object Lambda to decompress the stream seamlessly on GET (and update the content length on HEAD)?
- dantillberg 4y agoOne reason this could be messy to add is that the S3 API includes support for reading arbitrary byte ranges within a file. There's an inherent conflict between that API and support for transparent compression. Compression performs best when compressing the entire file from beginning to end; when decompressing you'd similarly decompress from the beginning. To support byte range reads, the data could be compressed in short blocks, but those blocks will not compress as well.
- 323 4y agozstd is quite slow at higher compression levels, if S3 would compress who pays the compute? Maybe they could offer it as an extra, and spawn EC2 instances as needed and bill you.
- broodbucket 4y agoI compress things with zstd solely so that I can find out if anyone's trying to consume my stuff with ancient tooling.
- yunohn 4y agoWhat kind of files exactly? User oriented stuff is hard to migrate.
- dools 4y agoI can’t find the Weissman score on the page, anyone know what it is? 3.8?
- kurante 4y agoWeissman scores are effectively fictitious and meaningless for any real application. Why are you looking for one instead of benchmarking data?
- khuey 4y agoYou're responding to a low-effort joke based on a plot line in HBO's Silicon Valley TV show.
- kurante 4y agoI'm aware of the reference, but I'd rather try and inform on the off-hand chance that they're serious about the question. I figured that there's little harm in making sure, other than looking a little silly :)
- CharlesW 4y ago2.97. https://github.com/facebook/zstd/issues/1087#issuecomment-447953776 https://github.com/facebook/zstd/issues/1087#issuecomment-44...
- tiffanyh 4y ago> This repository represents the reference implementation, provided as an open-source dual BSD and GPLv2 licensed C library Dumb question, why dual license? Why not just provide it as BSD (the more permissive of the two)?
- 8ytecoder 4y agoJust venturing a guess - may be that libraries that link this would want the option of GPL whereas commercial/non-libre software might prefer BSD?
- loeg 4y agoGPL code can already link BSD libraries without an explicit GPL license option.
- actionfromafar 4y agoIsn’t BSD flavour with advertisement clause + GPL problematic? Specifically that GPL requires no further hindrance to distribution.
- loeg 4y agoApproximately no one uses the 4-clause BSD (the problematic one). The one Zstd uses has three clauses: https://github.com/facebook/zstd/blob/dev/LICENSE https://github.com/facebook/zstd/blob/dev/LICENSE https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22) https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_...
- johannes1234321 4y agoThere are derivates of the 4 clause BSD, like PHP License, which bave some relevant users. Considering Facebook has some historic use of PHP things (while rewritten to Hack, but that uses some PHP libraries) pure GPL might cause issues for them. Still doesn't explain why pure BSD won't work for them, though
- fathyb 4y agoI needed to compress a TCP stream [0] for a block-storage project and I've been very impressed with zstd. Combined with io_uring, I'm getting less CPU usage and higher throughput to send and compress data than using socat to send uncompressed data. [0]: https://github.com/fathyb/zstd-proxy https://github.com/fathyb/zstd-proxy
- cecida 4y agoI'm always enormously impressed with efforts to move the dial when it comes to RTE, but slightly depressed when you see that it involves Facebook/Meta.
- sergiotapia 4y agoplease - out of all the tech giants the one constant is Meta delivers insane technology beyond their social networks. They are the goats of shipping cool shit.
- lytedev 4y agoSolid engineers work at all kinds of places and are capable of producing really cool stuff no matter where they are working. Don't judge something too much by its sponsors, IMO!
- stephc_int13 4y agoLZ4 and ZStd were started by Yann Collet (the main author) quite some time before he was hired by Facebook/Meta.
- fnordpiglet 4y agoDon’t worry, they track everything compressed with zstd and render appropriate advertisements to you using that data.
- nisa 4y agozstd rocks. I'm using it in zfs with zfs set compression=zstd-fast-1 for a recordsize=16k postgresql dataset and compressratio is about 2.0 and it's super fast. Doubling usable SSD space for as good as free is always neat.
- maximilianburke 4y agoI love everything about zstd. It compresses well, it compresses and decompresses fast. I wish browsers supported it for content encoding because it's a good deal faster and smaller than gzip, nevermind brotli.
- josephg 4y agoThe only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate is 714kb, and zstd has no native rust port presumably because of its size. I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include. The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.
- loeg 4y ago> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios. lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 There's also LZ4_HC.
- duskwuff 4y agoLZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.
- sn0wf1re 4y agoI feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.
- maximilianburke 4y agoExactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.
- gigatexal 4y agoGiven the table they have in their README I’d still choose LZ4 over ZSTD even if the ratio is 2.8 vs 2.1 or so for compression given the decompression is so much faster with LZ4 than with ZSTD. Also compressing is, as well.
- synergy20 4y agointeresting, it depends on which is more important: saving space or saving time. I never used LZ4 but I'm interested more in speed for some products, will use it in the future.
- fnordpiglet 4y agoIt’s not that straight forward. If you compress better before passing through a relatively low bandwidth channel you’re saving time with the slower but better compressing algorithm.
- punnerud 4y agoIn combination with SQLite to allow fast random access (query) and compression: https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd
- dang 4y agoDiscussed a few months ago: Show HN: Reduce SQLite database size by up to 80% with transparent compression - https://news.ycombinator.com/item?id=32303762 https://news.ycombinator.com/item?id=32303762 - Aug 2022 (90 comments) sqlite-zstd: Transparent dictionary-based row-level compression for SQLite - https://news.ycombinator.com/item?id=32297193 https://news.ycombinator.com/item?id=32297193 - July 2022 (1 comment)
- rkagerer 4y agoLooks awesome and I don't want to detract. But a few things from the documentation to be aware of: - Q: Is this library production ready? A: I wouldn't trust it with my data (yet). Make sure you have backups of everything. - usual zstd checksum is stripped from compressed data - columns you want to index can't be compressed - works best with rows containing less than ~30kB of data
- deleted 4y ago[deleted]
- charcircuit 4y agoFrom the readme Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases. Zstandard is considered safe for production environments.
- asveikau 4y agoThat's a statement about the block compression algorithm. The part that is claimed as not production ready is the third party sqlite integration. Skimming the code I can see why. It does some pretty gnarly things involving altering the schema of all your tables. Not to say it doesn't work or isn't good, but, it seems like it really does a lot of intrusive, potentially failure prone stuff.
- whalesalad 4y agoInteresting that the benchmarks were performed on a i7-6700K CPU which was released in 2015. Gonna err on the side of that just being outdated ... git blame shows that line has not changed in 5 years.
- jeffbee 4y agoThe 6th-generation Core CPU had BMI2 which is the only modern micro-optimization zstd uses.
- dang 4y agoRelated: How to get your backup to half of its size – ZSTD support - https://news.ycombinator.com/item?id=33644798 https://news.ycombinator.com/item?id=33644798 - Nov 2022 (51 comments) Zstd Compressed Debug Sections - https://news.ycombinator.com/item?id=32788343 https://news.ycombinator.com/item?id=32788343 - Sept 2022 (4 comments) Zstandard – Real-time data compression algorithm - https://news.ycombinator.com/item?id=32715933 https://news.ycombinator.com/item?id=32715933 - Sept 2022 (56 comments) AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage - https://news.ycombinator.com/item?id=32529412 https://news.ycombinator.com/item?id=32529412 - Aug 2022 (201 comments) Show HN: Reduce SQLite database size by up to 80% with transparent compression - https://news.ycombinator.com/item?id=32303762 https://news.ycombinator.com/item?id=32303762 - Aug 2022 (90 comments) The Bizarre Case of Zstd's Slow Performance on Arch Linux - https://news.ycombinator.com/item?id=31794958 https://news.ycombinator.com/item?id=31794958 - June 2022 (8 comments) Zstandard Worked Example - https://news.ycombinator.com/item?id=31411714 https://news.ycombinator.com/item?id=31411714 - May 2022 (50 comments) Zstandard RFC 8878 - https://news.ycombinator.com/item?id=29281360 https://news.ycombinator.com/item?id=29281360 - Nov 2021 (43 comments) Compressing JSON: Gzip vs. Zstd - https://news.ycombinator.com/item?id=27707834 https://news.ycombinator.com/item?id=27707834 - July 2021 (11 comments) Zstandard v1.5.0 - https://news.ycombinator.com/item?id=27155857 https://news.ycombinator.com/item?id=27155857 - May 2021 (36 comments) Zstandard v1.4.7 - https://news.ycombinator.com/item?id=25455314 https://news.ycombinator.com/item?id=25455314 - Dec 2020 (73 comments) Facebook/Zstandard – Fast real-time compression algorithm - https://news.ycombinator.com/item?id=24714854 https://news.ycombinator.com/item?id=24714854 - Oct 2020 (13 comments) Zstandard - https://news.ycombinator.com/item?id=24348394 https://news.ycombinator.com/item?id=24348394 - Sept 2020 (11 comments) Introduce ZSTD compression to ZFS - https://news.ycombinator.com/item?id=23210491 https://news.ycombinator.com/item?id=23210491 - May 2020 (38 comments) Now using Zstandard instead of xz for package compression - https://news.ycombinator.com/item?id=21958585 https://news.ycombinator.com/item?id=21958585 - Jan 2020 (153 comments) Zstandard v1.4.1 - https://news.ycombinator.com/item?id=20482424 https://news.ycombinator.com/item?id=20482424 - July 2019 (15 comments) Zstandard v1.4.0 - https://news.ycombinator.com/item?id=19678115 https://news.ycombinator.com/item?id=19678115 - April 2019 (67 comments) Improving compression at scale with Zstandard - https://news.ycombinator.com/item?id=18719592 https://news.ycombinator.com/item?id=18719592 - Dec 2018 (87 comments) Zstandard – Fast real-time compression algorithm - https://news.ycombinator.com/item?id=18530628 https://news.ycombinator.com/item?id=18530628 - Nov 2018 (51 comments) Zstandard Compression and the application/zstd Media Type - https://news.ycombinator.com/item?id=18133515 https://news.ycombinator.com/item?id=18133515 - Oct 2018 (19 comments) Zstandard v1.3.4 – faster everything - https://news.ycombinator.com/item?id=16718402 https://news.ycombinator.com/item?id=16718402 - March 2018 (33 comments) Zstandard – Real-time data compression algorithm - https://news.ycombinator.com/item?id=16226923 https://news.ycombinator.com/item?id=16226923 - Jan 2018 (96 comments) Zstandard: Fast real-time compression algorithm from Facebook Opensource - https://news.ycombinator.com/item?id=16222669 https://news.ycombinator.com/item?id=16222669 - Jan 2018 (2 comments) Better Compression with Zstandard - https://news.ycombinator.com/item?id=13813212 https://news.ycombinator.com/item?id=13813212 - March 2017 (62 comments) Comparing gzip, brotli and zstd - https://news.ycombinator.com/item?id=12634812 https://news.ycombinator.com/item?id=12634812 - Oct 2016 (1 comment) Smaller and faster data compression with Zstandard - https://news.ycombinator.com/item?id=12399804 https://news.ycombinator.com/item?id=12399804 - Aug 2016 (156 comments) Zstandard – Fast and efficient compression algorithm - https://news.ycombinator.com/item?id=8941955 https://news.ycombinator.com/item?id=8941955 - Jan 2015 (24 comments)
- mholt 4y agoCan someone help me understand why browsers don't support Zstandard yet? Caddy supports it natively, and it's quite good, if only browsers could decode it.
- Jarred 4y agoHere is a relevant link: https://bugs.chromium.org/p/chromium/issues/detail?id=1246971 https://bugs.chromium.org/p/chromium/issues/detail?id=124697... It sounds like they haven't decided whether to do it or not
- loeg 4y agoSee also https://datatracker.ietf.org/doc/html/rfc8878 https://datatracker.ietf.org/doc/html/rfc8878
- deleted 4y ago[deleted]
- Jarred 4y agoWe are planning to expose zstd as a runtime API in Bun. zstd is great
- deleted 4y ago[deleted]
- synergy20 4y agodid a quick check, zip on linux does not support lz4, zstd but 7z does, so I'm going to use 7z more, unfortunately 7z is still not universally installed on *nixes.
- loeg 4y agoWhy not just use zstd directly?
- dikei 4y agoYou are conflating compression algorithms with archive file formats. Use `tar` format and you can combine it with whatever compression utilities available such as `gzip`, `bzip2`, `lz4`, `zstd`, `xz`, etc. `tar` also preserves Unix permission. The downside is tar doesn't have an index for random-access, so you cannot do partial extraction quickly. Use `zip` format if you don't care about Unix permission, and want to do partial extraction. `Zip` technically support multiple compression algorithms, but is primarily only used with DEFLATE (the same one as `gzip`) for compatibility reason.
- TillE 4y agozstd with a well-trained dictionary is fantastic for a lot of applications, it's super fast, and can compress your data around 4-5x smaller than normal zstd (tested with JSON that has known keys and some known values).
- Const-me 4y agoThey have a nice table on that page: https://github.com/facebook/zstd#benchmarks https://github.com/facebook/zstd#benchmarks Looking at that table, I think LZ4 is a winner. The compression ratio is not too far, compression speed is slightly faster, decompression speed is significantly faster, the code is much simpler so the compiled binary is smaller, and the project is unrelated to Facebook.
- archseer 4y agoBoth ZSTD and LZ4 were built by the same author. They have slightly different target goals: https://news.ycombinator.com/item?id=23211804 https://news.ycombinator.com/item?id=23211804
- out_of_protocol 4y agoAre there any attempts to make zstd into standard general-purpose archive format? I think leap from .zip is big enough to bother
- Alifatisk 4y agoYou can install 7zip with zstandard!
- out_of_protocol 4y agoOverall good thing! except * standard 7z does not support zstandard * 7z file extension does not explain what's inside, so if you hand .7z file with zstd inside to someone, that person will blame you for unopenable archive well if only we could rebrand 7z-with-zstd as some new format, e.g. .zstd (no changes required) ...
- Alifatisk 4y agoI was thinking of https://community.chocolatey.org/packages/7zip-zstd/1.4.9.20 https://community.chocolatey.org/packages/7zip-zstd/1.4.9.20
- theandrewbailey 4y agoYou can use zstd in 7z files with Peazip: https://peazip.github.io/ https://peazip.github.io/
- out_of_protocol 4y agoRebrand "zstd in 7z" as separate format with separate extension and new format is born
- SergeAx 4y agoWe employed zstd to compress telemetry and logs from mobile devices. We reduced total bandwidth (not including CDN) by 70%, which is a big deal for mobile devices and battery life. CPU overhead was insignificant.