7 ms·
Chrome Feature: ZSTD Content-Encoding
- tutfbhuf 3y agoFinally. I hope we get zstd into the Go standard library. It feels kind of strange that we have to fetch a third-party library for that.
- roywashere 3y agoI just contributed zstd decoding to httpx, my favourite python http library!
- fl0ki 3y agoOnly if it's backed by security-reviewed and fuzzed asm. Go's gzip implementation is slow enough that klauspost is worth importing, following that pattern, I would probably still use klauspost's zstd even if Go had a slow reference implementation in the standard library. What I'm really hoping for is something as seamless as gzhttp but for zstd. https://github.com/klauspost/compress/tree/master/gzhttp https://github.com/klauspost/compress/tree/master/gzhttp
- xnx 3y ago> Zstd is roughly three times faster than Brotli for decompression. Combined with zstd being faster at compression, this will result in faster page load times.
- deleted 3y ago[deleted]
- linsomniac 3y agoPart of my infrastructure uses compressed "dd" disc images. Last week I switched from using "xz" to "zstd" for them and "zstd -19" compresses almost as well as xz (1.8GB vs 1.7GB), but the threaded decompression is >10x faster (72s -> 6s). I made this change coincidentally, the day before the xz compromise.
- cogman10 3y agoI've been nothing but happy with ZSTD. Really good compression trade offs with ratios close to the best of LZMA. ZSTD 3 is nearly as fast as LZ4 while getting a much better ratio. Consiquentially, on my linux boxes I use either ZFS or btrfs and use zstd 3 as the default for the file system.
- dralley 3y agoNearly as fast to compress, but much slower to decompress. Zstd is pretty good on decompression but LZ4 is top-tier. Zstd is often "fast enough" in decompression for most cases, though.
- chronogram 3y agoIt's why I use LZ4 for ZRAM. Only at RAM speeds do LZ4's speeds matter.
- everybackdoor 3y ago[dead]
- wood_spirit 3y ago(JiaT75 contributed to zstd as well as xz)
- cipherboy 3y agoCitation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75 https://github.com/facebook/zstd/commits/dev/?author=jiaT75
- wood_spirit 3y agohttps://news.ycombinator.com/item?id=39873017 https://news.ycombinator.com/item?id=39873017
- dmw_ng 3y agoA very happy moment for Yann no doubt, he mentioned Google's competing compression efforts directly impeding his work during a corecursive podcast interview ( https://corecursive.com/data-compression-yann-collet/ https://corecursive.com/data-compression-yann-collet/ ) For the rest of us, another half-decade+ wait to get useful levels of compatibility especially on mobile
- jeroenhd 3y ago> For the rest of us, another half-decade+ wait to get useful levels of compatibility especially on mobile Chrome already ships it with their currently released browsers, so you get a significant amount of mobile traffic (about 42%) from Android devices supporting it already. I don't know what the current status of Blink on iOS is, but once that releases, you'll also get iOS users. WebKit has reacted positively to the proposal, though the Github issue documenting their position refers to an internal radar ticket so I have no idea what their timeline is. If you build for Chrome today, compatibility will only grow. Chrome doesn't do compression levels above a certain point but I doubt Safari and Firefox will be affected by a different limitation when support eventually lands. Plus, if your web server is configured well, you can have the gzip/brotli/zstd content available right next to each other, and just follow the browser's preference to pick a supported compressed asset. There really are no downsides here.
- olliej 3y agoIf you “build for chrome today” you create a chrome only monopoly exactly as web devs did with IE. Or you could build your site properly and have it work on more than a single browser. You do you.
- arccy 3y agothe web still works without content encoding, so you optimize for the common case but it still works for everyone
- jeroenhd 3y agoOther browsers will implement the feature soon. This isn't like webgpu or webusb, which just appeared before other browsers even reacted. If Firefox and Webkit do introduce compatibility issues, you can always correct those later, though I doubt there will be any.
- beeboobaa3 3y agoCan we expect xz support next?
- rafaelturk 3y agohahaha
- theandrewbailey 3y agoNo. LZMA-based formats like xz are too slow for web transfer speeds.
- CodesInChaos 3y agoIsn't LZMA decompression pretty fast? For static assets its slow compression doesn't matter.
- pixl97 3y agoThey are having some problem compiling the test suite in Chrome.
- alserio 3y agoIt would be nice if CompressionStreams would also get brotli/zstd support. I wonder if the js engine uses the same implementation of the browser
- donatj 3y agoWhile looking to see how it compares to brotli, I found an interesting conversation between a zstd contributor and the author of brotli https://news.ycombinator.com/item?id=19678985 https://news.ycombinator.com/item?id=19678985
- Ono-Sendai 3y agoI implemented this in my server (https://substrata.info/ https://substrata.info/). Works great. Results in smaller files than zlib/deflate. One thing to watch out for - compression levels >= 20 aren't handled by chrome clients.
- executesorder66 3y agoI am surprised all major browsers hadn't implemented this years ago. Zstd is way better than gzip and brotli.
- baybal2 3y ago[dead]
- Y_Y 3y agoHardly surprising when you consider gzip's DEFLATE was around in 1990 and developed by one guy as shareware and zstd was produced by a megacorp in 2015.
- lifthrasiir 3y agoZstandard was also mainly designed and produced by a single person, Yann Collet.
- dralley 3y agoYann also developed LZ4
- KingMob 3y agoYes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors https://github.com/facebook/zstd/graphs/contributors, 300+ other contributors have made 4500+ commits to the zstd repo. It's not quite as small-scale as 90's-style shareware was.
- btdmaster 3y agoIt would be good if zstd had a standardised a static web dictionary, like Brotli's https://www.rfc-editor.org/rfc/rfc7932#appendix-A https://www.rfc-editor.org/rfc/rfc7932#appendix-A. This would mean that the overhead for small files would be smaller, for boilerplate like <!DOCTYPE html>.
- swinglock 3y agoPossibly just use the one from Brotli since it's already standardized. If it's any good then the work is mostly already done, right?
- mjevans 3y agoThey want it as a standard option (pre-shipped with all decompressors). Not something they could incorporate into a customized fork / client that uses zstd with their own extensions.
- a_e_k 3y agoThe zstd API does allow you to supply your own initial dictionary, so there's no need to fork it to allow a browser implementation to use the brotli dictionary. Personally, as someone who doesn't work in web, I'm just as happy that zstd is flexible this way. For my applications, the brotli dictionary is pure overhead that bloats the library.
- mjevans 3y agoAgain, since there's confusion. They want _every zstd decompressor_ to __already have__ the dictionary in question so that it can be specified as part of the standard. E.G. 'instead of empty / an initial in file dictionary, use the standard dict #3' Such reference dictionary starts would be not be included in .zstd files, but would be shipped with the compressor source code.
- TillE 3y agoYeah zstd is nice on its own, but a proper dictionary will give you several times the compression ratio on top of that, plus super fast decompression. It's just absurdly good for specialized data.
- yread 3y agoThe second document has a link to a benchmark [1] by peazip which shows that 7zip is by far the best when you have a bit of time to compress (241s) but its the fastest at decompression and has the best ratio. Interesting. How is the support in languages? Are browsers planning to implement it as well? https://peazip.github.io/fast-compression-benchmark-brotli-zstandard.html https://peazip.github.io/fast-compression-benchmark-brotli-z...
- mik1998 3y ago7z is an archive format, it just uses LZMA2 compression I believe.
- arp242 3y agoWell it's also an LZMA implementation, and an implementation can be fast or slow, good or bad. From what I gather 7zip tends to be quite a bit faster than xz, since it's been worked on/optimized quite a bit more (I didn't test this myself to verify, but I've seen several people comment on that over the last few years).
- modeless 3y agoWill zstd be added to the compression streams API for use by JavaScript as well?
- lxgr 3y agoWho would have thought that Chrome can add new compression formats too as opposed to removing them! This is nice – but please, bring back JPEG XL too.
- OttoCoddo 3y agoSane choice, albeit delayed, as Zstandard has been the leading standard algorithm in the field for quite some time. I tested most of them for developing Pack [1] and Zstandard looked like the best alternative to the good old DEFLATE. The dictionary feature [2] will help design some new ways of getting small resources. [1] https://news.ycombinator.com/item?id=39793805 https://news.ycombinator.com/item?id=39793805 [2] https://facebook.github.io/zstd/zstd_manual.html#Chapter10 https://facebook.github.io/zstd/zstd_manual.html#Chapter10
- lemper 3y agoas usual, I am always in the skeptic camp when chrome people add features. there's an unexplainable suspicion that they do it for EEE strategy like web integrity api. remember folks, if you love open Web, you'd advocate to use non-chrome browser.
- hyperpape 3y agoThey're implementing a widely used compression standard created by a developer at a competing company. Even if Google is the absolute most evil company in the world, they will still make many smaller decisions every year that are just reasonable engineering. If you forget that, you end up going down a conspiracy rabbit hole and losing any ability to understand the world.
- kccqzy 3y agoOr you can look at whether Firefox will add the same feature. They do.
- cpeterso 3y agoThere is some recent development activity on Firefox's zstd bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1301878 https://bugzilla.mozilla.org/show_bug.cgi?id=1301878
- fastest963 3y agoWe have been live with zstd encoding for several weeks now and last week public traffic really started to use it. We're now at 50% brotli, 40% zstd, 9% gzip. Our payload is almost exclusively JavaScript and compression rates are 73% zstd (level 6), 72% brotli (level 4) and 65% gzip (level 4). However, for 95th percentile encoding speed zstd is 4.3ms, brotli is 2.1ms, and gzip is 0.1ms. At the 75th percentile they're all essentially the same. This is a Golang application and we use a cgo brotli library but a pure-go zstd library.
- saurik 3y agoAre you unable to cache the zstd-compressed JavaScript in your app server (so as to save the 4.3ms of CPU on every request)? If not, are you sure the 4.3ms of CPU costs less than the 1% of bandwidth? I saw a linked discussion here with a Brotli author talking to a zstd author that started making me stare into these questions more.
- fastest963 3y agoWe compile/build JS on the fly per-request so we wouldn't be able to cache the JS ahead of time. The difference between brotli and zstd is 2.2ms at the 95th percentile, at lower percentiles the differences are much smaller. This is still an area of active investigation/development for us. There are possibly content sizes where one is better than another. There are also probably tweaks on the server-side to improve the encoding time.
- eddd-ddde 3y agoUnrelated but I'm really curious what's the use case for sending custom js per request?
- fastest963 3y agoWe offer a lot of customization in our product so we can eliminate whole blocks or code if we know the customer doesn't have x turned on. We also have several geo-based products and so we know we don't need to send GDPR-related code to users in the USA for example. The goal is to ship the minimal amount of code to the end-user.
- fl0ki 3y agoI stan zstd hard. Hopefully now that browsers support it, HTTP middleware libraries will finally support it too. I already use it everywhere else.
- qingcharles 3y agoHere's the Microsoft .NET and ASP.NET feature tracking: Add support for Zstandard to System.IO.Compression: https://github.com/dotnet/runtime/issues/59591 https://github.com/dotnet/runtime/issues/59591 Support zstd Content-Encoding: https://github.com/dotnet/aspnetcore/issues/50643 https://github.com/dotnet/aspnetcore/issues/50643
- hinkley 3y agoAnother important variable is reverse proxies. You don’t necessarily need Rust or Node.js or Java to support zstd but you do need Traefic and nginx and haproxy to do so.
- qingcharles 3y agoThat's an excellent point. Frustratingly I use YARP, so I still need MS to implement zstd hah.
- 2OEH8eoCRo0 3y agoStupid question: Why do we keep adding things like this to the browser when they could be a library on the system?
- wmanley 3y agoI feel like I’m missing something here. How would having a library on the system help with serving zstd compressed assets?
- toast0 3y agoBrowsers benefit from having a deep integration into the http stack. You could build a browser around libcurl, but none of the majors are built like that.
- snvzz 3y agoFor "realtime" compression, wouldn't lz4 make more sense? Much lighter in both the server and the client.
- bastawhiz 3y agoLZ4 has quite poor compression, its advantage is speed. The difference between decompressing ten megs of data with LZ4 versus zstd is a completely negligible number when the difference in bytes sent over the wire is might approach a full megabyte.
- deleted 3y ago[deleted]
- BobbyTables2 3y agoThis may not age well…
- everybackdoor 3y ago[dead]