9 ms·
‘You can't use Brotli for dynamic content’ and other misconceptions
- mozumder 8y agoIs there a good Brotli library for Python?
- klodolph 8y agoHave you tried searching Pypi? https://brotlipy.readthedocs.io/en/latest/ https://brotlipy.readthedocs.io/en/latest/
- 11235813213455 8y agowhy not already compressing the bundles during build process?
- jbergstroem 8y agoIf you want to attempt building nginx with brotli support, let me save you 15 minutes: Google's original module is more or less abandoned. Look here instead: https://github.com/eustas/ngx_brotli https://github.com/eustas/ngx_brotli Another alternative is exploring h2o/caddy as a web server replacement.
- nailer 8y ago(deleted, I see your point now)
- icebraining 8y agoBut that PPA is using Google's repo as upstream; what's the point of the OS alerting you to updates, if there are none?
- wildpeaks 8y agoThis still relies on a special build of Nginx (from the PPA repo) instead of LTS Nginx, although I agree that it's already more convenient than unpackaged sources.
- reificator 8y ago> Brotli can be significantly smaller and compress faster than gzip I have a custom compression algorithm that is fairly competitive as well. It transforms the UK DVD first edition of the second season of Monty Python's Flying Circus to a bitwise identical copy of the third frame of the dancing baby gif. It is otherwise an identity transform. It can perform better than both brotli and gzip on both axes, I tested by compressing the UK DVD first edition of the second season of Monty Python's Flying Circus. It blew brotli and gzip out of the water. Or at least it did until I got a scratch on the disk. But I'm sure I can optimize for the new target as well. EDIT: I was trying to make the point that the title is so handwavey as to be meaningless. You could probably reverse them and show a similar result with different cases. The article does go on to link to a page[0] showing performance on the top 1000 sites, and discussing how to tune brotli to provide the best space/time tradeoffs. [0]: https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html https://blogs.akamai.com/2016/02/understanding-brotlis-poten...
- nailer 8y agoEdit: the title was changed again to it's current form(a moment ago it was 'brotli can be smaller and faster than gzip'). Thanks HN mods. Original post below. > Author here. This was submitted with: 'You can't use Brotli for dynamic content' and other horses**t (2018) Odd the title was changed, you can see from the timestamp I spent yesterday updating it for 2018 (brotli is more of a big deal now iOS users can use it). Also the original title highlighted the very specific point that a lot of what is written about brotli online is incorrect. Most HN users are grown ups and have handle a starred-out swear word. Oh well.
- reificator 8y agoI like your title much better than the title it was replaced with. The title as it stands now is meaningless, because it applies equally to nearly any pair of compression algorithms.
- paulie_a 8y agoI never understood the starring out thing. If the forum is inappropriate for that word, then using stars is also inappropriate.
- sctb 8y agoOK, we've put the title back closer to the original. “Horseshit” (we don't need to bowdlerize) is clickbait, so we've gone with “misconception”. Happy to update again if there's a better word.
- tzahola 8y ago>Modern websites in particular often have large JavaScript bundles - the front page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed! Mike, I’m afraid you have bigger problems than inefficient compression...
- nailer 8y agoCertSimple is a fairly complex web app with a bunch of ractive components that use webcrypto (requiring ASN1, PKI libraries etc) and URLs (using a WhatWG URL polyflll) which takes up a large chunk of the bundle. We use https://github.com/131/discify https://github.com/131/discify to analyse this and should be able to cut down a little further once the URL spec is more widely implemented.
- k__ 8y agoAre you doing code splitting? If not, this would be my first target for gains :)
- nailer 8y agoSeperate bundles for different parts of the app, but not doing lazy loading or anything like that. Might look at it again if/when ES6 modules ever become popular (I just built a smaller project (https://mikemaccana.com https://mikemaccana.com) using ES6 modules and right now the lack of ES6 modules on npm makes any gains from tree shaking not worth the productivity loss). Perf is always fun to optimise, but we're got features customers want that we need to work on first!
- jbergstroem 8y agoThis together with cache-control: immutable is a great bandwidth saver: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...
- lucideer 8y agoIf you're interested in analysing CertSimple performance, removing the artificial 8s delay on the CertSimple blog might be another line of inquiry. animation:-amp-start 8s steps(1,end) 0s 1 normal both
- noname120 8y agoWorth noting for backend engineers: every major browser supports Brotli compression, and 85.22% of users run a browser that supports it[1]. [1] https://caniuse.com/#feat=brotli https://caniuse.com/#feat=brotli
- saagarjha 8y agoIt's kind of sad that more people use Internet Explorer 11 than Edge…
- robocat 8y ago9 IE users for every 1 Edge user in our stats. Compared with 9 Win7 users for every 7 Win10 users. We serve small to medium sized businesses for our customer base in NZ. Although I am not surprised: the Edge UI is buggy and the browser engine has its problems with sites too. And we have put a lot of work into ensuring Edge works properly for our clients.
- WorkLifeBalance 8y agoI wish it wasn't, but IE11 is still a major browser, at least too big to just dump completely in my experience. The UK is worse than most countries in that regard though with higher than average IE11 usage.
- dictum 8y agoIndeed, but this is not a problem for this enhancement specifically. IE11 asks for gzipped files, and the server can still send them when the client doesn't support Brotli, based on the Accept-Encoding header.
- linsomniac 8y agoWe just had tried brotli and decided not to implement it yet. One of our devs just returned from a conference super excited about brotli. I'm the ops guy and want to try to support excitement, so I spent the better part of 2 days setting up an experiment, but I was skeptical. I'd been keeping my eye on brotli for years. I first went out and looked for some existing evaluations of brotli, and found years old write-ups, one from Cloudflare from 2015, and another but I can't remember the details there. They both concluded that brotli wasn't ready yet, but they were also old... Our stack uses Apache on Ubuntu 16.04, and the first stumbling block was that the Apache version on 16.04 is just a bit before the mod_brotli was added. So I used Ansible to spin up an 18.04 web server which did have the right version of Apache. But, apache is built without brotli support. So I ended up having to build my own custom packages, which went fairly smoothly. We have one particular asset that right now we would like to speed up. It is a big JSON data structure. The dev did some testing and found that his 350K data-set (with gzip) ended up as 320K with brotli. Maintaining our own custom Apache packages, and possibly having to upgrade our entire stack from 16.04 to 18.04, plus resolve known problems with OS packages that we've seen in 18.04, for a <10% compression gain, for our cherry-picked "best example for brotli wins"? Our decision was that if we could have just turned it on via changes to the Apache configs, we would have done it. But as it is, it's not worth it for us to do. YMMV.
- JyrkiAlakuijala 8y agoCloudflare analysis divides size difference by time difference. That function is commutative and cannot be used for meaningful comparisons. This leads to faulty numbers and faulty analysis.
- rgbrenner 8y agoNone of that has anything to do with brotli. Blame the distro or the package maintainer.
- adamc 8y agoHis point is that brotli's advantages aren't big enough to justify it until it becomes ubiquitous. Compression improvements of less than 10% just don't justify much effort to get ahead of the curve (in most situations).
- dvfjsdhgfv 8y ago> Modern websites in particular often have large JavaScript bundles - the front page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed! What puzzles me though... How much of this 1.1MB is really needed? You can do form validation with HTML5. I don't see any other candidate on their front page. So instead of getting a 14% advantage, what about pushing it to 100% or so?
- Klathmon 8y agoSo you looked at their front page and concluded that they should be able to get rid of all javascript? CertSimple is a full dashboard and application that does a lot of crypto in the browser which requires quite a lot of code. Sure, there are probably gains to be had with code splitting and lazy loading, but those get significantly more complex and can cause issues for codebases that aren't setup to take advantage of it from the start. And improvements are improvements. This was a blog post showing you how and why to implement brotli, not a request for how to improve page loading speed for their application specifically.
- dvfjsdhgfv 8y agoI looked at the front page because they specifically mentioned it in the article: > the front page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed! Of course the full app is a completely different thing.
- nailer 8y ago> Of course the full app is a completely different thing. Author here: no. The app is the front page. /blog is a seperate AMP based site and tiny. Search 'discify' in this thread for a breakdown of what's in the module.
- chrismorgan 8y agoHTML5’s built-in validation is actually not all it’s cracked up to be just yet, because the currently defined and supported CSS selectors don’t cover all cases needed: you don’t want all your fields to be styled invalid on page load, only once the user has interacted with them or tried to submit. Firefox has :-moz-ui-invalid, I don’t think others have anything, and :user-invalid is the direction Selectors Level 4 is going, see https://drafts.csswg.org/selectors-4/#user-pseudos https://drafts.csswg.org/selectors-4/#user-pseudos. If you want a pleasant experience, you can’t use HTML form validation by itself. You need some added JavaScript. (I am not defending over a megabyte of JavaScript. I like to do such things from scratch and optimise for size, and all the functionality that I see on that page looks like it’d comfortably fit in under ten kilobytes of JavaScript. The CSS can probably also be decimated. But these things only happen that way if you value such performance, and few do—perhaps justifiably, perhaps not.)
- jgrahamc 8y agoMirrors our experience: https://blog.cloudflare.com/results-experimenting-brotli/ https://blog.cloudflare.com/results-experimenting-brotli/ but we chose to use Brotli level 5.
- jbergstroem 8y agoDoes Cloudflare pass a dictionary when compressing for your clients? Would be interesting to hear about excursions in testing dictionary sizes/variations at that scale.
- jgrahamc 8y agoI'll suggest to someone internally that they write up all the details of how we support Brotli.
- wildpeaks 8y agoI wish there was a simpler way to use Brotli with Nginx: at the moment, either you have to compile from sources, either rely on a PPA to use a special build of Nginx (and an extra module) instead of the one from the LTS packages, even in 18.04. I'd especially like to use Brotli because it has good support in modern browsers (pretty much everything except IE11) and would be good replacement for gzip (which should be disabled for HTTPS).
- chrisweekly 8y ago> "gzip (which should be disabled for HTTPS)" ? Please explain.
- evil-olive 8y agoBREACH and CRIME [0,1] are attacks on the combination of HTTPS and compression. There are workarounds / mitigations with various effectiveness, but disabling compression when TLS is used is the simple way to prevent the attack. 0: https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH 1: https://en.wikipedia.org/wiki/CRIME https://en.wikipedia.org/wiki/CRIME
- emj 8y agoI thought the TLS length hiding extension was supposed to fix this, I wonder why it did not progress. Maybe it is just easier to disable compression for dynamic content. [0] https://tools.ietf.org/html/draft-pironti-tls-length-hiding-02 https://tools.ietf.org/html/draft-pironti-tls-length-hiding-...
- therealmarv 8y agoThis reminds me I have to develop and send sometime a PR for Axios to support Brotli. We turned brotli on and figured out we only get garbage because Axios itself also needs to understand on how to handle brotli https://github.com/axios/axios/blob/cb630218303095c0075182b542ccb2f72d20dd9d/lib/adapters/http.js#L156 https://github.com/axios/axios/blob/cb630218303095c0075182b5...
- thecompilr 8y agoFWIW at Cloudflare we were running brotli for dynamic content for a while now. However the Cloudflare gzip library is much faster than brotli. You can find a benchmark in here: https://blog.cloudflare.com/arm-takes-wing/ https://blog.cloudflare.com/arm-takes-wing/
- paulddraper 8y agoBrotli is amazing. But... > Brotli can compress faster than gzip and still produce smaller files Huh? > For dynamic content, we'll use 4, which still produces smaller responses but takes less time to compress than gzip or brotli on a higher setting. Oh if you compare to gzip max. But who runs gzip max? Gzip 6 gives decent low cost compression. Gzip 9 is barely smaller but much slower. Algorithms have their sweet spots. Brotli (small), LZMA (fast), gzip (somewhere in between). Knowing nothing about my clients, bandwidth, or dynamism, I can turn on gzip without really a second thought.
- mkesper 8y agoSane defaults matter. Why does brotli do maximum compression by default?
- JyrkiAlakuijala 8y agoWe chose max as default because Brotli actually compresses more at max setting. Gzip does not compress much more at max than it does at faster middle qualities, so they chose a default from middle qualities.
- klodolph 8y agoThere really is no such thing as a sane default for compression in general. There are three common use cases and each one demands different tradeoffs. Maximum compression is often suitable for compression once and decompression many times. Lower compression is useful for compression/decompression once. A middle value is useful for compression once/decompression less than once. Then you choose different values based on the relative costs of CPU, memory, network bandwidth, disk space, etc. “Maximum” is good because it handles a common use case. When you’re running a CLI tool, you’re much more likely to be processing static assets which are served many times. So “maximum” is a good guess of what the user wants.
- rossy 8y agoThe offline brotli compression tool uses maximum compression by default. The ngx_brotli module uses 6. I think that's reasonable. If you're compressing a file offline, you probably don't have big time constraints, so it makes sense to try to get the smallest size possible.
- halayli 8y agoWhy not standardize zstd? it seems to beat brotli in ratio,comp/decomp based on the benchmarks below: https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
- JyrkiAlakuijala 8y agoAlways look for third party benchmarking. High quality benchmarks can be found at lz turbo or from encode.ru
- JyrkiAlakuijala 8y agoOne reason might be that browser manufacturers are not keen to have a decompression that might allocate 128 mb per connection.
- klodolph 8y agoJust get browsers to implement it, I think by the time Zstandard was open sourced, there was some browser support for Brotli already (probably because Google controls a major website, a major browser, and the standard library for a major OS). Timeline: Zstandard was released August 31, 2016. By that point, Firefox, Chrome, and Opera already had Brotli support. Safari and Edge got it shortly afterwards.
- khaled 8y agoI think browser support for Brotli is mostly a side effect of supporting WOFF2 font format (which uses Brotli).
- Solar19 8y agoThe CertSimple article completely ignores CPU and memory usage. That is where Brotli can become completely unacceptable for dynamic compression – it uses vastly more CPU and RAM than gzip. Brotli is fine if you can precompress static files on a prep server or desktop. It's unprofessional and unscientific to evaluate compression codecs without measuring how much CPU and memory they use. Brotli is a disappointment, coming 20 years after gzip. In 20 years we should expect something better than Brotli, something that is better optimized for modern CPUs, vectorized by default, and with a much better dictionary.
- JoachimSchipper 8y agoCPU usage (but not memory usage) seems to be addressed by the section "Brotli can compress faster than gzip and still produce smaller files", quoting https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html https://blogs.akamai.com/2016/02/understanding-brotlis-poten...?
- d33 8y agoHow about zstd? https://github.com/facebook/zstd https://github.com/facebook/zstd
- joosters 8y agoI find zstd to be almost twice the speed of brotli when compressing files (my simple test case is a 55M directory of small files - about 2k each) However, zstd doesn't have the browser support that brotli does. So you can't easily replace brotli with zstd for web use.
- JyrkiAlakuijala 8y agoZstd compresses 5% less densely. Zstd compresses slower to a given density. Decoding zstd may use 128 mb of ram whereas Brotli may use up to 16 mb.
- KirinDave 8y agoIsn't it like within 10% of the default gzip settings in every case?
- bestboy 8y agoThe fact that there is no native JAVA encoder for Brotli is an obstacle for the adoption in the enterprise segment.
- rocqua 8y agoAfaik combining compression and dynamic content has a tendency to weaken encryption (BREACH). Compression ratios give information about the contents of a message. Hence, it seems to make sense to only compress static content from a security standpoint.
- cm2187 8y agoFor those who wish to put Brotli support on Microsoft's radar: https://windowsserver.uservoice.com/forums/310252-iis-and-web-server-role/suggestions/33465607-support-brotli-compression https://windowsserver.uservoice.com/forums/310252-iis-and-we...
- powturbo 8y agoSee [Static/Dynamic web content compression benchmark] One of the best benchmark about web compression including brotli + different gzip compatible libraries: https://sites.google.com/site/powturbo/home/web-compression https://sites.google.com/site/powturbo/home/web-compression