5 ms·
This article touches on "Request Coalescing" which is a super important concept - I've also seen this called "dog-pile prevention" in the past. Varnish has thi
by simonw 5y ago
This article touches on "Request Coalescing" which is a super important concept - I've also seen this called "dog-pile prevention" in the past.
Varnish has this built in - good to see it's easy to configure with NGINX too.
One of my favourite caching proxy tricks is to run a cache with a very short timeout, but with dog-pile prevention baked in.
This can be amazing for protecting against sudden unexpected traffic spikes. Even a cache timeout of 5 seconds will provide robust protection against tens of thousands of hits per second, because request coalescing/dog-pile prevention will ensure that your CDN host only sends a request to the origin a maximum of once ever five seconds.
I've used this on high traffic sites and seen it robustly absorb any amount of unauthenticated (hence no variety on a per-cookie basis) traffic.
- anonymoushn 5y agoDo you know if varnish's request coalescing allows it to send partial responses to every client? For example, if an origin server sends headers immediately then takes 10 minutes to send the response body at a constant rate, will every client have half of the response body after 5 minutes? Thanks!
- simonw 5y agoI don't know for certain, but my hunch is that it streams the output to multiple waiting clients as it receives it from the origin. Would have to do some testing to confirm that though.
- elithrar 5y agoI don’t know about Varnish, but having worked on other implementations, you would usually have a timeout on the initial lock (semaphore) to prevent a slow connection from impacting all clients. But this is much, much harder to do once you are already streaming the response - if the time to first byte (TTFB) is quick, but the connection is low-throughout, you can’t do much at this point. But nearly all modern implementations stream the bytes to all clients immediately; they don’t try to fill the cache first (they do it simultaneously). Some implementations might avoid fanning in too much - maintaining a smaller pool of connections rather than trying get to ”1”, but that’s ultimately a trade-off at each layer of the onion, as they can still add up. (I worked at both Cloudflare and Google, and it was a common topic: request coalescing is a big deal for large customers)
- deleted 5y ago[deleted]
- anonymoushn 5y agoI think the nginx that members of the public can get from their package manager does not have this feature, and will force each client other than the first to either wait for the entire body to be downloaded or wait for a timeout and hit the origin in a non-cacheable request.
- mnutt 5y agoVarnish has defaulted to streaming responses since varnish 4. I think it gets used for a lot of video streaming use cases.
- sleepy_keita 5y agoBack when I was just getting started, we were doing a lot of WordPress stuff. A client contacted us, "oh yeah, later today we're probably going to have 1000x the traffic because of a popular promotion". I had no idea what to do so I thought, I'll just set the varnish cache to 1 second, that way WordPress will only get a maximum of 60 requests per second. It worked pretty much flawlessly, and taught me a lot about the importance of request coalescing and how caches work.
- skunkworker 5y agoI've implemented this manually in some golang web applications I've written. It really helps when you have an expensive cache-miss operation, as it can stack the specific requests so that once the original request is served, all of the stacked requests are served with the cached copy.
- cortesoft 5y ago"Thundering herd" problem is how I have always heard it called.
- thaumasiotes 5y agoThe thundering herd problem isn't really about high levels of traffic. To the extent that that's a problem, it's just an ordinary DOS. The thundering herd problem specifically refers to what happens if you coordinate things so that all your incoming requests occur simultaneously. Imagine that over the course of a week, you tell everyone who needs something from you "I'm busy right now; please come back next Tuesday at 11:28 am". You'll be overwhelmed on Tuesday at 11:28 am regardless of whether your average weekly workload is high or low, because you concentrated your entire weekly workload into the same one minute. You solve the thundering herd problem by not giving out the same retry time to everyone who contacts you while you're busy.
- 8note 5y agoHmm. I think of thundering Herd being about retries. All your failing requests batch up when your retry strategy sucks, then you end up really high traffic on every retry, and very little in between
- thaumasiotes 5y agoWhere does your perspective differ from what I said above?
- thraxil 5y agoRetries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.
- ithkuil 5y ago
- mnutt 5y agoIn varnish, if you have some requirements flexibility you can enable grace mode in order to serve stale responses but update from the origin, and avoid long requests every [5] seconds. Not quite the same layer, but in node.js I’m a fan of the memoize(fn)->promise pattern where you wrap a promise-returning function to return the _same_ promise for any callers passing the same arguments. It’s a fairly simple caching mechanism that coalesces requests and the promise resolves/rejects for all callers at once.
- sciurus 5y agoI'll echo what Simon said; we share some experiences here. There's a potential footgun, though, anyone getting started with this should know about- Request coalescing can be incredibly beneficial for cacheable content, but for uncacheable content you need to turn it off! Otherwise you'll cause your cache server to serialize requests to your backend for it. Let's imagine a piece of uncacheable content takes one second for your backend to generate. What happens if your users request it at a rate of twice a second? Those requests are going to start piling up, breaking page loads for your users while your backend servers sit idle. If you are using Varnish, the hit-for-miss concept addresses this. However, it's easy to implement wrong when you start writing your own VCL. Be sure to read https://info.varnish-software.com/blog/hit-for-miss-and-why-a-null-ttl-is-bad-for-you https://info.varnish-software.com/blog/hit-for-miss-and-why-... and related posts. My general answer to getting your VCL correct is writing tests, but this is a tricky behavior to validate. I'm unsure how nginx's caching handles this, which would make me nervous using the proxy_cache_lock directive for locations with a mix of cacheable and uncacheable content.
- endymi0n 5y agoAnd to add the last big one from the trifecta: Know how to deal with cacheable data. Know how to deal with uncacheable data. But by all means, know how to keep them apart. Accidentally caching uncacheable data has lead so some of the most ugly and avoidable data leaks and compromises in recent times. If you go down the "route everything through a CDN route (that can be as easy as ticking a box in the Google Cloud Platform backend), make extra sure to flag authenticated data as cache-control: private / no-cache.
- arghwhat 5y agono-cache does not mean content must not be cached - in fact, it specifies the opposite! no-cache means that the response may be stored in any cache, but cached content MUST be revalidated before use. public means that the response may be cached in any cache even if the response was not normally cacheable, while private restricts this to only the user agent's cache. no-store specifies that this response must not be stored in any cache. Note that this does not invalidate previous cached responses from being used. max-age=0 can added to no-store to also invalidate old cached responses should one have accidentally sent a cacheable response for this resource. No other directives have any effect when using no-store.
- philsnow 5y agounrelated to CDNs but IIRC vitess did/does query coalescing too -- if it starts to serve a query for "select * from users where id = 123" and then another 20 connections all want the same query result, vitess doesn't send all 21 select queries to the backend, it sends the first one and then has all the connections wait on the backend response, then serves the same response to them all.
- samlambert 5y agoVitess still does this. It can also do similar with writes on hot rows where someone is incrementing a counter for example.
- philsnow 5y agoWait, how can it do it with writes/increments? Does it keep track of which rows are hot and add a short but stochastically distributed delay to writes to try to coalesce more updates into a single hit to the DB? I would think you'd need to do it that way, you wouldn't want to reply "done" to the first increment if that operation is going to be batched up with other ops; you'd want to keep that connection hanging until all the increments you're going to aggregate have all been committed by the backend. In the select coalescing case, except for bookkeeping overhead, none of the queries are slower (it's a big net win all around because not only do clients get their answers on average somewhat sooner, but the DB doesn't have to parse those queries, check for them in the query cache, or marshal N responses). But in the increment/write case, it seems like in order to spare some DB resources, some clients will perceive increased write delays (or does it still net a win because the DB backend doesn't have to deal with the contention?).
- dbbk 5y agoIs this the same idea as `stale-while-revalidate`?