34 ms·
The most important part of this article is the concept of back pressure and being able to detect it. It's common in a ton of other engineering disciplines but e
by jtchang 10y ago
The most important part of this article is the concept of back pressure and being able to detect it. It's common in a ton of other engineering disciplines but especially important when designing fault tolerant or load balancing systems at scale.
Basically it is just some type of feedback so that you don't overload subsystems. One of the most common failure modes I see in load balanced systems is when one box goes down the others try to compensate for the additional load. But there is nothing that tells the system overall "hey there is less capacity now because we lost a box". So you overwhelm all the other boxes and then you get this crazy cascade of failures.
- user5994461 10y agoYes. You need to adapt the capacity of the system to handle the full load with -N- boxes dead. Corollary: If you have 2 boxes, each of them has to be able to handle all the traffic, so you can't save money by using smaller boxes :D Corollary #2: If you have 2 datacenters, each of them has to be able to handle all the traffic, so you burn a lot of money :D
- xxpor 10y agoOr just the ability to tell your clients to back off. Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can send half of the requests 503 until you're back at a manageable load. Hopefully detecting the load/generating the 503 is really cheap.
- user5994461 10y ago> until you're back at a manageable load. Nope. Gotta have the capacity at hand. You can't rely on getting it later when everything is fucked and you're already down. Down = Loosing clients = Loosing money = NoNonononono Source: Lost 4 million dollars last week because of that. Exponential backoff is to prevent cascading failure (and retry causing DDoS), that is a failsafe in case of failure, but an excuse for failing.
- inopinatus 10y agoSystem capacity isn't always limited by resources within your control e.g. seats going on sale for an enormously popular concert, you're immediately throttled down to the sales abandonment rate. Everyone else needs to enter a queue. More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
- user5994461 10y agoWell. The concert reseller still has to serve all the clients. It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either. That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
- xxpor 10y agoPeople tend to come back more than you think. Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. This can be mitigated though if you have say 3 datacenters and you want to be able to run with 1 down, because that divides the load.
- user5994461 10y ago> Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. Till DynDns goes down for 8 hours and you're the only player left in town. Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8) YMMV.
- rahimnathwani 10y agoInstead of 2 boxes that can handle all your load, why not 4 that can each handle 1/3rd of the load?
- user5994461 10y agoLet's say that one box is down (don't care why, boxes die all the time). You have 3 boxes, they're all running at 100%, or worse if the load is not perfectly balanced (it never is). You've got bad latency and you're up for cascading failures. Your system is in bad shape. Now, supposing you have to deploy your application. Gotta take a server offline temporarily, the load will be 100% on the last 2 servers left... that can only take 66%. Booya! You're down :p That's what happen when people are cheap on servers, in high performance systems.
- deleted 10y ago[deleted]
- vidarh 10y agoYes, but his point is still valid: You can to an extent reduce the overprovisioning you need by adding more, smaller units. In your example, two boxes would have meant downtime no matter the capacity of each box too. All your example demonstrates is a reason why you might want to have capacity that allows taking more than one box offline.... Which could be handled with bigger servers, or more smaller servers.
- user5994461 10y agoYes to all of this. I write this to illustrate that you must have a significant batch of servers (about 5) in order to achieve costs savings (by using smaller servers) WHILE not affecting reliability. Well, that is, if you care about rare cascading failures and performances issues.
- aaronblohowiak 10y agoyep, and it is cheaper as well.
- pdexter 10y agohttp://ferd.ca/handling-overload.html http://ferd.ca/handling-overload.html This is a good article about overload and back pressure. It also lists some tools in Erlang to solve these sorts of issues. It also mentions genstage (very) briefly.
- Sikul 10y agoThis short book by Fred is also a great read about Erlang production systems in general https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/
- nurettin 10y agoBackpressure is more common than water across engineering disciplines and it isn't an integrated part of every distributed system out there? Isn't that a bit of an oversight?
- phamilton 10y agoWell, it is a core part of TCP. It's also part of http in a lot of ways. Browsers can make N simultaneous connections to a server. If those requests get queued up on the server side, then the client won't proceed with a new request until one finishes. The pattern I see all to frequently is asynchronous worker queues. Those can very easily undermine backpressure. The rise of Ruby/Python and their limited concurrency models has placed a taboo on synchronous operations. However, synchronicity natural lends itself to backpressure.