3 ms·
Yes. You need to adapt the capacity of the system to handle the full load with -N- boxes dead. Corollary: If you have 2 boxes, each of them has to be able to h
by user5994461 10y ago
Yes. You need to adapt the capacity of the system to handle the full load with -N- boxes dead.
Corollary: If you have 2 boxes, each of them has to be able to handle all the traffic, so you can't save money by using smaller boxes :D
Corollary #2: If you have 2 datacenters, each of them has to be able to handle all the traffic, so you burn a lot of money :D
- xxpor 10y agoOr just the ability to tell your clients to back off. Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can send half of the requests 503 until you're back at a manageable load. Hopefully detecting the load/generating the 503 is really cheap.
- user5994461 10y ago> until you're back at a manageable load. Nope. Gotta have the capacity at hand. You can't rely on getting it later when everything is fucked and you're already down. Down = Loosing clients = Loosing money = NoNonononono Source: Lost 4 million dollars last week because of that. Exponential backoff is to prevent cascading failure (and retry causing DDoS), that is a failsafe in case of failure, but an excuse for failing.
- inopinatus 10y agoSystem capacity isn't always limited by resources within your control e.g. seats going on sale for an enormously popular concert, you're immediately throttled down to the sales abandonment rate. Everyone else needs to enter a queue. More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
- user5994461 10y agoWell. The concert reseller still has to serve all the clients. It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either. That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
- xxpor 10y agoPeople tend to come back more than you think. Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. This can be mitigated though if you have say 3 datacenters and you want to be able to run with 1 down, because that divides the load.
- user5994461 10y ago> Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. Till DynDns goes down for 8 hours and you're the only player left in town. Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8) YMMV.
- rahimnathwani 10y agoInstead of 2 boxes that can handle all your load, why not 4 that can each handle 1/3rd of the load?
- user5994461 10y agoLet's say that one box is down (don't care why, boxes die all the time). You have 3 boxes, they're all running at 100%, or worse if the load is not perfectly balanced (it never is). You've got bad latency and you're up for cascading failures. Your system is in bad shape. Now, supposing you have to deploy your application. Gotta take a server offline temporarily, the load will be 100% on the last 2 servers left... that can only take 66%. Booya! You're down :p That's what happen when people are cheap on servers, in high performance systems.
- deleted 10y ago[deleted]
- vidarh 10y agoYes, but his point is still valid: You can to an extent reduce the overprovisioning you need by adding more, smaller units. In your example, two boxes would have meant downtime no matter the capacity of each box too. All your example demonstrates is a reason why you might want to have capacity that allows taking more than one box offline.... Which could be handled with bigger servers, or more smaller servers.
- user5994461 10y agoYes to all of this. I write this to illustrate that you must have a significant batch of servers (about 5) in order to achieve costs savings (by using smaller servers) WHILE not affecting reliability. Well, that is, if you care about rare cascading failures and performances issues.
- aaronblohowiak 10y agoyep, and it is cheaper as well.