4 ms·

Or just the ability to tell your clients to back off. Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can sen

by xxpor 10y ago

Or just the ability to tell your clients to back off. Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can send half of the requests 503 until you're back at a manageable load. Hopefully detecting the load/generating the 503 is really cheap.

user5994461 10y ago
> until you're back at a manageable load. Nope. Gotta have the capacity at hand. You can't rely on getting it later when everything is fucked and you're already down. Down = Loosing clients = Loosing money = NoNonononono Source: Lost 4 million dollars last week because of that. Exponential backoff is to prevent cascading failure (and retry causing DDoS), that is a failsafe in case of failure, but an excuse for failing.
- inopinatus 10y ago
  System capacity isn't always limited by resources within your control e.g. seats going on sale for an enormously popular concert, you're immediately throttled down to the sales abandonment rate. Everyone else needs to enter a queue. More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
  - user5994461 10y ago
    Well. The concert reseller still has to serve all the clients. It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either. That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
- xxpor 10y ago
  People tend to come back more than you think. Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. This can be mitigated though if you have say 3 datacenters and you want to be able to run with 1 down, because that divides the load.
  - user5994461 10y ago
    > Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. Till DynDns goes down for 8 hours and you're the only player left in town. Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8) YMMV.

by xxpor 10y ago

user5994461 10y ago

> until you're back at a manageable load. Nope. Gotta have the capacity at hand. You can't rely on getting it later when everything is fucked and you're already down. Down = Loosing clients = Loosing money = NoNonononono Source: Lost 4 million dollars last week because of that. Exponential backoff is to prevent cascading failure (and retry causing DDoS), that is a failsafe in case of failure, but an excuse for failing.

inopinatus 10y ago
System capacity isn't always limited by resources within your control e.g. seats going on sale for an enormously popular concert, you're immediately throttled down to the sales abandonment rate. Everyone else needs to enter a queue. More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
- user5994461 10y ago
  Well. The concert reseller still has to serve all the clients. It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either. That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
xxpor 10y ago
People tend to come back more than you think. Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. This can be mitigated though if you have say 3 datacenters and you want to be able to run with 1 down, because that divides the load.
- user5994461 10y ago
  > Running at a 2N (or more) configuration 24/7 is a great way to light money on fire. Till DynDns goes down for 8 hours and you're the only player left in town. Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8) YMMV.