2 ms·

> Another colleague in the chat remarked up-thread (apropos cold reboot thinking): > I have seen this at <Indian eCommerce Giant> and at <a FAANG>. Most of it

by tivert 3y ago

> Another colleague in the chat remarked up-thread (apropos cold reboot thinking): > I have seen this at <Indian eCommerce Giant> and at <a FAANG>. Most of it is related to cached data. Cold starts with empty caches causes too much load on databases. And then the failures cascade. > — Another M'colleague in the Slackroom. Isn't that not really a problem with cold restart per se, but more the restart procedure? If caches are so critical, wouldn't you need a feature to throttle the load to what the databases can handle, as the caches populate? E.g., if you're cold-rebooting Facebook, start by blocking all connections except those geolocated to North Dakota, then add other regions as your caches fill.

jbnorth 3y ago
That’s spot on. I work at a large cloud provider and one of our larger eCommerce customers had an outage in a kubernetes cluster which handled the front end traffic routed through a large CDN provider. Well sure enough “just turn it back on” wasn’t an option since the surge of traffic was too rapid for the services and the cluster to scale out. They ended up having to turn the traffic back on incrementally to let things scale up to the point where they could handle the load.
- donalhunt 3y ago
  One of the earliest incidents I worked on in the late 90s involved students DDOSing a university webserver in anticipation of exam results being posted. The server load was so high we had to pull the physical plugs on the server. :/
benlivengood 3y ago
Specifically you want load shedding and in the servers and retry with back off in the clients. Clients should do their best to exponentially back off on retrying failed requests and only try to contact healthy servers and maintain internal rate-limiting based on error rate, and servers should do their best to reply with failure quickly and cheaply to drive good client's backoffs/rate-limiting and just drop bad client's traffic, and the service discovery should try to detect and spread load across healthy servers (but this isn't always available at first in a cold start because the metrics or metadata probably aren't available yet), but in the end it's up to servers to reliably drop traffic they can't handle instead of building up giant queues and slowing to a crawl. Middleware is the hardest because it has to be a good client and also fail fast on overload as a server by correctly interpreting upstream behavior. Deadlines in RPCs that get passed across system boundaries can work pretty well for tall stacks of system layers where service health discovery or dependency discovery is hard, but require careful configuration to avoid failure or very slow starts under heavy load.

by tivert 3y ago

jbnorth 3y ago

That’s spot on. I work at a large cloud provider and one of our larger eCommerce customers had an outage in a kubernetes cluster which handled the front end traffic routed through a large CDN provider. Well sure enough “just turn it back on” wasn’t an option since the surge of traffic was too rapid for the services and the cluster to scale out. They ended up having to turn the traffic back on incrementally to let things scale up to the point where they could handle the load.

donalhunt 3y ago
One of the earliest incidents I worked on in the late 90s involved students DDOSing a university webserver in anticipation of exam results being posted. The server load was so high we had to pull the physical plugs on the server. :/

benlivengood 3y ago

Specifically you want load shedding and in the servers and retry with back off in the clients. Clients should do their best to exponentially back off on retrying failed requests and only try to contact healthy servers and maintain internal rate-limiting based on error rate, and servers should do their best to reply with failure quickly and cheaply to drive good client's backoffs/rate-limiting and just drop bad client's traffic, and the service discovery should try to detect and spread load across healthy servers (but this isn't always available at first in a cold start because the metrics or metadata probably aren't available yet), but in the end it's up to servers to reliably drop traffic they can't handle instead of building up giant queues and slowing to a crawl. Middleware is the hardest because it has to be a good client and also fail fast on overload as a server by correctly interpreting upstream behavior. Deadlines in RPCs that get passed across system boundaries can work pretty well for tall stacks of system layers where service health discovery or dependency discovery is hard, but require careful configuration to avoid failure or very slow starts under heavy load.