4 ms·

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there'

by siavosh 14y ago

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

laumars 14y ago
It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole in a damn can buckle and blow open from the force of the water pushing through.
- ef4 14y ago
  > It's not really possible without adding resources (ie a new server). That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure. One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.
  - laumars 14y ago
    Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that siavosh's method of taking servers out of service has the inverse effect of what he was trying to achieve). Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle. The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> n" active TCP/IP connections.
trotsky 14y ago
in linux cgroups (control groups) allow you to allocate resources to groups of processes, which makes it easy to ensure things like ssh or npre or your load balancer ping (as long as you run it out of band) still have cpu/mem even when your main server cgroup gets pegged
Andys 14y ago
The problem is queueing at the worker end and not the proxy side. Using a fair queueing system (such as Nginx HttpUpstreamFairModule) can limit the number of concurrent requests sent to workers so they don't get overloaded. It will make requests queue up at the proxy and then get distributed evenly. See: recent discussion on problems caused by Heroku's random order load balancing.
brown9-2 14y ago
I think it really has more to do with a lack of spare capacity - each server is so maxed out that it can't handle a 1/n increase in load when one server goes away.

by siavosh 14y ago

laumars 14y ago

It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole in a damn can buckle and blow open from the force of the water pushing through.

ef4 14y ago
> It's not really possible without adding resources (ie a new server). That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure. One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.
- laumars 14y ago
  Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that siavosh's method of taking servers out of service has the inverse effect of what he was trying to achieve). Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle. The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> n" active TCP/IP connections.

trotsky 14y ago

in linux cgroups (control groups) allow you to allocate resources to groups of processes, which makes it easy to ensure things like ssh or npre or your load balancer ping (as long as you run it out of band) still have cpu/mem even when your main server cgroup gets pegged

Andys 14y ago

The problem is queueing at the worker end and not the proxy side. Using a fair queueing system (such as Nginx HttpUpstreamFairModule) can limit the number of concurrent requests sent to workers so they don't get overloaded. It will make requests queue up at the proxy and then get distributed evenly. See: recent discussion on problems caused by Heroku's random order load balancing.

brown9-2 14y ago

I think it really has more to do with a lack of spare capacity - each server is so maxed out that it can't handle a 1/n increase in load when one server goes away.