7 ms·

A self-killing web site requested by a customer (2011)

ciprian_craciun 6y ago
I found it interesting because such a simple task (requiring at least a number of on-line servers before the load-balancer starts serving requests), required a custom binary controlled the webserver and had to cross-monitor each server. For example with HAProxy (my favorite load-balancer and HTTP "router") this can be easily achieved by using `nbsrv`, creating an ACL and only routing requests to the backend based on that ACL. Based on the documentation bellow: * http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#7.3.1-nbsrv http://cbonte.github.io/haproxy-dconv/2.1/configuration.html... * http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-monitor%20fail http://cbonte.github.io/haproxy-dconv/2.1/configuration.html... One can write this: ~~~~ frontend www mode http acl site_alive nbsrv(dynamic) gt 2 use_backend dynamic if site_alive ~~~~ [This article was linked from the original article described in (https://news.ycombinator.com/item?id=23099347 https://news.ycombinator.com/item?id=23099347).]
- jefftk 6y ago
  Things are a lot easier these days, mostly because there's been more time for people to have problems and code up good generalizable solutions
  - ciprian_craciun 6y ago
    I just wanted to mention that I didn't fault the original author for the proposed solution. It was 2011, they seemed to use some commercial load-balancer, and the customer didn't seem to want to actually solve the solution. I actually found the original solution interesting, and I also proposed an alternative achievable with HAProxy as an load-balancer.
    - dumbfounder 6y ago
      This was written in 2011, I think it's something that happened many years before. Sounds more like a 2001 thing to do.
- wrkronmiller 6y ago
  That config looks like it kills the site if there are 2 or fewer back-end servers (similar to the setup in the article). Wouldn't it make more sense simply to limit the number of connections per back-end server?
  - ciprian_craciun 6y ago
    Indeed if I were to configure the load-balancer I would limit both the number of concurrent requests and the number of queued requests. (In fact this is what I do in my HAProxy deployments.) However I assume that the customer (the original author was referring to) didn't want to apply any limitations. (Why is beyond my reasoning...)
klausjensen 6y ago
Lovely hack and an example of how thinking outside the box can create solutions, that are order of magnitude cheaper than the "obvious" solution.
MaxBarraclough 6y ago
> The load now rebalanced to four remaining machines is just far too big, and they all die as a result. Perhaps I'm missing something terribly obvious here, but why would that happen? I can understand requests being dropped and processing-times worsening, but a full system-wide crash? edit My bad, I'd missed this in the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't, and so now I have a story
- sneak 6y ago
  System latency is not linear with load. At a certain threshold it starts swapping way beyond the capacity of the disk i/o and basically nothing gets done. I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.
  - MaxBarraclough 6y ago
    Why doesn't the server just drop incoming requests until it's able to handle them again? As I mentioned in another comment, this is what routers do. edit I hadn't noticed this at the bottom of the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
    - rovr138 6y ago
      Apache has a way to limit the amount of processes. Tuning that would have helped.
- brabel 6y ago
  This will easily happen with systems that do not have any kind of upper limits on what they _try_ to do. For example: not limiting your thread pool size so that your server might try to create more threads than the system has memory for, dying due to lack of memory. Similar things can happen even if you're using async IO as you still need memory for all those stacks being run asynchronously.
nicbou 6y ago
This is called a cascading failure. It's also a problem with the electric grid, and more terrifyingly with global finance. https://en.wikipedia.org/wiki/Cascading_failure https://en.wikipedia.org/wiki/Cascading_failure
- Beltiras 6y ago
  I'd imagine this is the next couple of years in finance due to COVID. Shocks and reverberations for a long long while.
  - nicbou 6y ago
    Travel, immigration, entertainment, restaurants etc are severely affected. A lot of people have lost their income, and spend less. There is also some feat that this will exacerbate problems with an already stressed financial system. Banks are massively increasing their cash reserves, and many organisations have a grim outlook on the upcoming months.
    - nvahalik 6y ago
      There may also end up being a potential flip-side to this. Many people (dubbed essential) have found that last couple of months that the lack of open businesses + increased pay have meant that in a few months they are going to be in the market for larger purchases (i.e. houses, cars, etc.). Their savings have gone up and at least person I've talked too has mentioned opening up a new business after all of this is over.
- joncrane 6y ago
  This is a really interesting failure pattern and has led to the development of so-called "circuit breakers" to avoid the cascade. (Many of the recent (last ~5 years) AWS failures have been due to cascading failures and it's neat to read their postmortems).
- DaiPlusPlus 6y ago
  The cascading failure of NYC’s electricity grid in 1965 was the background of episode 1 of Connections.
londons_explore 6y ago
A better solution would be to simply configure the loadbalancer to have a maximum number of requests per second per endpoint and then to drop any requests over that. An even better loadbalancer will poll a load endpoint, representing CPU load, queue length, percentage of time GC'ing, or some similar metric, and scale back requests as that metric gets too high.
- MichaelApproved 6y ago
  Maybe their “load balancer” was just a simple round robin service which wouldn’t have those abilities.
ninkendo 6y ago
Wait, why are the servers “crashing” when under too much load in the first place? If there’s some sort of natural limit to how many simultaneous connections they can handle, why can’t they just return some 4xx error code for connections beyond that? (And have clients implement an exponential back off?) Or if that’s too difficult, the load balancer could keep track of some maximum number of connections (or even requests per second) each backend is capable of, and throttle (again with some 4xx error code) when the limit has been reached by all backends? This is pretty basic functionality for load balancers. You’re going to need actual congestion control anyway, when the number of client connections is unbounded like this. Even when your servers aren’t crashing, what if the client apps whose clicks you’re tracking becomes suddenly more popular and you can’t handle the load even with all of your servers up?
- laurentdc 6y ago
  > If there’s some sort of natural limit to how many simultaneous connections they can handle Yup, it's pretty much a DDoS. It's simple to test. Spin up Wordpress + Woocommerce (a common ecommerce stack) in a Digitalocean 1 GB droplet. Now ab -n 1000 -c 30 the home page, that's 30 concurrent clients Watch MySQL die, Apache get killed because out of memory, and... root@wootest01:~# reboot -bash: fork: Cannot allocate memory
  - ninkendo 6y ago
    So, maybe fix that? Why is Apache continuing to fork new workers ad infinitum? That’s a denial of service attack waiting to happen, and the answer surely isn’t “oh let’s just automate the rebooting”... Edit: you’ve edited your comment to say it’s a DoS as well, so looks like we’re on the same page here, the software is garbage.
    - laurentdc 6y ago
      > the software is garbage. Yeah, I agree in many cases that's the issue, as the author pointed out in the last paragraph. Let me also add that on this particular stack this happens with caching enabled (W3 Total Cache). Without cache it struggles with 10 concurrent clients (TTFB > 5 seconds). 100 MB per client to serve what's pretty much a grid of products and a button, it's scary
mgkimsal 6y ago
was it just a cost thing that would prevent people from just adding another server in to the mix? given that 4 was the magic number, add another server or two to add buffer to time between servers dying and 'it all breaks'? I'm realizing the cost factor may have been it, depending on size/location/etc. - would there be any other reason?
- Beltiras 6y ago
  Might have been tried and not worked. Might have been a limitation in scaling (f.ex. only being able to do master-slave replication and not being able to add more master nodes). Remember: it's bad software to begin with.
- jasonlotito 6y ago
  Not necessarily the reason here, but to answer your question: "would there be any other reason?" Another reason is backend constraints. Adding another server might simply cause more pressure on backend systems like a database. So they can't simply add more servers, as then the additional load on the database might cause that to crash, and result in even longer downtime.
Random_ernest 6y ago
I am not a webdev, but isn't that a task for the loadbalancer in the first place?
- ciprian_craciun 6y ago
  Unfortunately the load-balancer is not a magic-bullet curing every issue a system has. A load-balancer can be configured to do lots of things, like for example: * limit the number of concurrent requests, and drop the others; * limit the number of concurrent requests, but queue the others (with a timeout); * distribute all requests uniformly (randomly or in round-robin fashion) to all backends; * (any combination of the above); However if the "customer" asks you to not drop or queue requests, then there is nothing the load-balancer can actually do...
  - Random_ernest 6y ago
    I took it from the article that dropping requests was permitted (since that happens when all servers go down). So my assumption is still that a better solution is that the load balancer allows only a specific number of requests per server and rejects or caches the rest. I would even argue that requests being rejected is more understandable for the user than the website simply not being there for a certain time.
    - legacynl 6y ago
      I think you are right, except that the article reads as if they were using a loadbalancer that wasn't in their control (a third-party service). If you can't control your loadbalancer to not pass requests when you're overloaded, the next best thing is to keep track of it on each node, and basically do what the article described.
jrockway 6y ago
There is actually quite a bit of complexity with load balancing, but the good news is that a lot of the complexity is understood and is configurable on the load balancer. I think what Rachel calls a "suicide pact" is now commonly called a circuit breaker. After a certain number of requests fail, the load balancer simply removes all the backends for a certain period of time, and causes all requests to immediately fail. This attempts to mitigate the cascading failure by simply isolating the backend from the frontend for a period of time. If you have something like a "stateless" web-app that shares a database with the other replicas, and the database stops working, this is exactly what you want. No replica will be able to handle the request, so don't send it to any replica. Another option to look into is the balancer's "panic threshold". Normally your load balancer will see which backends are healthy, and only route requests to those. That is what the load balancer in the article did, and the effect was that it overloaded the other backends to the point of failure (and this is a somewhat common failure mode). With a panic threshold set, when that many backends become unhealthy, the balancer stops using health as a routing criterion. It will knowingly send some requests to an unhealthy backend. This means that the healthy backends will receive traffic load that they can handle, so at least (healthy/total)% of requests will be handled successfully (instead of causing a cascading failure). Finally, other posts mention a common case like running ab against apache/mysql/php on a small machine. The OOM eventually kicks in and starts killing things. Luckily, people are also more careful on that front now. Envoy, for example, has the overload manager, so you can configure exactly how much memory you are going to use, and what happens when you get close to the limits. For my personal site, I use 64M of RAM for Envoy, and when it gets to 99% of that, it just stops accepting new connections. This sucks, of course, but it's better than getting OOM killed entirely. (A real website would probably want to give it more than 64M of RAM, but with my benchmarking I can't get anywhere close with 8000 requests/second going through it... and I'm never going to see that kind of load.) I guess the TL;DR is that in 2011 it sounded scary to have a "suicide pact" but now it's normal. Sometimes you've got to kill yourself to save others. If you're a web app, that is.
- rrmoelker 6y ago
  While not actively developing infrastructure myself, I've always like the concept presented in the Hystrix package: https://github.com/Netflix/Hystrix https://github.com/Netflix/Hystrix Even though it seems it is no longer maintained, the circuit breakers, fail over modes and all that are well documented. And I don't know why Hystrix hasn't been adopted by a wide audience yet. It seems like a necessity in the micro service landscape.
rjkennedy98 6y ago
HA is definitely super tricky. Not many products do it well. One of the last NoSQL databases I used for instance was quicker to restart than for failover to be detected so DBAs would just restart the cluster instead of waiting for failover to happen during an upgrade.
hangonhn 6y ago
Oh boy! I had a similar cascading failure situation once with a Nagios "cluster" I inherited. The previous engineer distributed the work between a master and 3 slave nodes with a backup mechanism such that if any of the slaves died, the load would go to the master. This was fine when he first created it but as more slaves were added, the master was running at capacity just dealing with the incoming data. So each each additional slave node, the probability of one of them failing and sending its load to overwhelm the master increased. Sometimes a poorly designed distributed system is worse than a single big server. I ended up leveraging Consul to do leadership election (only for the alerting bit) and monitor the health of all the nodes in the cluster. If one of them failed, the load was redistributed equally among the remaining nodes.

I found it interesting because such a simple task (requiring at least a number of on-line servers before the load-balancer starts serving requests), required a custom binary controlled the webserver and had to cross-monitor each server. For example with HAProxy (my favorite load-balancer and HTTP "router") this can be easily achieved by using `nbsrv`, creating an ACL and only routing requests to the backend based on that ACL. Based on the documentation bellow: * http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#7.3.1-nbsrv http://cbonte.github.io/haproxy-dconv/2.1/configuration.html... * http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-monitor%20fail http://cbonte.github.io/haproxy-dconv/2.1/configuration.html... One can write this: ~~~~ frontend www mode http acl site_alive nbsrv(dynamic) gt 2 use_backend dynamic if site_alive ~~~~ [This article was linked from the original article described in (https://news.ycombinator.com/item?id=23099347 https://news.ycombinator.com/item?id=23099347).]

jefftk 6y ago
Things are a lot easier these days, mostly because there's been more time for people to have problems and code up good generalizable solutions
- ciprian_craciun 6y ago
  I just wanted to mention that I didn't fault the original author for the proposed solution. It was 2011, they seemed to use some commercial load-balancer, and the customer didn't seem to want to actually solve the solution. I actually found the original solution interesting, and I also proposed an alternative achievable with HAProxy as an load-balancer.
  - dumbfounder 6y ago
    This was written in 2011, I think it's something that happened many years before. Sounds more like a 2001 thing to do.
wrkronmiller 6y ago
That config looks like it kills the site if there are 2 or fewer back-end servers (similar to the setup in the article). Wouldn't it make more sense simply to limit the number of connections per back-end server?
- ciprian_craciun 6y ago
  Indeed if I were to configure the load-balancer I would limit both the number of concurrent requests and the number of queued requests. (In fact this is what I do in my HAProxy deployments.) However I assume that the customer (the original author was referring to) didn't want to apply any limitations. (Why is beyond my reasoning...)

klausjensen 6y ago

Lovely hack and an example of how thinking outside the box can create solutions, that are order of magnitude cheaper than the "obvious" solution.

MaxBarraclough 6y ago

> The load now rebalanced to four remaining machines is just far too big, and they all die as a result. Perhaps I'm missing something terribly obvious here, but why would that happen? I can understand requests being dropped and processing-times worsening, but a full system-wide crash? edit My bad, I'd missed this in the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't, and so now I have a story

sneak 6y ago
System latency is not linear with load. At a certain threshold it starts swapping way beyond the capacity of the disk i/o and basically nothing gets done. I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.
- MaxBarraclough 6y ago
  Why doesn't the server just drop incoming requests until it's able to handle them again? As I mentioned in another comment, this is what routers do. edit I hadn't noticed this at the bottom of the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
  - rovr138 6y ago
    Apache has a way to limit the amount of processes. Tuning that would have helped.
brabel 6y ago
This will easily happen with systems that do not have any kind of upper limits on what they _try_ to do. For example: not limiting your thread pool size so that your server might try to create more threads than the system has memory for, dying due to lack of memory. Similar things can happen even if you're using async IO as you still need memory for all those stacks being run asynchronously.

nicbou 6y ago

This is called a cascading failure. It's also a problem with the electric grid, and more terrifyingly with global finance. https://en.wikipedia.org/wiki/Cascading_failure https://en.wikipedia.org/wiki/Cascading_failure

Beltiras 6y ago
I'd imagine this is the next couple of years in finance due to COVID. Shocks and reverberations for a long long while.
- nicbou 6y ago
  Travel, immigration, entertainment, restaurants etc are severely affected. A lot of people have lost their income, and spend less. There is also some feat that this will exacerbate problems with an already stressed financial system. Banks are massively increasing their cash reserves, and many organisations have a grim outlook on the upcoming months.
  - nvahalik 6y ago
    There may also end up being a potential flip-side to this. Many people (dubbed essential) have found that last couple of months that the lack of open businesses + increased pay have meant that in a few months they are going to be in the market for larger purchases (i.e. houses, cars, etc.). Their savings have gone up and at least person I've talked too has mentioned opening up a new business after all of this is over.
joncrane 6y ago
This is a really interesting failure pattern and has led to the development of so-called "circuit breakers" to avoid the cascade. (Many of the recent (last ~5 years) AWS failures have been due to cascading failures and it's neat to read their postmortems).
DaiPlusPlus 6y ago
The cascading failure of NYC’s electricity grid in 1965 was the background of episode 1 of Connections.

londons_explore 6y ago

A better solution would be to simply configure the loadbalancer to have a maximum number of requests per second per endpoint and then to drop any requests over that. An even better loadbalancer will poll a load endpoint, representing CPU load, queue length, percentage of time GC'ing, or some similar metric, and scale back requests as that metric gets too high.

MichaelApproved 6y ago
Maybe their “load balancer” was just a simple round robin service which wouldn’t have those abilities.

ninkendo 6y ago

Wait, why are the servers “crashing” when under too much load in the first place? If there’s some sort of natural limit to how many simultaneous connections they can handle, why can’t they just return some 4xx error code for connections beyond that? (And have clients implement an exponential back off?) Or if that’s too difficult, the load balancer could keep track of some maximum number of connections (or even requests per second) each backend is capable of, and throttle (again with some 4xx error code) when the limit has been reached by all backends? This is pretty basic functionality for load balancers. You’re going to need actual congestion control anyway, when the number of client connections is unbounded like this. Even when your servers aren’t crashing, what if the client apps whose clicks you’re tracking becomes suddenly more popular and you can’t handle the load even with all of your servers up?

laurentdc 6y ago
> If there’s some sort of natural limit to how many simultaneous connections they can handle Yup, it's pretty much a DDoS. It's simple to test. Spin up Wordpress + Woocommerce (a common ecommerce stack) in a Digitalocean 1 GB droplet. Now ab -n 1000 -c 30 the home page, that's 30 concurrent clients Watch MySQL die, Apache get killed because out of memory, and... root@wootest01:~# reboot -bash: fork: Cannot allocate memory
- ninkendo 6y ago
  So, maybe fix that? Why is Apache continuing to fork new workers ad infinitum? That’s a denial of service attack waiting to happen, and the answer surely isn’t “oh let’s just automate the rebooting”... Edit: you’ve edited your comment to say it’s a DoS as well, so looks like we’re on the same page here, the software is garbage.
  - laurentdc 6y ago
    > the software is garbage. Yeah, I agree in many cases that's the issue, as the author pointed out in the last paragraph. Let me also add that on this particular stack this happens with caching enabled (W3 Total Cache). Without cache it struggles with 10 concurrent clients (TTFB > 5 seconds). 100 MB per client to serve what's pretty much a grid of products and a button, it's scary

mgkimsal 6y ago

was it just a cost thing that would prevent people from just adding another server in to the mix? given that 4 was the magic number, add another server or two to add buffer to time between servers dying and 'it all breaks'? I'm realizing the cost factor may have been it, depending on size/location/etc. - would there be any other reason?

Beltiras 6y ago
Might have been tried and not worked. Might have been a limitation in scaling (f.ex. only being able to do master-slave replication and not being able to add more master nodes). Remember: it's bad software to begin with.
jasonlotito 6y ago
Not necessarily the reason here, but to answer your question: "would there be any other reason?" Another reason is backend constraints. Adding another server might simply cause more pressure on backend systems like a database. So they can't simply add more servers, as then the additional load on the database might cause that to crash, and result in even longer downtime.

Random_ernest 6y ago

I am not a webdev, but isn't that a task for the loadbalancer in the first place?

ciprian_craciun 6y ago
Unfortunately the load-balancer is not a magic-bullet curing every issue a system has. A load-balancer can be configured to do lots of things, like for example: * limit the number of concurrent requests, and drop the others; * limit the number of concurrent requests, but queue the others (with a timeout); * distribute all requests uniformly (randomly or in round-robin fashion) to all backends; * (any combination of the above); However if the "customer" asks you to not drop or queue requests, then there is nothing the load-balancer can actually do...
- Random_ernest 6y ago
  I took it from the article that dropping requests was permitted (since that happens when all servers go down). So my assumption is still that a better solution is that the load balancer allows only a specific number of requests per server and rejects or caches the rest. I would even argue that requests being rejected is more understandable for the user than the website simply not being there for a certain time.
  - legacynl 6y ago
    I think you are right, except that the article reads as if they were using a loadbalancer that wasn't in their control (a third-party service). If you can't control your loadbalancer to not pass requests when you're overloaded, the next best thing is to keep track of it on each node, and basically do what the article described.

jrockway 6y ago

There is actually quite a bit of complexity with load balancing, but the good news is that a lot of the complexity is understood and is configurable on the load balancer. I think what Rachel calls a "suicide pact" is now commonly called a circuit breaker. After a certain number of requests fail, the load balancer simply removes all the backends for a certain period of time, and causes all requests to immediately fail. This attempts to mitigate the cascading failure by simply isolating the backend from the frontend for a period of time. If you have something like a "stateless" web-app that shares a database with the other replicas, and the database stops working, this is exactly what you want. No replica will be able to handle the request, so don't send it to any replica. Another option to look into is the balancer's "panic threshold". Normally your load balancer will see which backends are healthy, and only route requests to those. That is what the load balancer in the article did, and the effect was that it overloaded the other backends to the point of failure (and this is a somewhat common failure mode). With a panic threshold set, when that many backends become unhealthy, the balancer stops using health as a routing criterion. It will knowingly send some requests to an unhealthy backend. This means that the healthy backends will receive traffic load that they can handle, so at least (healthy/total)% of requests will be handled successfully (instead of causing a cascading failure). Finally, other posts mention a common case like running ab against apache/mysql/php on a small machine. The OOM eventually kicks in and starts killing things. Luckily, people are also more careful on that front now. Envoy, for example, has the overload manager, so you can configure exactly how much memory you are going to use, and what happens when you get close to the limits. For my personal site, I use 64M of RAM for Envoy, and when it gets to 99% of that, it just stops accepting new connections. This sucks, of course, but it's better than getting OOM killed entirely. (A real website would probably want to give it more than 64M of RAM, but with my benchmarking I can't get anywhere close with 8000 requests/second going through it... and I'm never going to see that kind of load.) I guess the TL;DR is that in 2011 it sounded scary to have a "suicide pact" but now it's normal. Sometimes you've got to kill yourself to save others. If you're a web app, that is.

rrmoelker 6y ago
While not actively developing infrastructure myself, I've always like the concept presented in the Hystrix package: https://github.com/Netflix/Hystrix https://github.com/Netflix/Hystrix Even though it seems it is no longer maintained, the circuit breakers, fail over modes and all that are well documented. And I don't know why Hystrix hasn't been adopted by a wide audience yet. It seems like a necessity in the micro service landscape.

rjkennedy98 6y ago

HA is definitely super tricky. Not many products do it well. One of the last NoSQL databases I used for instance was quicker to restart than for failover to be detected so DBAs would just restart the cluster instead of waiting for failover to happen during an upgrade.

hangonhn 6y ago

Oh boy! I had a similar cascading failure situation once with a Nagios "cluster" I inherited. The previous engineer distributed the work between a master and 3 slave nodes with a backup mechanism such that if any of the slaves died, the load would go to the master. This was fine when he first created it but as more slaves were added, the master was running at capacity just dealing with the incoming data. So each each additional slave node, the probability of one of them failing and sending its load to overwhelm the master increased. Sometimes a poorly designed distributed system is worse than a single big server. I ended up leveraging Consul to do leadership election (only for the alerting bit) and monitor the health of all the nodes in the cluster. If one of them failed, the load was redistributed equally among the remaining nodes.