6 ms·

Wait, why are the servers “crashing” when under too much load in the first place? If there’s some sort of natural limit to how many simultaneous connections th

Wait, why are the servers “crashing” when under too much load in the first place? If there’s some sort of natural limit to how many simultaneous connections they can handle, why can’t they just return some 4xx error code for connections beyond that? (And have clients implement an exponential back off?) Or if that’s too difficult, the load balancer could keep track of some maximum number of connections (or even requests per second) each backend is capable of, and throttle (again with some 4xx error code) when the limit has been reached by all backends? This is pretty basic functionality for load balancers. You’re going to need actual congestion control anyway, when the number of client connections is unbounded like this. Even when your servers aren’t crashing, what if the client apps whose clicks you’re tracking becomes suddenly more popular and you can’t handle the load even with all of your servers up?

laurentdc 7y ago
> If there’s some sort of natural limit to how many simultaneous connections they can handle Yup, it's pretty much a DDoS. It's simple to test. Spin up Wordpress + Woocommerce (a common ecommerce stack) in a Digitalocean 1 GB droplet. Now ab -n 1000 -c 30 the home page, that's 30 concurrent clients Watch MySQL die, Apache get killed because out of memory, and... root@wootest01:~# reboot -bash: fork: Cannot allocate memory
- ninkendo 7y ago
  So, maybe fix that? Why is Apache continuing to fork new workers ad infinitum? That’s a denial of service attack waiting to happen, and the answer surely isn’t “oh let’s just automate the rebooting”... Edit: you’ve edited your comment to say it’s a DoS as well, so looks like we’re on the same page here, the software is garbage.
  - laurentdc 7y ago
    > the software is garbage. Yeah, I agree in many cases that's the issue, as the author pointed out in the last paragraph. Let me also add that on this particular stack this happens with caching enabled (W3 Total Cache). Without cache it struggles with 10 concurrent clients (TTFB > 5 seconds). 100 MB per client to serve what's pretty much a grid of products and a button, it's scary
    - nicbou 7y ago
      By default, Apache is rather poorly optimised. MySQL is even worse. WordPress' poor performance is the result of mashing components together without any concern for performance. WordPress itself is not garbage (at least not for those reasons), but WordPress + a bunch of plugins hammered together is a different beast.
  - pjc50 7y ago
    There's an Apache setting to limit that, or at least there was when I last had to deal with this problem 20 years ago on webservers that had less processing power than today's midrange smartphones. The problem is you have to manually tune the number of max workers a bit based on estimated resource usage.
    - lathiat 7y ago
      Part of the problem is that you usually need a lot more Apache workers to process static file requests. But limit the number of expensive PHP invocations This was pretty much not easily possible (with suphp, mod_php) until FPM which has a per site worker pool. Solved a lot of problems for us as a shared hosting provider.
      - ColinWright 7y ago
        When did that happen? I'm interested in the timeline.
        atsaloli 7y ago
        I dealt with a similar issue in 2001. About 6 mid-range Sun servers (like small refrigerators in size), running Netscape Application Server, with the load distributed using round-robin DNS (A record round-robin). Due to some DNS servers caching the A record longer than intended, it would create uneven distribution of load -- too much on one server and it'd go bad, requests would start taking too long and it'd start throwing 500 errors or just wouldn't respond. Once it stopped accepting connections, other servers would take on its load (the client would retry to the next IP in the round-robin) but then similar thing would happen to the fifth server, and the rest would follow shortly thereafter in a death spiral. Restarting was a bear. We had to "warm up" the application server by hitting some of the web pages so caches would get populated etc., before opening the system to public traffic (which, as I recall, we controlled with IP address aliases) or else it would die again. Finally we put the system behind an F5 load balancer which resulted in a much more even distribution of traffic -- that, coupled with the "warm up" crawl, let to a greatly stabilized system and highest ever (and growing) page views.
  - ColinWright 7y ago
    > ... the software is garbage. I try to be slow to jump to that conclusion ... so often there are reasons for things to be the way they are. The software tools and the hardware capabilities were very, very different in the aughties, and many people currently writing software don't really seem to appreciate by just how much. It may be the case that things could have been done differently, perhaps better, but then again, once you know the full story, maybe not. Some time ago I wrote up a war story from the mid-nineties and had some current software people crap all over it. When I started to explain about the machine limitations of the time the bluster increased, and among the replies I got was "The software was crap". Ever since then I've been interested in the contexts for these stories. So often I hear "Well you shouldn't have done it like that!" rather than an enquiring: "OK, let's assume some clever people wrote this, I wonder what the pressures were that made them come out with that solution." I've learned a lot by approaching things that way, instead of simply declaring: The software was garbage.
    - DaiPlusPlus 7y ago
      > “OK, let's assume some clever people wrote this, I wonder what the pressures were that made them come out with that solution." Pressure from everyone to get something shipped and out-the-door - never-mind the technical debt - and after it ships management is only interested in further growth fuelled by more features - deepening the technical debt. As for the problem of Apache keeling over too easily - that’s because Apache (at the time - and I think still now) is based on “one thread per HTTP request” - or one thread per connection. Asynchronous (or rather: coroutine-based) request handling was very esoteric in the late-1990s and early-2000s - and required fundamentally different program architecture - no-one was going to rewrite their CGI programs to do that - and Perl script based applications couldn’t do anything at all. Asynchronous HTTP request handling only really became mainstream when NodeJS started getting popular about 7-8 years ago - but, to my knowledge, besides ASP.NET Core - no other web application platform treats asynchronous request handling as a first-class feature.
      - pdonis 7y ago
        > Asynchronous HTTP request handling only really became mainstream when NodeJS started getting popular about 7-8 years ago Nginx first came out in 2002. It was being used widely by something like 2008. > no other web application platform treats asynchronous request handling as a first-class feature Huh? Web application platforms that do this have been around since the late 1990s. (Heck, there was one written in Python in the late 1990s, it was called "Medusa".) They just weren't "popular" back then.
Neil44 7y ago
Exactly this, limit your apache processes to match the ram you have available so the machine returns 4xx or 5xx to excess requests but otherwise remains responsive and normal. Problem solved. The options are right there in the conf. Start with a super low limit then slowly increase while watching reqs/sec. When reqs/sec stops increasing or you run out of ram then there’s your sweet spot.
jonhohle 7y ago
This. Least-conns load balancing with max-conns per host allows this type of failure to be handled as gracefully as possible without human intervention or apps needing to share state. It amazes me that these aren’t defaults on very popular load balancing platforms in 2020.

laurentdc 7y ago

> If there’s some sort of natural limit to how many simultaneous connections they can handle Yup, it's pretty much a DDoS. It's simple to test. Spin up Wordpress + Woocommerce (a common ecommerce stack) in a Digitalocean 1 GB droplet. Now ab -n 1000 -c 30 the home page, that's 30 concurrent clients Watch MySQL die, Apache get killed because out of memory, and... root@wootest01:~# reboot -bash: fork: Cannot allocate memory

ninkendo 7y ago
So, maybe fix that? Why is Apache continuing to fork new workers ad infinitum? That’s a denial of service attack waiting to happen, and the answer surely isn’t “oh let’s just automate the rebooting”... Edit: you’ve edited your comment to say it’s a DoS as well, so looks like we’re on the same page here, the software is garbage.
- laurentdc 7y ago
  > the software is garbage. Yeah, I agree in many cases that's the issue, as the author pointed out in the last paragraph. Let me also add that on this particular stack this happens with caching enabled (W3 Total Cache). Without cache it struggles with 10 concurrent clients (TTFB > 5 seconds). 100 MB per client to serve what's pretty much a grid of products and a button, it's scary
  - nicbou 7y ago
    By default, Apache is rather poorly optimised. MySQL is even worse. WordPress' poor performance is the result of mashing components together without any concern for performance. WordPress itself is not garbage (at least not for those reasons), but WordPress + a bunch of plugins hammered together is a different beast.
- pjc50 7y ago
  There's an Apache setting to limit that, or at least there was when I last had to deal with this problem 20 years ago on webservers that had less processing power than today's midrange smartphones. The problem is you have to manually tune the number of max workers a bit based on estimated resource usage.
  - lathiat 7y ago
    Part of the problem is that you usually need a lot more Apache workers to process static file requests. But limit the number of expensive PHP invocations This was pretty much not easily possible (with suphp, mod_php) until FPM which has a per site worker pool. Solved a lot of problems for us as a shared hosting provider.
    - ColinWright 7y ago
      When did that happen? I'm interested in the timeline.
      - atsaloli 7y ago
        I dealt with a similar issue in 2001. About 6 mid-range Sun servers (like small refrigerators in size), running Netscape Application Server, with the load distributed using round-robin DNS (A record round-robin). Due to some DNS servers caching the A record longer than intended, it would create uneven distribution of load -- too much on one server and it'd go bad, requests would start taking too long and it'd start throwing 500 errors or just wouldn't respond. Once it stopped accepting connections, other servers would take on its load (the client would retry to the next IP in the round-robin) but then similar thing would happen to the fifth server, and the rest would follow shortly thereafter in a death spiral. Restarting was a bear. We had to "warm up" the application server by hitting some of the web pages so caches would get populated etc., before opening the system to public traffic (which, as I recall, we controlled with IP address aliases) or else it would die again. Finally we put the system behind an F5 load balancer which resulted in a much more even distribution of traffic -- that, coupled with the "warm up" crawl, let to a greatly stabilized system and highest ever (and growing) page views.
- ColinWright 7y ago
  > ... the software is garbage. I try to be slow to jump to that conclusion ... so often there are reasons for things to be the way they are. The software tools and the hardware capabilities were very, very different in the aughties, and many people currently writing software don't really seem to appreciate by just how much. It may be the case that things could have been done differently, perhaps better, but then again, once you know the full story, maybe not. Some time ago I wrote up a war story from the mid-nineties and had some current software people crap all over it. When I started to explain about the machine limitations of the time the bluster increased, and among the replies I got was "The software was crap". Ever since then I've been interested in the contexts for these stories. So often I hear "Well you shouldn't have done it like that!" rather than an enquiring: "OK, let's assume some clever people wrote this, I wonder what the pressures were that made them come out with that solution." I've learned a lot by approaching things that way, instead of simply declaring: The software was garbage.
  - DaiPlusPlus 7y ago
    > “OK, let's assume some clever people wrote this, I wonder what the pressures were that made them come out with that solution." Pressure from everyone to get something shipped and out-the-door - never-mind the technical debt - and after it ships management is only interested in further growth fuelled by more features - deepening the technical debt. As for the problem of Apache keeling over too easily - that’s because Apache (at the time - and I think still now) is based on “one thread per HTTP request” - or one thread per connection. Asynchronous (or rather: coroutine-based) request handling was very esoteric in the late-1990s and early-2000s - and required fundamentally different program architecture - no-one was going to rewrite their CGI programs to do that - and Perl script based applications couldn’t do anything at all. Asynchronous HTTP request handling only really became mainstream when NodeJS started getting popular about 7-8 years ago - but, to my knowledge, besides ASP.NET Core - no other web application platform treats asynchronous request handling as a first-class feature.
    - pdonis 7y ago
      > Asynchronous HTTP request handling only really became mainstream when NodeJS started getting popular about 7-8 years ago Nginx first came out in 2002. It was being used widely by something like 2008. > no other web application platform treats asynchronous request handling as a first-class feature Huh? Web application platforms that do this have been around since the late 1990s. (Heck, there was one written in Python in the late 1990s, it was called "Medusa".) They just weren't "popular" back then.

Neil44 7y ago

Exactly this, limit your apache processes to match the ram you have available so the machine returns 4xx or 5xx to excess requests but otherwise remains responsive and normal. Problem solved. The options are right there in the conf. Start with a super low limit then slowly increase while watching reqs/sec. When reqs/sec stops increasing or you run out of ram then there’s your sweet spot.

jonhohle 7y ago

This. Least-conns load balancing with max-conns per host allows this type of failure to be handled as gracefully as possible without human intervention or apps needing to share state. It amazes me that these aren’t defaults on very popular load balancing platforms in 2020.