3 ms·

> The load now rebalanced to four remaining machines is just far too big, and they all die as a result. Perhaps I'm missing something terribly obvious here, bu

> The load now rebalanced to four remaining machines is just far too big, and they all die as a result. Perhaps I'm missing something terribly obvious here, but why would that happen? I can understand requests being dropped and processing-times worsening, but a full system-wide crash? edit My bad, I'd missed this in the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't, and so now I have a story

sneak 7y ago
System latency is not linear with load. At a certain threshold it starts swapping way beyond the capacity of the disk i/o and basically nothing gets done. I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.
- MaxBarraclough 7y ago
  Why doesn't the server just drop incoming requests until it's able to handle them again? As I mentioned in another comment, this is what routers do. edit I hadn't noticed this at the bottom of the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
  - rovr138 7y ago
    Apache has a way to limit the amount of processes. Tuning that would have helped.
brabel 7y ago
This will easily happen with systems that do not have any kind of upper limits on what they _try_ to do. For example: not limiting your thread pool size so that your server might try to create more threads than the system has memory for, dying due to lack of memory. Similar things can happen even if you're using async IO as you still need memory for all those stacks being run asynchronously.
- MaxBarraclough 7y ago
  So there are two ways a server can be engineered to handle overload: 1. A throttling scheme of some sort 2. Ignore the possibility, and just crash disastrously if/when it occurs Is that right? If so, surely that's what we should be talking about here, no? (I see ninkendo mentioned it in their comment too.) Am I right in thinking the whole problem that the blog-post discusses - avoiding a small pool of servers getting overloaded and blowing up - follows from poorly engineered servers that go with Option 2? Edit This is what I get for reading the article too quickly. She addresses this quite explicitly: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
  - brabel 7y ago
    I think options 1 and 2 are exactly right. Depending on your system, you may not even have choice 1... it's quite hard actually to control a system so well that you can confidently say it will never exhaust the available resources. The JVM for example can only kill its process once its memory usage exceeds `Xmx`, but it can't prevent that from happening in the first place. If you had a few JVMs (or Python, Ruby or any other process) taking up all of your memory and the system started swapping disk like crazy, you could easily have the system crash anyway as different processes don't "throttle" or coordinate with each other. Don't assume it's an easy thing to avoid.
    - MaxBarraclough 7y ago
      > Don't assume it's an easy thing to avoid. I'm really not sold on this attitude. Ordinary exception handling may be a lot of effort, but if you don't do it, that means your work is sloppy. Same here. More generally, a program shouldn't ever outright explode, regardless of input. That's not quite the same thing as exit-with-error, but I suppose the distinction is ultimately fuzzy. > The JVM for example can only kill its process once its memory usage exceeds `Xmx`, but it can't prevent that from happening in the first place I don't see that the particulars of JVM memory-management have any bearing here. A web server should be capable of detecting when it has reached capacity, and to reject additional requests as appropriate. There's no reason this couldn't be done in Java. From a quick search, this is indeed how modern Java servers behave. If there are many communicating processes, I can imagine that could complicate things greatly. That's a downside of using many communicating processes.
Beltiras 7y ago
It's like thrashing for memory. Once you can't keep up with requests it's extra strain which inevitably will bring down the system.
- MaxBarraclough 7y ago
  With respect, that's a metaphor, not an answer. Why shouldn't the system be able to just drop incoming requests until it's able to handle them? That's what routers do.
  - Beltiras 7y ago
    I've never had to think about this. Always had the option of just opening up more capacity. My choice for application server (uWSGI) doesn't have easily found configurations for dropping connections on high load. Lots of documentation on fallbacks but no pre-emptive dropping (that I can find).
aasasd 7y ago
In the article, swap is mentioned. Apparently either processing each request requires some amount of memory, or new processes are fired up for concurrent requests (the Apache model). So, what I've seen happen in similar circumstances is that once swapping starts, some of the processes run 10x slower, while new load keeps coming in and piling up, running the server even deeper into the swap. Soon you have a thousand processes, load-average >100, and you can't run anything in the terminal. As mentioned in other comments, a proper defense against this would be to have a front-line system that anticipates such a situation and actively manages it: puts outstanding requests in a queue (throttling the load) and then errors out on new requests if there's still too many of them coming in.

sneak 7y ago

System latency is not linear with load. At a certain threshold it starts swapping way beyond the capacity of the disk i/o and basically nothing gets done. I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.

MaxBarraclough 7y ago
Why doesn't the server just drop incoming requests until it's able to handle them again? As I mentioned in another comment, this is what routers do. edit I hadn't noticed this at the bottom of the article: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
- rovr138 7y ago
  Apache has a way to limit the amount of processes. Tuning that would have helped.

brabel 7y ago

This will easily happen with systems that do not have any kind of upper limits on what they _try_ to do. For example: not limiting your thread pool size so that your server might try to create more threads than the system has memory for, dying due to lack of memory. Similar things can happen even if you're using async IO as you still need memory for all those stacks being run asynchronously.

MaxBarraclough 7y ago
So there are two ways a server can be engineered to handle overload: 1. A throttling scheme of some sort 2. Ignore the possibility, and just crash disastrously if/when it occurs Is that right? If so, surely that's what we should be talking about here, no? (I see ninkendo mentioned it in their comment too.) Am I right in thinking the whole problem that the blog-post discusses - avoiding a small pool of servers getting overloaded and blowing up - follows from poorly engineered servers that go with Option 2? Edit This is what I get for reading the article too quickly. She addresses this quite explicitly: > they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
- brabel 7y ago
  I think options 1 and 2 are exactly right. Depending on your system, you may not even have choice 1... it's quite hard actually to control a system so well that you can confidently say it will never exhaust the available resources. The JVM for example can only kill its process once its memory usage exceeds `Xmx`, but it can't prevent that from happening in the first place. If you had a few JVMs (or Python, Ruby or any other process) taking up all of your memory and the system started swapping disk like crazy, you could easily have the system crash anyway as different processes don't "throttle" or coordinate with each other. Don't assume it's an easy thing to avoid.
  - MaxBarraclough 7y ago
    > Don't assume it's an easy thing to avoid. I'm really not sold on this attitude. Ordinary exception handling may be a lot of effort, but if you don't do it, that means your work is sloppy. Same here. More generally, a program shouldn't ever outright explode, regardless of input. That's not quite the same thing as exit-with-error, but I suppose the distinction is ultimately fuzzy. > The JVM for example can only kill its process once its memory usage exceeds `Xmx`, but it can't prevent that from happening in the first place I don't see that the particulars of JVM memory-management have any bearing here. A web server should be capable of detecting when it has reached capacity, and to reject additional requests as appropriate. There's no reason this couldn't be done in Java. From a quick search, this is indeed how modern Java servers behave. If there are many communicating processes, I can imagine that could complicate things greatly. That's a downside of using many communicating processes.

Beltiras 7y ago

It's like thrashing for memory. Once you can't keep up with requests it's extra strain which inevitably will bring down the system.

MaxBarraclough 7y ago
With respect, that's a metaphor, not an answer. Why shouldn't the system be able to just drop incoming requests until it's able to handle them? That's what routers do.
- Beltiras 7y ago
  I've never had to think about this. Always had the option of just opening up more capacity. My choice for application server (uWSGI) doesn't have easily found configurations for dropping connections on high load. Lots of documentation on fallbacks but no pre-emptive dropping (that I can find).

aasasd 7y ago

In the article, swap is mentioned. Apparently either processing each request requires some amount of memory, or new processes are fired up for concurrent requests (the Apache model). So, what I've seen happen in similar circumstances is that once swapping starts, some of the processes run 10x slower, while new load keeps coming in and piling up, running the server even deeper into the swap. Soon you have a thousand processes, load-average >100, and you can't run anything in the terminal. As mentioned in other comments, a proper defense against this would be to have a front-line system that anticipates such a situation and actively manages it: puts outstanding requests in a queue (throttling the load) and then errors out on new requests if there's still too many of them coming in.