5 ms·

Here's another article about the same issue I think https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html . Solution: "

Here's another article about the same issue I think https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html . Solution: "Step 1. Identify the bottleneck. Step 2: ask the bottleneck for permission to pile more data in"

cecilpl2 5y ago
This is a fantastic article, thank you for sharing!
hammock 5y ago
Love that solution. It's plainly unfair but in my experience so critical to getting things done in business, even setting aside engineering.
bombcar 5y ago
This is wonderful and shows the problem very clearly. If your system doesn't have a way to shed load, it will eventually overload. The problem for many is that when it does finally shed load, the load shedder gets blamed (and turned off). See how many people look for ways to turn off the OOMKiller and how few look to figure out how to get more RAM.
- dcow 5y ago
  This is also wonderful (emphasis mine): > All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more!
prometheus76 5y ago
I work in a custom fabrication environment, and this solution doesn't really apply, because what happens in a dynamic system is that the bottleneck shifts all around the shop dynamically. It's never just one operation that is the constant bottleneck.
- owenmarshall 5y ago
  How doesn't it? I've lived in a world of coordinated microservices before, where a nice linear progression of queues only happens at the atomic level; when you zoom out you've got a digraph full of cycles, nodes that come & go in response to various events & schedules, and the transitory nature of those services entering could cause a smooth-sailing pipeline to suddenly seize up because some downstream dependency is servicing higher-priority requests. The only way we could stay sane was aggressively following Ferd's rule – nobody got to surface an API unless it surfaced HTTP 429s on overload[1], and if you called a downstream API without checking for and responding to a 429, tough noogies, your requests got dropped and you had to figure out what to do to fix it. Funny enough, at the time I did software for a manufacturing facility that did high-throughput manufacturing for many varying products. I was lucky enough to do a "gemba walk" where they made us engineers put on hard-hats and eye protection and reminded us who actually made the company money, and by god those guys were doing exactly what we did: they had JIT delivery of raw materials / sub-components to assembly lines, and the delivery of resources to the line was always constrained by a fixed buffer, quite literally "how many units you could put on a shelf". You couldn't just toss more product at a line that wasn't ready for it, it would literally not fit. Sure, they used different terms than we did – muda and muri and value-stream mapping – but the outcome was very similar. Their version of the NOC was a team of managers who watched the cycle time of each line, how long it took material to be transported, which areas were bottlenecking and which were at risk of running into resource starvation, and they had the ability to balance the hot spots by bursting product in, caching upstream components into hold docks, releasing held work product to downstream lines, or even turning on a flex line to work around longer-term constraints. They rarely had fixed, static bottlenecks, because they optimized the life out of them. Their bottlenecks arose when a shipment didn't arrive (and you better believe those were tracked just as aggressively), a machine broke down, an operator got sick, things like that. But at the end of the day? Same principle: downstream components would only accept as much as they could process from upstream components; you asked before you pushed. [1]: and if you said "I swear, you can't overload this service, we don't need to send 429s" a swarm of senior engineers descended upon you and only left until you convinced them they were wrong, or – way more likely – they convinced you.
  - 5y ago

cecilpl2 5y ago

This is a fantastic article, thank you for sharing!

hammock 5y ago

Love that solution. It's plainly unfair but in my experience so critical to getting things done in business, even setting aside engineering.

bombcar 5y ago

This is wonderful and shows the problem very clearly. If your system doesn't have a way to shed load, it will eventually overload. The problem for many is that when it does finally shed load, the load shedder gets blamed (and turned off). See how many people look for ways to turn off the OOMKiller and how few look to figure out how to get more RAM.

dcow 5y ago
This is also wonderful (emphasis mine): > All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more!

prometheus76 5y ago

I work in a custom fabrication environment, and this solution doesn't really apply, because what happens in a dynamic system is that the bottleneck shifts all around the shop dynamically. It's never just one operation that is the constant bottleneck.

owenmarshall 5y ago
How doesn't it? I've lived in a world of coordinated microservices before, where a nice linear progression of queues only happens at the atomic level; when you zoom out you've got a digraph full of cycles, nodes that come & go in response to various events & schedules, and the transitory nature of those services entering could cause a smooth-sailing pipeline to suddenly seize up because some downstream dependency is servicing higher-priority requests. The only way we could stay sane was aggressively following Ferd's rule – nobody got to surface an API unless it surfaced HTTP 429s on overload[1], and if you called a downstream API without checking for and responding to a 429, tough noogies, your requests got dropped and you had to figure out what to do to fix it. Funny enough, at the time I did software for a manufacturing facility that did high-throughput manufacturing for many varying products. I was lucky enough to do a "gemba walk" where they made us engineers put on hard-hats and eye protection and reminded us who actually made the company money, and by god those guys were doing exactly what we did: they had JIT delivery of raw materials / sub-components to assembly lines, and the delivery of resources to the line was always constrained by a fixed buffer, quite literally "how many units you could put on a shelf". You couldn't just toss more product at a line that wasn't ready for it, it would literally not fit. Sure, they used different terms than we did – muda and muri and value-stream mapping – but the outcome was very similar. Their version of the NOC was a team of managers who watched the cycle time of each line, how long it took material to be transported, which areas were bottlenecking and which were at risk of running into resource starvation, and they had the ability to balance the hot spots by bursting product in, caching upstream components into hold docks, releasing held work product to downstream lines, or even turning on a flex line to work around longer-term constraints. They rarely had fixed, static bottlenecks, because they optimized the life out of them. Their bottlenecks arose when a shipment didn't arrive (and you better believe those were tracked just as aggressively), a machine broke down, an operator got sick, things like that. But at the end of the day? Same principle: downstream components would only accept as much as they could process from upstream components; you asked before you pushed. [1]: and if you said "I swear, you can't overload this service, we don't need to send 429s" a swarm of senior engineers descended upon you and only left until you convinced them they were wrong, or – way more likely – they convinced you.
- 5y ago