6 ms·

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently see

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can perform a mix of time-critical work (serving queries) and background tasks (indexing, etc). If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, and I suspect it's almost never the case that a large service sees an unexpected 30% load increase in less time than it takes to boot another machine, so such a system could be both efficient and robust with few downsides. Implementing that system isn't as easy as just throwing more Kubernetes nodes at your problem, especially given that the ecosystem of tooling isn't designed to make it easy. For example, it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level. Perhaps the real lesson is that "many of the easiest ways to achieve robustness involve sacrificing large amounts of performance", but I reject the idea that we should use that as an excuse to accept terrible performance, with all the monetary and environmental impacts it brings.

kqr 4y ago
> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks." As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are. Your proposal only works if - Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower); - You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or - The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)
- deleted 4y ago
  [deleted]
- HALtheWise 4y ago
  I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so it's not inconceivable to design an on-prem infrastructure to allow significant scaling with short notice. Option 3 is sorta always true as well, since there's always compute load at your company that's lower priority, and can be shed in an emergency (for example, your CI system). In any case, I suspect the common pattern of scaling is that most companies grow in a smooth enough way that they can predict demand in future weeks within a couple percent, so you can just run at 98% CPU instead of 100%
  - threeseed 4y ago
    The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.
    - ownagefool 4y ago
      Perhaps today this is true. In 2007 I was working for a webhostet / nascent iaas provider, and we had workloads doing exactly that. Specific example would be Celtic FC who had a baseline of dedicated, but would scale into our VMs during events, i.e. uefa cup games.
    - hinkley 4y ago
      Reaching into on-prem hardware from the cloud is a substantial engineering effort, and one that can land you in the news if you do it wrong.
threeseed 4y ago
> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less maintained than something that is industry standard.
- deleted 4y ago
  [deleted]
- remram 4y ago
  Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask. (scrape interval is 15s, pod creation requires pulling images)
  - pyth0 4y ago
    > (scrape interval is 15s, pod creation requires pulling images) You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.
- HALtheWise 4y ago
  I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in. Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.
twawaaay 4y ago
Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases. The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily. I would like to use this occasion to point out that all the discussion about unused CPU is at this time completely pointless. Most services I have seen waste ORDERS of magnitude by being inefficient. Rather than focusing on trying to saturate the CPU and other resources it is almost always better to just make your application more efficient. That last 30% should be a cherry on top.
- bluepizza 4y ago
  How do you use control theory?
  - twawaaay 4y ago
    You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes. As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I can add a delay to each response and I can have even something as simple as PID controller regulate the CPU usage by changing the dalay. Larger delay will usually cause the clients to slow down requests (requests being usually a result of previous request completing). This is simple and naive example but this is more or less what I do. (Of course, in reality, it is much better to just have a backpressure mechanism and whenever possible you should use one rather than try to work around HTTP inadequacy. But you can't always do it, especially if you have a public API.) I also typically have lots of other controllers. For example something that regulates memory usage by limiting transactions in flight or something that regulates latency as seen by priority clients or database replication rate/delay, or error rates or a bunch of other parameters. I also routinely take care of babysitting downstream systems like databases or other APIs. I may have a regulator that will automatically start backing off certain types of traffic as a response to increasing error rates or latencies in a downstream system. All this because those downstream systems are usually shit and not designed to deal with overload and it is easier for me to deal with this proactively than do what everybody else does -- keep bugging those people to fix their issues when their evidently don't know how.
silvestrov 4y ago
Beware of I/O, memory usage and cache eviction. A low-priority background job can issue I/O requests that result in high I/O latencies for everything else (by e.g. making lot of random seeks). This is very easy to get when using spinning rust.
- HALtheWise 4y ago
  If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the number of cores allocated to foreground vs background tasks (within less than your latency SLO). That means each core is only doing one kind of work for many milliseconds in a row so CPU caches remain hot, and I/O can be sorted into separate low and high priority queues. IO blocking is much less of a concern with NVMe drvies anyway, since they're intrinsically so parallel. Memory usage is still a concern, since you do need to have the code and in-memory data for both workloads loaded on some number of your servers in order to rapidly change what tasks they're doing. Physical machines have so much memory today (compared to the code size or working set of most programs) that I suspect this isn't actually a blocker in practice, and it's not necessarily required for every machine in the cluster to have both workloads resident as long as some of them do. This definitely feels like diving into an alternate universe of cluster management, and I'm not sure how easily Linux gives you tools to do things like this.
kamma4434 4y ago
A long time ago, I was surprised to discover that a mainframe always runs at 100% cpu usage. Its OS makes sure that multiple VMs, each of which has a long list of real-time and batch jobs, neatly prioritized, keep it well fed. An interactive user need an answer now, but a clean-up job can literally take weeks to run. When I looked amazed, the operators told me ‘This thing is so expensive- it cannot be wasted’

kqr 4y ago

> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks." As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are. Your proposal only works if - Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower); - You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or - The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)

deleted 4y ago
[deleted]
HALtheWise 4y ago
I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so it's not inconceivable to design an on-prem infrastructure to allow significant scaling with short notice. Option 3 is sorta always true as well, since there's always compute load at your company that's lower priority, and can be shed in an emergency (for example, your CI system). In any case, I suspect the common pattern of scaling is that most companies grow in a smooth enough way that they can predict demand in future weeks within a couple percent, so you can just run at 98% CPU instead of 100%
- threeseed 4y ago
  The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.
  - ownagefool 4y ago
    Perhaps today this is true. In 2007 I was working for a webhostet / nascent iaas provider, and we had workloads doing exactly that. Specific example would be Celtic FC who had a baseline of dedicated, but would scale into our VMs during events, i.e. uefa cup games.
  - hinkley 4y ago
    Reaching into on-prem hardware from the cloud is a substantial engineering effort, and one that can land you in the news if you do it wrong.

threeseed 4y ago

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like. And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less maintained than something that is industry standard.

deleted 4y ago
[deleted]
remram 4y ago
Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask. (scrape interval is 15s, pod creation requires pulling images)
- pyth0 4y ago
  > (scrape interval is 15s, pod creation requires pulling images) You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.
HALtheWise 4y ago
I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in. Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.

twawaaay 4y ago

Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases. The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily. I would like to use this occasion to point out that all the discussion about unused CPU is at this time completely pointless. Most services I have seen waste ORDERS of magnitude by being inefficient. Rather than focusing on trying to saturate the CPU and other resources it is almost always better to just make your application more efficient. That last 30% should be a cherry on top.

bluepizza 4y ago
How do you use control theory?
- twawaaay 4y ago
  You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes. As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I can add a delay to each response and I can have even something as simple as PID controller regulate the CPU usage by changing the dalay. Larger delay will usually cause the clients to slow down requests (requests being usually a result of previous request completing). This is simple and naive example but this is more or less what I do. (Of course, in reality, it is much better to just have a backpressure mechanism and whenever possible you should use one rather than try to work around HTTP inadequacy. But you can't always do it, especially if you have a public API.) I also typically have lots of other controllers. For example something that regulates memory usage by limiting transactions in flight or something that regulates latency as seen by priority clients or database replication rate/delay, or error rates or a bunch of other parameters. I also routinely take care of babysitting downstream systems like databases or other APIs. I may have a regulator that will automatically start backing off certain types of traffic as a response to increasing error rates or latencies in a downstream system. All this because those downstream systems are usually shit and not designed to deal with overload and it is easier for me to deal with this proactively than do what everybody else does -- keep bugging those people to fix their issues when their evidently don't know how.

silvestrov 4y ago

Beware of I/O, memory usage and cache eviction. A low-priority background job can issue I/O requests that result in high I/O latencies for everything else (by e.g. making lot of random seeks). This is very easy to get when using spinning rust.

HALtheWise 4y ago
If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the number of cores allocated to foreground vs background tasks (within less than your latency SLO). That means each core is only doing one kind of work for many milliseconds in a row so CPU caches remain hot, and I/O can be sorted into separate low and high priority queues. IO blocking is much less of a concern with NVMe drvies anyway, since they're intrinsically so parallel. Memory usage is still a concern, since you do need to have the code and in-memory data for both workloads loaded on some number of your servers in order to rapidly change what tasks they're doing. Physical machines have so much memory today (compared to the code size or working set of most programs) that I suspect this isn't actually a blocker in practice, and it's not necessarily required for every machine in the cluster to have both workloads resident as long as some of them do. This definitely feels like diving into an alternate universe of cluster management, and I'm not sure how easily Linux gives you tools to do things like this.

kamma4434 4y ago

A long time ago, I was surprised to discover that a mainframe always runs at 100% cpu usage. Its OS makes sure that multiple VMs, each of which has a long list of real-time and batch jobs, neatly prioritized, keep it well fed. An interactive user need an answer now, but a clean-up job can literally take weeks to run. When I looked amazed, the operators told me ‘This thing is so expensive- it cannot be wasted’