8 ms·

Efficiency trades off against resiliency

124 points by tim_sw 4y ago

omginternets 4y ago
Now apply this same reasoning to, say, hospital capacity. Some things need to optimize for the ability to absorb peak load, not steady-state operating costs.
- cmilton 4y ago
  How is peak load defined in a hospital? (I assume you mean patients. I apologize if that is incorrect.)
  - kuhewa 4y ago
    There will be different definitions for related concepts but I reckon the parent comment is referring to "surge capacity"
  - SECProto 4y ago
    > How is peak load defined in a hospital? Same as everywhere else - largest demand on resources. For a hospital, the resources are staffing and equipment (rather than cpu/storage/ram). An MRI machine is cheapest when you have as many scans as possible for a given level of staffing - reducing the time per scan from 6mins to 5mins means that you can get 20 extra patients scanned per 8 hour shift (with one machine). In support of the initial commenter's point ("Some things need to optimize for the ability to absorb peak load, not steady-state operating costs"), for hospitals, you need look no further than covid - horrendous health outcomes for routine things, because the capacity was soaked up dealing with respiratory virus. And if you disagree with that (you're wrong), look no further than a strong flu season - staffing capacity that is totally fine for a random august is nowhere near adequate. Or look at Saturday night in any Emerg - there's a huge peak in emergency medical situations from drunk people doing stupid things.
  - quickthrower2 4y ago
    For a public health system you run the stats, using the $ cost of a life (or equivalent for a well life vs. sick life). Sounds sinister but ultimately they have so many $ and need to decide how to optimise that money.
    - bumby 4y ago
      >optimise that money Perhaps the problem is defining a hospitals utility function as “optimizing money” rather than “optimizing a patient outcome”. I agree that money is definitely a constraint, but I’m not sure it’s what should be optimized.
    - LastTrain 4y ago
      You do that for a non-public health system too, but you also figure in margins.
- kqr 4y ago
  Even hospitals make that trade-off. Every hundred years there are at least a few events that overload steady-state hospitals. The important part is not building the capacity in, it's remaining flexible and adapting to what happens.
  - omginternets 4y ago
    >The important part is not building the capacity in, it's remaining flexible and adapting to what happens. At some point, flexibility requires extra capacity. You can't improvise when you're spending 100% of your resources on operations.
bsder 4y ago
Maximally efficient is minimally robust.
- Tagbert 4y ago
  This also applies to supply chains and economies in the whole.
  - bsder 4y ago
    And biology and ...
  - 1bent 4y ago
    Indeed, when I saw the headline, I thought this was going to be commenting on the various supply chain crashes triggered by the pandemic's sudden changes, after decades of cleverly squeezing slack out of shipping, ports, warehouses, and logistics coordination.
- closeparen 4y ago
  Catastrophic failure is pretty bad for efficiency. Over any serious time horizon, being maximally efficient means finding the optimal level of robustness, given the likelihoods and consequences of possible failures and the costs involved in preventing or mitigating them.
  - bumby 4y ago
    The problem is the incentives of the decision makers often don’t align with that time horizon. Consider a CEO who is rewarded by quarterly outcomes rather than how healthy the company will be in two or three decades. Or a politician who proposes a short-term policy that looks like a short term win but will undermine constituents after they are long out of office.
  - bsder 4y ago
    > Catastrophic failure is pretty bad for efficiency. For the system, but not necessarily for any individual actors in that system. There is no inherent force that automatically keeps those incentive aligned. Everybody knows that the supply chains are brittle. Covid, earthquakes, war, etc. all can disrupt them. Yet, do you see anybody holding inventory? Do you see any diversification occurring (very minimally--and mostly to shift from cheap Chinese labor to cheaper Vietnamese/Indonesian labor)? Do you see anybody buying personal protective equipment domestically in the US (nope, everybody went back to buying cheap Chinese crap)?
defrost 4y ago
Yes I code, a few million+ SLOC in geophysics | imaging | GIS | et al, but more and more I consult on policy with companies and local to Federal government in Australia on the back of having an engineering | applied math background. Agricultural and farmers might seem inefficient but they add national resiliency. Pre COVID meat packing in the USofA may have been hyper efficient on paper but it broke hard under stress; too many concentrated assembley lines that put too many people too close together in too few places in total. Day to day one of my favorite redundancies is the triple bowline for multiple anchors - under load if one anchor fails the knot rebalances load to other anchors and slowly contracts the "dead" loop with no sudden jerk. [TEXT]: https://www.survivalworld.com/knots/triple-bowline/ https://www.survivalworld.com/knots/triple-bowline/ [VIDEO]: https://www.youtube.com/watch?v=O81l4ss4Dqk https://www.youtube.com/watch?v=O81l4ss4Dqk
HALtheWise 4y ago
While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can perform a mix of time-critical work (serving queries) and background tasks (indexing, etc). If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, and I suspect it's almost never the case that a large service sees an unexpected 30% load increase in less time than it takes to boot another machine, so such a system could be both efficient and robust with few downsides. Implementing that system isn't as easy as just throwing more Kubernetes nodes at your problem, especially given that the ecosystem of tooling isn't designed to make it easy. For example, it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level. Perhaps the real lesson is that "many of the easiest ways to achieve robustness involve sacrificing large amounts of performance", but I reject the idea that we should use that as an excuse to accept terrible performance, with all the monetary and environmental impacts it brings.
- kqr 4y ago
  > If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks." As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are. Your proposal only works if - Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower); - You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or - The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)
kqr 4y ago
Many of these examples focus on the wrong kind of efficiency. There's efficiency from the production perspective -- how large a fraction of time are my resources busy adding value? Then there's efficiency from a consumption perspective -- how large a fraction of time am I receiving value? The first kind of efficiency is inflexible, brittle, and associated with generated demand, clearly. What about the second kind? In my experience, properly optimising for consumption-side efficiency leads to more resilience, but I'm willing to be wrong here.
chrsjxn 4y ago
100% is an odd choice for a utilization target. Just about every service I've ever monitored has had some fairly clear inflection points where higher utilization starts to affect performance. It could be things like more time spent on garbage collection, or just unlucky collisions with async work that make things take longer.
- steve_gh 4y ago
  Basic queueing theory says 100% load is not possible with any sort of variance in the arrival rate or service rate. Realistically 70% is a good starting point for a max continuous load - then tune it. You might end up a bit lower or higher
pclmulqdq 4y ago
On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't have to go all the way to raw structs to gain efficiency over using JSON. Only the last tiny bit of efficiency (the gap between a struct and a flatbuffer) actually comes at a meaningful resiliency cost. The same goes for single-threaded vs multi-threaded services and single-machine vs horizontal scaling. With some thought, you can do all of these things and create highly-scalable systems very efficiently - not quite as efficiently as a single thread, but a lot more efficiently than the standard "web backend" thing of creating 1000 microservices using 10 different databases and 50 caching layers for that single task. That efficiency is available at no cost to resiliency. The common thread of "web" solutions like microservices and JSON is that they help you save developer time. That is orthogonal to the efficiency vs resiliency tradeoff.
- anonymoushn 4y ago
  This comes up a lot, do you have any benchmarks for protobuf deserialization that demonstrate that it is faster than json for the same structs? Protobuf seems much slower to decode due to the inability to know where an integer ends before one is done decoding it. Benchmarks I could find indicated protobuf decoding is maybe ~5x slower than json, but obviously this varies based on the payload and I'm not sure how much larger json encodings of the same data tend to be.
  - akvadrako 4y ago
    How would you know where an integer in JSON ends without looking ahead? You won't even know if it is an integer or not. It seems like a more expensive operation than protobuf.
    - anonymoushn 4y ago
      You use about 2 SIMD instructions to compare 16 or 32 or 64 characters to whitespace, commas, and closing brackets, and one of these will be the character after the end of the integer literal. Theres no dependency between decoding the integer and tokenizing or decoding the next thing.
thoughtsbykean 4y ago
and at this point using an autoscaling solution, provided by a big cloud service is ideal.
thoughtsbykean 4y ago
However, the comparison about efficiency to resiliency against small teams is something I havn't thought about. Interesting point about declaring a system is wasteful and inefficient without considering what that 'waste' is buying.
regularfry 4y ago
Some of these examples are trivially equivalent to queues and queueing theory tells us that 100% utilisation results in unbounded queue lengths, and is therefore bad. The interesting thing would be to try to see if these other situations that on the surface sort of smell similar (serialisation and distribution systems, for instance) are actually isomorphic to the extent that the same proofs apply.
bullen 4y ago
The only interesting challenge is non-trivially parallelizable multicore applications that need to share memory across threads. And the only choice you need to make is arrays of 64 byte structures C OR Java. In other words do you need to be able to sleep after a day of launching new feaures? The way I tend to do these things is to prototype with Java and once the concept/protocol is finalized and fossilized; rewrite it in C atleast on the client. Java is the best language for server platforms, with classloader hot-deployment for sub second global turnaround and no crashes VM+GC. About services being infinitely accessible: companies that have non-trivial solutions are now moving away from this with queues. Limit the number of concurrent users and when saturated customers have to wait. Open-source/source-available is the best way to scale though. If people are prepared to pay, others will run you service. You need to allow others to make money though. I recommend revenue scaled monthly licence fees recurring on something like gumroad. Finally I'm pretty sure the new 4nm chips will proove to be more fragile than the 14nm stack, only alot of 100% CPU time will tell.
hsn915 4y ago
The articles starts out with the premise absolutely backwards on its head: efficiency is the exact opposite of constant 100% CPU utilization. The more efficient the program is, the _less_ CPU resources it will consume to achieve its task. Specially when that task is serving http requests. Unless you are always serving one million concurrent connections with one servers, you should never see chronic 100% CPU utilization.
didgetmaster 4y ago
>developers or operators can use that slack to step in and handle unexpected load or resolve underlying issues before they become catastrophic or externally visible. They can use 'slack' to pay off some technical debt that tends to build up when developers are always pushed to devote 100% of their time and effort to new features. There is often no time to go back and fix things that can bite you when the load reaches a certain level.
angarg12 4y ago
The 100% CPU utilization is an odd one because, although theoretically it's more "efficient" to use a system at full capacity, performance tends to suffer and practically make it unusable (others' pointed here to queueing theory). We are building a high load, low latency inference system, and we noticed that latency is correlated to CPU usage. Latency grows slowly linearly with CPU usage, up to a point where it starts to grow exponentially. We had a bug where a host was pegged at 100% CPU, and our tail latency suffered greatly, since any request that hit that host was basically toast. Funnily enough OP could have used examples outside computing where efficiency directly trades off with resiliency. For example, in farming large monocultures are the most efficient, but a single disease can wipe entire plantations. In manufacturing, keeping stock is seen as an inefficiency, but problems in the supply chain can get you in hot water (as COVID made painfully obvious).

124 points by tim_sw 4y ago

Now apply this same reasoning to, say, hospital capacity. Some things need to optimize for the ability to absorb peak load, not steady-state operating costs.

cmilton 4y ago
How is peak load defined in a hospital? (I assume you mean patients. I apologize if that is incorrect.)
- kuhewa 4y ago
  There will be different definitions for related concepts but I reckon the parent comment is referring to "surge capacity"
- SECProto 4y ago
  > How is peak load defined in a hospital? Same as everywhere else - largest demand on resources. For a hospital, the resources are staffing and equipment (rather than cpu/storage/ram). An MRI machine is cheapest when you have as many scans as possible for a given level of staffing - reducing the time per scan from 6mins to 5mins means that you can get 20 extra patients scanned per 8 hour shift (with one machine). In support of the initial commenter's point ("Some things need to optimize for the ability to absorb peak load, not steady-state operating costs"), for hospitals, you need look no further than covid - horrendous health outcomes for routine things, because the capacity was soaked up dealing with respiratory virus. And if you disagree with that (you're wrong), look no further than a strong flu season - staffing capacity that is totally fine for a random august is nowhere near adequate. Or look at Saturday night in any Emerg - there's a huge peak in emergency medical situations from drunk people doing stupid things.
- quickthrower2 4y ago
  For a public health system you run the stats, using the $ cost of a life (or equivalent for a well life vs. sick life). Sounds sinister but ultimately they have so many $ and need to decide how to optimise that money.
  - bumby 4y ago
    >optimise that money Perhaps the problem is defining a hospitals utility function as “optimizing money” rather than “optimizing a patient outcome”. I agree that money is definitely a constraint, but I’m not sure it’s what should be optimized.
  - LastTrain 4y ago
    You do that for a non-public health system too, but you also figure in margins.
kqr 4y ago
Even hospitals make that trade-off. Every hundred years there are at least a few events that overload steady-state hospitals. The important part is not building the capacity in, it's remaining flexible and adapting to what happens.
- omginternets 4y ago
  >The important part is not building the capacity in, it's remaining flexible and adapting to what happens. At some point, flexibility requires extra capacity. You can't improvise when you're spending 100% of your resources on operations.

bsder 4y ago

Maximally efficient is minimally robust.

Tagbert 4y ago
This also applies to supply chains and economies in the whole.
- bsder 4y ago
  And biology and ...
- 1bent 4y ago
  Indeed, when I saw the headline, I thought this was going to be commenting on the various supply chain crashes triggered by the pandemic's sudden changes, after decades of cleverly squeezing slack out of shipping, ports, warehouses, and logistics coordination.
closeparen 4y ago
Catastrophic failure is pretty bad for efficiency. Over any serious time horizon, being maximally efficient means finding the optimal level of robustness, given the likelihoods and consequences of possible failures and the costs involved in preventing or mitigating them.
- bumby 4y ago
  The problem is the incentives of the decision makers often don’t align with that time horizon. Consider a CEO who is rewarded by quarterly outcomes rather than how healthy the company will be in two or three decades. Or a politician who proposes a short-term policy that looks like a short term win but will undermine constituents after they are long out of office.
- bsder 4y ago
  > Catastrophic failure is pretty bad for efficiency. For the system, but not necessarily for any individual actors in that system. There is no inherent force that automatically keeps those incentive aligned. Everybody knows that the supply chains are brittle. Covid, earthquakes, war, etc. all can disrupt them. Yet, do you see anybody holding inventory? Do you see any diversification occurring (very minimally--and mostly to shift from cheap Chinese labor to cheaper Vietnamese/Indonesian labor)? Do you see anybody buying personal protective equipment domestically in the US (nope, everybody went back to buying cheap Chinese crap)?

defrost 4y ago

Yes I code, a few million+ SLOC in geophysics | imaging | GIS | et al, but more and more I consult on policy with companies and local to Federal government in Australia on the back of having an engineering | applied math background. Agricultural and farmers might seem inefficient but they add national resiliency. Pre COVID meat packing in the USofA may have been hyper efficient on paper but it broke hard under stress; too many concentrated assembley lines that put too many people too close together in too few places in total. Day to day one of my favorite redundancies is the triple bowline for multiple anchors - under load if one anchor fails the knot rebalances load to other anchors and slowly contracts the "dead" loop with no sudden jerk. [TEXT]: https://www.survivalworld.com/knots/triple-bowline/ https://www.survivalworld.com/knots/triple-bowline/ [VIDEO]: https://www.youtube.com/watch?v=O81l4ss4Dqk https://www.youtube.com/watch?v=O81l4ss4Dqk

HALtheWise 4y ago

While I generally agree that the pattern this article describes is real (there's some degree of tradeoff between robustness and efficiency), I've frequently seen engineers fall back on that logic rather than actually thinking about the specific problem they're facing, and spending five minutes trying to come up with a creative solution. For example, talking about the CPU utilization of a web service, your service can perform a mix of time-critical work (serving queries) and background tasks (indexing, etc). If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, and I suspect it's almost never the case that a large service sees an unexpected 30% load increase in less time than it takes to boot another machine, so such a system could be both efficient and robust with few downsides. Implementing that system isn't as easy as just throwing more Kubernetes nodes at your problem, especially given that the ecosystem of tooling isn't designed to make it easy. For example, it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level. Perhaps the real lesson is that "many of the easiest ways to achieve robustness involve sacrificing large amounts of performance", but I reject the idea that we should use that as an excuse to accept terrible performance, with all the monetary and environmental impacts it brings.

kqr 4y ago
> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks." As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are. Your proposal only works if - Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower); - You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or - The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)

kqr 4y ago

Many of these examples focus on the wrong kind of efficiency. There's efficiency from the production perspective -- how large a fraction of time are my resources busy adding value? Then there's efficiency from a consumption perspective -- how large a fraction of time am I receiving value? The first kind of efficiency is inflexible, brittle, and associated with generated demand, clearly. What about the second kind? In my experience, properly optimising for consumption-side efficiency leads to more resilience, but I'm willing to be wrong here.

chrsjxn 4y ago

100% is an odd choice for a utilization target. Just about every service I've ever monitored has had some fairly clear inflection points where higher utilization starts to affect performance. It could be things like more time spent on garbage collection, or just unlucky collisions with async work that make things take longer.

steve_gh 4y ago
Basic queueing theory says 100% load is not possible with any sort of variance in the arrival rate or service rate. Realistically 70% is a good starting point for a max continuous load - then tune it. You might end up a bit lower or higher

pclmulqdq 4y ago

On the one hand, this is somewhat true, but on the other hand, there are lots of efficiencies you can get without trading resiliency, and the author sets up a few false dichotomies here. The author mentions JSON vs struct serialization as an example, but flatbuffers and protobufs (or any binary protocol format that builds in an ID and a version number) give you the same resiliency benefits mentioned here. You don't have to go all the way to raw structs to gain efficiency over using JSON. Only the last tiny bit of efficiency (the gap between a struct and a flatbuffer) actually comes at a meaningful resiliency cost. The same goes for single-threaded vs multi-threaded services and single-machine vs horizontal scaling. With some thought, you can do all of these things and create highly-scalable systems very efficiently - not quite as efficiently as a single thread, but a lot more efficiently than the standard "web backend" thing of creating 1000 microservices using 10 different databases and 50 caching layers for that single task. That efficiency is available at no cost to resiliency. The common thread of "web" solutions like microservices and JSON is that they help you save developer time. That is orthogonal to the efficiency vs resiliency tradeoff.

anonymoushn 4y ago
This comes up a lot, do you have any benchmarks for protobuf deserialization that demonstrate that it is faster than json for the same structs? Protobuf seems much slower to decode due to the inability to know where an integer ends before one is done decoding it. Benchmarks I could find indicated protobuf decoding is maybe ~5x slower than json, but obviously this varies based on the payload and I'm not sure how much larger json encodings of the same data tend to be.
- akvadrako 4y ago
  How would you know where an integer in JSON ends without looking ahead? You won't even know if it is an integer or not. It seems like a more expensive operation than protobuf.
  - anonymoushn 4y ago
    You use about 2 SIMD instructions to compare 16 or 32 or 64 characters to whitespace, commas, and closing brackets, and one of these will be the character after the end of the integer literal. Theres no dependency between decoding the integer and tokenizing or decoding the next thing.

thoughtsbykean 4y ago

and at this point using an autoscaling solution, provided by a big cloud service is ideal.

thoughtsbykean 4y ago

However, the comparison about efficiency to resiliency against small teams is something I havn't thought about. Interesting point about declaring a system is wasteful and inefficient without considering what that 'waste' is buying.

regularfry 4y ago

Some of these examples are trivially equivalent to queues and queueing theory tells us that 100% utilisation results in unbounded queue lengths, and is therefore bad. The interesting thing would be to try to see if these other situations that on the surface sort of smell similar (serialisation and distribution systems, for instance) are actually isomorphic to the extent that the same proofs apply.

bullen 4y ago

The only interesting challenge is non-trivially parallelizable multicore applications that need to share memory across threads. And the only choice you need to make is arrays of 64 byte structures C OR Java. In other words do you need to be able to sleep after a day of launching new feaures? The way I tend to do these things is to prototype with Java and once the concept/protocol is finalized and fossilized; rewrite it in C atleast on the client. Java is the best language for server platforms, with classloader hot-deployment for sub second global turnaround and no crashes VM+GC. About services being infinitely accessible: companies that have non-trivial solutions are now moving away from this with queues. Limit the number of concurrent users and when saturated customers have to wait. Open-source/source-available is the best way to scale though. If people are prepared to pay, others will run you service. You need to allow others to make money though. I recommend revenue scaled monthly licence fees recurring on something like gumroad. Finally I'm pretty sure the new 4nm chips will proove to be more fragile than the 14nm stack, only alot of 100% CPU time will tell.

hsn915 4y ago

The articles starts out with the premise absolutely backwards on its head: efficiency is the exact opposite of constant 100% CPU utilization. The more efficient the program is, the _less_ CPU resources it will consume to achieve its task. Specially when that task is serving http requests. Unless you are always serving one million concurrent connections with one servers, you should never see chronic 100% CPU utilization.

didgetmaster 4y ago

>developers or operators can use that slack to step in and handle unexpected load or resolve underlying issues before they become catastrophic or externally visible. They can use 'slack' to pay off some technical debt that tends to build up when developers are always pushed to devote 100% of their time and effort to new features. There is often no time to go back and fix things that can bite you when the load reaches a certain level.

angarg12 4y ago

The 100% CPU utilization is an odd one because, although theoretically it's more "efficient" to use a system at full capacity, performance tends to suffer and practically make it unusable (others' pointed here to queueing theory). We are building a high load, low latency inference system, and we noticed that latency is correlated to CPU usage. Latency grows slowly linearly with CPU usage, up to a point where it starts to grow exponentially. We had a bug where a host was pegged at 100% CPU, and our tail latency suffered greatly, since any request that hit that host was basically toast. Funnily enough OP could have used examples outside computing where efficiency directly trades off with resiliency. For example, in farming large monocultures are the most efficient, but a single disease can wipe entire plantations. In manufacturing, keeping stock is seen as an inefficiency, but problems in the supply chain can get you in hot water (as COVID made painfully obvious).