3 ms·

> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In

by kqr 4y ago

> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%, In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks." As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are. Your proposal only works if - Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower); - You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or - The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)

deleted 4y ago
[deleted]
HALtheWise 4y ago
I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so it's not inconceivable to design an on-prem infrastructure to allow significant scaling with short notice. Option 3 is sorta always true as well, since there's always compute load at your company that's lower priority, and can be shed in an emergency (for example, your CI system). In any case, I suspect the common pattern of scaling is that most companies grow in a smooth enough way that they can predict demand in future weeks within a couple percent, so you can just run at 98% CPU instead of 100%
- threeseed 4y ago
  The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.
  - ownagefool 4y ago
    Perhaps today this is true. In 2007 I was working for a webhostet / nascent iaas provider, and we had workloads doing exactly that. Specific example would be Celtic FC who had a baseline of dedicated, but would scale into our VMs during events, i.e. uefa cup games.
  - hinkley 4y ago
    Reaching into on-prem hardware from the cloud is a substantial engineering effort, and one that can land you in the news if you do it wrong.
  - HALtheWise 4y ago
    Good points. For security concerns, you would probably be selective about which services and workloads you move to cloud compute, possibly preferring to offload internal services like CI before customer data. Regarding cost, are you implying that there's large monetary cost to preserving the ability to spin up cloud workers in an emergency, or just that once you do it's expensive per day to keep them running? Presumably if you're running an on-prem cluster for cost reasons, you would work to scale it up quickly after an unexpected sustained demand spike, and the cloud supplement would be temporary. If you're getting "unexpected" spikes every week, you need to work harder at forecasting. In any case, I suspect today that people running on-prem clusters provision them with the intent of being able to handle a load growth and keep weekly average utilization at <60% during the spike. Investing in building infrastructure that keeps working smoothly when weekly average load is 90-100% means you can provision a smaller cluster either way.
  - theptip 4y ago

by kqr 4y ago

deleted 4y ago

[deleted]

HALtheWise 4y ago

I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true. If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so it's not inconceivable to design an on-prem infrastructure to allow significant scaling with short notice. Option 3 is sorta always true as well, since there's always compute load at your company that's lower priority, and can be shed in an emergency (for example, your CI system). In any case, I suspect the common pattern of scaling is that most companies grow in a smooth enough way that they can predict demand in future weeks within a couple percent, so you can just run at 98% CPU instead of 100%

threeseed 4y ago
The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens. Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.
- ownagefool 4y ago
  Perhaps today this is true. In 2007 I was working for a webhostet / nascent iaas provider, and we had workloads doing exactly that. Specific example would be Celtic FC who had a baseline of dedicated, but would scale into our VMs during events, i.e. uefa cup games.
- hinkley 4y ago
  Reaching into on-prem hardware from the cloud is a substantial engineering effort, and one that can land you in the news if you do it wrong.
- HALtheWise 4y ago
  Good points. For security concerns, you would probably be selective about which services and workloads you move to cloud compute, possibly preferring to offload internal services like CI before customer data. Regarding cost, are you implying that there's large monetary cost to preserving the ability to spin up cloud workers in an emergency, or just that once you do it's expensive per day to keep them running? Presumably if you're running an on-prem cluster for cost reasons, you would work to scale it up quickly after an unexpected sustained demand spike, and the cloud supplement would be temporary. If you're getting "unexpected" spikes every week, you need to work harder at forecasting. In any case, I suspect today that people running on-prem clusters provision them with the intent of being able to handle a load growth and keep weekly average utilization at <60% during the spike. Investing in building infrastructure that keeps working smoothly when weekly average load is 90-100% means you can provision a smaller cluster either way.
- theptip 4y ago