3 ms·

Another of those somewhat counter-intuitive results is the answer to the question "how much do we need to scale up to avoid response time regression when the re

by kqr 2y ago

Another of those somewhat counter-intuitive results is the answer to the question "how much do we need to scale up to avoid response time regression when the request rate is doubled?" It is very easy to blurt out "well obviously we need twice the processing power!" but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. To many people that sounds weird; it sounds like we got something for nothing. If I invite twice as many people to a party and buy twice as many cookies, it's not like each guest will get twice as many cookies – that just leaves the originally planned number of cookies for each guest. But for response time it comes back to the first equation in TFA: T = 1/μ · 1/(1 - ρ) Doubling both arrival rate and maximum service rate leaves ρ – and the second factor with it – unchanged, but still halves the 1/μ factor, resulting in half the response time. The appropriate amount to scale by is the k that solves the equation we get when we set the old response time T at request rate λ equal to the one at request rate 2λ and kμ processing power. This is something like T = 1/μ · 1/(1 - λ/μ) = 1/kμ · 1/(1 - 2λ/kμ) but rearranges to the much simpler k = ρ + 1 which a colleague of mine told me interprets intuitively as "the processing power that needs to be added is exactly that which will handle an additional unit of the current utilisation on top of the current utilisation, i.e. twice as much." This is mostly good news for people doing capacity planning in advance of events etc. If you run your systems at reasonable utilisation levels normally, you don't actually need that much additional capacity to handle peak loads.

CaptainNegative 2y ago
A simple way to view the first statement is that by both doubling the request rate and halving the processing time, you're effectively speeding up time by a factor of 2. If you were to record a video of the entire system, someone switching the parameters as you describe and someone putting the recorded video on a 2x fast forward would be fundamentally indistinguishable. So, naturally, the response time gets cut down on half as well.
steveBK123 2y ago
> This is mostly good news for people doing capacity planning in advance of events etc. If you run your systems at reasonable utilisation levels normally, you don't actually need that much additional capacity to handle peak loads. Stop using math that will put the hyperscalers/serverless/microservices bros out of business! /s
krisoft 2y ago
> It is very easy to blurt out "well obviously we need twice the processing power!" but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. I don't understand what you are saying. Are you talking about the time the request is buffered in some queue on average assuming they are arriving randomly? Or something like that? Here is what I'm thinking. We are operating a hospital which does only one type of surgery which last an hour exactly. (Presumably it is a veterinary practice for spherical cows.) A fully staffed operating room can operate on 24 spherical cows a day. If due to a spherical cow jamboree we expect more patients and we set up a second operating theatre we will be able to serve 48 of them a day. But we are still serving them for an hour each. (because that is how long the operation takes.) Even if we are talking about latency when 24 cows show up at the stroke of midnight to the one operating room hospital they each will be served on average in 12.5h. Same average if 48 cows show up to the two operating room hospital. So what am I thinking wrong here? Is "scale to twice the processing power" not the same as getting a second operating room? I'm not seeing where "we will actually be serving each request in half the time" comes from.
- lesuorac 2y ago
  > So what am I thinking wrong here? Is "scale to twice the processing power" not the same as getting a second operating room? I'm not seeing where "we will actually be serving each request in half the time" comes from. Single Core vs Multi Core (ish). With a Single thread you must work twice as fast to handle the increased load which also means the work is done at half the speed. With Multi threading you can shuffle two units of work out at the same time so it's twice the load but the same speed. To go back to the cow analogy, rather than adding 24 rooms (more threads) you give each surgeon a powersaw and they work twice as fast. So if you scale the processing power per thread up then the time goes down, if you scale the processing power by adding threads (cores) the time stays the same (ish).
  - krisoft 2y ago
    > To go back to the cow analogy, rather than adding 24 rooms (more threads) you give each surgeon a powersaw and they work twice as fast. I was thinking that maybe that is what we are talking about. But convinced myself otherwise. Surely we don't need that much math to show if we cut the processing time in half the processing time will be cut in half. But if that is all we are saying I guess I can accept that as quite trivial.
mgerdts 2y ago
> but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. Not necessarily. If processing power is increased by doubling the clock of a processor or using a hard disk that spins and seeks twice as fast, this may be the case. But we all know that when you have a single threaded work item, adding a second core does not cause the single threaded work to complete in half the time. If the arrival rate is substantially lower than 1/S, the second core will be of negligible value, and maybe of negative value due to synchronization overhead. This overhead is unlikely to be seen when doubling from 1 to 2, but is more likely at high levels of scaling. If the processor is saturated, service time includes queue time, and service time is dominated by queue time, increasing processing power by doubling the number of processors may make it so that service time can almost be cut in half. How close to half depends on the ratio of queue time to processing time.
- kqr 2y ago
  This is a useful addition. Yes, the above reasoning was assuming that we (a) had located a bottleneck, (b) were planning to vertically scale the capacity of that bottleneck, and (c) in doing so won't run into a different bottleneck! It is still useful for many cases of horizontal scaling because from sufficiently far away, c identical servers looks a lot like a single server at c times the capacity of a single one. Many applications I encounter in practise does not require one to be very far away to make that simplifying assumption. > How close to half depends on the ratio of queue time to processing time. I don't think this is generally true, but I think I see what you're going for: you're using queue length as a proxy for how far away we have to be to pretend horizontal scaling is a useful proxy for vertical scaling, right?
kaashif 2y ago
> but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. People usually add processing power by adding more parallelism - more machines, VMs, pods, whatever. In this case, the "blurted out" answer is correct. If I take one second to serve a request on a machine then I add another machine and start serving twice the requests, the first machine doesn't get faster. Maybe what you're saying is true if you make your CPUs twice as fast, but that's not usually possible on a whim.
- kqr 2y ago
  It can still be true when scaling horizontally, depending on utilisation levels and other system characteristics, as well as the economics of errors. Statistically speaking, I more often find the blurted out answer to be further from truth than 1 + rho.
callamdelaney 2y ago
Ah yes The Tenant Farmers Association's (TFA) famous first equation.. Wait that does't sound right, Trifluoroacetic acid? Trusted Financial Advice? Tropical Forest Alliance? Two Factor Auth?
- IncreasePosts 2y ago
  If you google for "TFA meaning", which I'm sure you did, you should have seen the first result, which is a link to hackernews where someone is complaining about TFA not being defined.

by kqr 2y ago

CaptainNegative 2y ago

A simple way to view the first statement is that by both doubling the request rate and halving the processing time, you're effectively speeding up time by a factor of 2. If you were to record a video of the entire system, someone switching the parameters as you describe and someone putting the recorded video on a 2x fast forward would be fundamentally indistinguishable. So, naturally, the response time gets cut down on half as well.

steveBK123 2y ago

> This is mostly good news for people doing capacity planning in advance of events etc. If you run your systems at reasonable utilisation levels normally, you don't actually need that much additional capacity to handle peak loads. Stop using math that will put the hyperscalers/serverless/microservices bros out of business! /s

krisoft 2y ago

> It is very easy to blurt out "well obviously we need twice the processing power!" but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. I don't understand what you are saying. Are you talking about the time the request is buffered in some queue on average assuming they are arriving randomly? Or something like that? Here is what I'm thinking. We are operating a hospital which does only one type of surgery which last an hour exactly. (Presumably it is a veterinary practice for spherical cows.) A fully staffed operating room can operate on 24 spherical cows a day. If due to a spherical cow jamboree we expect more patients and we set up a second operating theatre we will be able to serve 48 of them a day. But we are still serving them for an hour each. (because that is how long the operation takes.) Even if we are talking about latency when 24 cows show up at the stroke of midnight to the one operating room hospital they each will be served on average in 12.5h. Same average if 48 cows show up to the two operating room hospital. So what am I thinking wrong here? Is "scale to twice the processing power" not the same as getting a second operating room? I'm not seeing where "we will actually be serving each request in half the time" comes from.

lesuorac 2y ago
> So what am I thinking wrong here? Is "scale to twice the processing power" not the same as getting a second operating room? I'm not seeing where "we will actually be serving each request in half the time" comes from. Single Core vs Multi Core (ish). With a Single thread you must work twice as fast to handle the increased load which also means the work is done at half the speed. With Multi threading you can shuffle two units of work out at the same time so it's twice the load but the same speed. To go back to the cow analogy, rather than adding 24 rooms (more threads) you give each surgeon a powersaw and they work twice as fast. So if you scale the processing power per thread up then the time goes down, if you scale the processing power by adding threads (cores) the time stays the same (ish).
- krisoft 2y ago
  > To go back to the cow analogy, rather than adding 24 rooms (more threads) you give each surgeon a powersaw and they work twice as fast. I was thinking that maybe that is what we are talking about. But convinced myself otherwise. Surely we don't need that much math to show if we cut the processing time in half the processing time will be cut in half. But if that is all we are saying I guess I can accept that as quite trivial.

mgerdts 2y ago

> but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. Not necessarily. If processing power is increased by doubling the clock of a processor or using a hard disk that spins and seeks twice as fast, this may be the case. But we all know that when you have a single threaded work item, adding a second core does not cause the single threaded work to complete in half the time. If the arrival rate is substantially lower than 1/S, the second core will be of negligible value, and maybe of negative value due to synchronization overhead. This overhead is unlikely to be seen when doubling from 1 to 2, but is more likely at high levels of scaling. If the processor is saturated, service time includes queue time, and service time is dominated by queue time, increasing processing power by doubling the number of processors may make it so that service time can almost be cut in half. How close to half depends on the ratio of queue time to processing time.

kqr 2y ago
This is a useful addition. Yes, the above reasoning was assuming that we (a) had located a bottleneck, (b) were planning to vertically scale the capacity of that bottleneck, and (c) in doing so won't run into a different bottleneck! It is still useful for many cases of horizontal scaling because from sufficiently far away, c identical servers looks a lot like a single server at c times the capacity of a single one. Many applications I encounter in practise does not require one to be very far away to make that simplifying assumption. > How close to half depends on the ratio of queue time to processing time. I don't think this is generally true, but I think I see what you're going for: you're using queue length as a proxy for how far away we have to be to pretend horizontal scaling is a useful proxy for vertical scaling, right?

kaashif 2y ago

> but if we scale to twice the processing power, then start accepting twice the request rate – we will actually be serving each request in half the time we originally did. People usually add processing power by adding more parallelism - more machines, VMs, pods, whatever. In this case, the "blurted out" answer is correct. If I take one second to serve a request on a machine then I add another machine and start serving twice the requests, the first machine doesn't get faster. Maybe what you're saying is true if you make your CPUs twice as fast, but that's not usually possible on a whim.

kqr 2y ago
It can still be true when scaling horizontally, depending on utilisation levels and other system characteristics, as well as the economics of errors. Statistically speaking, I more often find the blurted out answer to be further from truth than 1 + rho.

callamdelaney 2y ago

Ah yes The Tenant Farmers Association's (TFA) famous first equation.. Wait that does't sound right, Trifluoroacetic acid? Trusted Financial Advice? Tropical Forest Alliance? Two Factor Auth?

IncreasePosts 2y ago
If you google for "TFA meaning", which I'm sure you did, you should have seen the first result, which is a link to hackernews where someone is complaining about TFA not being defined.