5 ms·
And if memory serves if you care about minimizing latency you want all of your workers running an average of 60% occupied. (Which is also pretty close to when I
by hinkley 2y ago
And if memory serves if you care about minimizing latency you want all of your workers running an average of 60% occupied. (Which is also pretty close to when I saw P95 times dog-leg on the last cluster I worked on).
Queuing theory is really weird.
- deepsun 2y agoWhy 60%? I suppose if they are less than 1% then latency will be even less.
- hinkley 2y agoPoisson could probably explain it elegantly, but he's not here, just us.
- atombender 2y agoMost analyses I've read say the threshold is around the 80% mark [1], although it depends on how model the distribution, and there's nothing magical about the number. The main thing is to avoid getting close to 100%, because wait times go up exponentially as you get closer to the max. Little's Law is fundamental to queueing theory, but there's also the less well-known Kingman's formula, which incorporates variability of arrival rate and task size [2]. [1] https://www.johndcook.com/blog/2009/01/30/server-utilization-joel-on-queuing/ https://www.johndcook.com/blog/2009/01/30/server-utilization... [2] https://taborsky.cz/posts/2021/kingman-formula/ https://taborsky.cz/posts/2021/kingman-formula/
- hinkley 2y agoIf you look at the chart in the second link, where does the wait time leave the origin? Around 60%. The first one is even worse; by 80% you're already seeing twice the delay of 70%. If I were to describe the second chart I'd say 80% is when you start to get into trouble, not just noticing a slowdown. I said minimize latency, not optimize latency.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- dude187 2y agoReally both of those models show 60% as about the limit to where you're still effectively at the baseline for latency. 80% is just about the limit to where you're up there in the exponential rise, any higher and things become unusable. 0-60 and you're still at minimum latency. 60-80 you're at twice the latency but it's probably worth the cost savings of the extra compute density since it's still pretty low. Higher than 80 and things are already slowing down and getting exponentially worse by the request
- gunian 2y agodog food dog leg dog ram lol
- hinkley 2y agodog leg is what people who aren't pretentious prats call an 'inflection point'.
- bubblyworld 2y agoWhat do people who aren't pretentious prats call "latency" and "quantiles"? =P
- TeMPOraL 2y agoFirst time I ever hear of this term, even though it's apparently applied to everything from pixel graphics to traffic engineering to rocket launches. I mean, you US-ians seem obsessed with dog legs to an unhealthy degree[0]. -- [0] - https://en.wikipedia.org/wiki/Dogleg https://en.wikipedia.org/wiki/Dogleg
- winrid 2y agoSomehow I was building distributed systems earlier in my career before I learned about queuing theory and learned this the hard way. Nowadays with DB stuff I tend to get assigned new infra leads who see a DB cluster at 50% CPU utilization and think they can go down two instances sizes without severely impacting latency.
- hinkley 2y agoFor me it was seeing a machine at 70% utilization and thinking they could squeeze another small service onto the box. The first time it didn’t sound right, and after that I knew it was a bad idea. By the third time I was willing to throw a veto if they wouldn’t listen to reason. And the thing is, even if you stuff a low priority service onto a bunch of boxes, and convince the OS to honor that priority fairly, the fact that the service runs reasonably at all gets baked in as an expectation. Maybe it’s the hedonic treadmill, one kid expects dessert with every meal because they’ve always gotten it, and another knows it’s a special occasion. But anything given is jealously guarded when you have to take it away. Even a “best effort” batch process that is supposed to finish on some interval, is missed when it no longer does. And somehow it’s always your fault. I’m sure the grocery store employees who are assigned as backup tellers constantly get grief for not getting their other tasks done “on time”.
- hinkley 2y agoThis is, by the way, one of those problems the Cloud solved, because management couldn’t use friction with the procurement and operations departments to try to pin you into oversubscribing machines. Cloud computing usually leaves the barn door open so that you can requisition another server and ask forgiveness later. And I think at some level developers all know this, so we went along with the dope deal. Because fuck Operations and Accounting for making us the scapegoats to problems they cause.
- lurking_swe 2y agoAny resources you’d recommend to learn more about this? for context: 10 yrs experience as a software engineer and only a couple on high traffic products. Some things (like this) continue to surprise me.
- hiAndrewQuinn 2y agoPeep https://yzr95924.github.io/pdf/book/Basic-Queueing-Theory.pdf https://yzr95924.github.io/pdf/book/Basic-Queueing-Theory.pd... , honestly just being aware of it is already half the battle.
- hinkley 2y agoThe more examples you read the easier you can recall this stuff. I find the single to two teller examples very compelling myself. How quickly a line can build up and how long it lasts when people/tasks are arriving at 90% of capacity and you get one burst of extra traffic is quite shocking the first time you understand the math.