3 ms·

Even IT professionals can't tell the difference between latency and bandwidth, or capacity and speed. A simple rule of thumb is: If a single user experiences p

Even IT professionals can't tell the difference between latency and bandwidth, or capacity and speed. A simple rule of thumb is: If a single user experiences poor performance with your otherwise idle cluster of servers, then adding more servers will not help. You can't imagine how often I have to have this conversation with developers, devops people, architects, business owners, etc... "Let's just double the cores and see what happens." "Let's not, it'll just double your costs and do nothing to improve things." Also another recent conversation: "Your network security middlebox doesn't use a proper network stack and is adding 5 milliseconds of latency to all datacentre communications." "We can scale out by adding more instances if capacity is a concern!" "That's... not what I said."

JohnMakin 2y ago
It’s astounding how many people that work in infrastructure should understand things like this but don’t, particularly network bottlenecks or bottlenecks in general. I’ve seen situations where someone wants to increase the number of replicas for some service because the frontend is 504’ing, but the real reason is because the database has become saturated by the calls from the service. It is possible (a little unlikely, but possible, and the rule with infra at scale is “unlikely” always becomes “certain”) to actually make the problem worse by scaling up there. The number of blank stares I get when explaining things like this is demoralizing sometimes, especially in consulting situations where you have some pig headed engineer manager that thinks he knows everything about everything.
- dtaht 2y ago
  Being one of the authors of fq_codel, cake and to a small extent, LibreQos, I remain boggled after 15 years of trying, to get these simple points across.. Have some laughs: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/ https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-...
ipython 2y ago
As they say, if you’re getting impatient for your baby to arrive, just get more pregnant ladies together! The cluster of pregnant women make the process move along quicker! /s
- codesections 2y ago
  And the "they" in question is Warren Buffet, https://nymag.com/intelligencer/2009/06/you_cant_make_a_baby_in_a_mont.html https://nymag.com/intelligencer/2009/06/you_cant_make_a_baby...
  - vitus 2y ago
    I thought this was first attributed to Fred Brooks in the 70s. > Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, "nine women can't make a baby in one month". https://en.wikipedia.org/wiki/Brooks%27s_law https://en.wikipedia.org/wiki/Brooks%27s_law
    - fragmede 2y ago
      You can't take a random 9 women and have a baby in one month, but that makes me wonder, statistically, how many women from the whole population would you need to get to 1 month? 9 women can't make a baby in one month, but given 9 million women, the chances are, one of them are giving birth right now. if I needed a baby tomorrow, what do the statistics say on how many women it would take to have a baby, tomorrow? taking the population of the earth and the birth rate and doing some math, you get around to needing 12,000 women of reproductive age for you to have a baby tomorrow. 12,000 is a lot of women! it's well above Dunbar's number. think about that, next time the 9 women one month baby topic comes up.
dtaht 2y ago
I share your pain. I really really really share your pain.
floating-io 2y ago
That is the perfect high-level overview of my previous job. I'm semi-retired now... (edit: I forgot to note that the "let's not" part was always overridden by "You're wrong, this will fix it. Do it!" by management. Then we would eventually find and fix the actual problem (because it didn't go away), but the cluster size -- and the cost -- would remain because "No, it was too slow with so few replicas".)
- ozim 2y ago
  You know that management get bonus points for blaming developers for increasing it at the end of year when annual budget review comes by.
  - jiggawatts 2y ago
    I literally had the same manager complaining to me about his cloud bill just months after rejecting my suggestion for more efficient hosting options (still in the cloud, just less wasteful.)
hinkley 2y ago
“Just try it anyway.”
ozim 2y ago
I can see how increasing resources can be useful as "try to stop the bleeding RIGHT NOW!", but yeah if a single user has issues if there is not much other activity that's like doing CPR on a person that is screaming "stop, my foot is bleeding" and "hey man we will try this first, stop screaming" ;)

JohnMakin 2y ago

It’s astounding how many people that work in infrastructure should understand things like this but don’t, particularly network bottlenecks or bottlenecks in general. I’ve seen situations where someone wants to increase the number of replicas for some service because the frontend is 504’ing, but the real reason is because the database has become saturated by the calls from the service. It is possible (a little unlikely, but possible, and the rule with infra at scale is “unlikely” always becomes “certain”) to actually make the problem worse by scaling up there. The number of blank stares I get when explaining things like this is demoralizing sometimes, especially in consulting situations where you have some pig headed engineer manager that thinks he knows everything about everything.

dtaht 2y ago
Being one of the authors of fq_codel, cake and to a small extent, LibreQos, I remain boggled after 15 years of trying, to get these simple points across.. Have some laughs: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/ https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-...

ipython 2y ago

As they say, if you’re getting impatient for your baby to arrive, just get more pregnant ladies together! The cluster of pregnant women make the process move along quicker! /s

codesections 2y ago
And the "they" in question is Warren Buffet, https://nymag.com/intelligencer/2009/06/you_cant_make_a_baby_in_a_mont.html https://nymag.com/intelligencer/2009/06/you_cant_make_a_baby...
- vitus 2y ago
  I thought this was first attributed to Fred Brooks in the 70s. > Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, "nine women can't make a baby in one month". https://en.wikipedia.org/wiki/Brooks%27s_law https://en.wikipedia.org/wiki/Brooks%27s_law
  - fragmede 2y ago
    You can't take a random 9 women and have a baby in one month, but that makes me wonder, statistically, how many women from the whole population would you need to get to 1 month? 9 women can't make a baby in one month, but given 9 million women, the chances are, one of them are giving birth right now. if I needed a baby tomorrow, what do the statistics say on how many women it would take to have a baby, tomorrow? taking the population of the earth and the birth rate and doing some math, you get around to needing 12,000 women of reproductive age for you to have a baby tomorrow. 12,000 is a lot of women! it's well above Dunbar's number. think about that, next time the 9 women one month baby topic comes up.

dtaht 2y ago

I share your pain. I really really really share your pain.

floating-io 2y ago

That is the perfect high-level overview of my previous job. I'm semi-retired now... (edit: I forgot to note that the "let's not" part was always overridden by "You're wrong, this will fix it. Do it!" by management. Then we would eventually find and fix the actual problem (because it didn't go away), but the cluster size -- and the cost -- would remain because "No, it was too slow with so few replicas".)

ozim 2y ago
You know that management get bonus points for blaming developers for increasing it at the end of year when annual budget review comes by.
- jiggawatts 2y ago
  I literally had the same manager complaining to me about his cloud bill just months after rejecting my suggestion for more efficient hosting options (still in the cloud, just less wasteful.)

hinkley 2y ago

“Just try it anyway.”

ozim 2y ago

I can see how increasing resources can be useful as "try to stop the bleeding RIGHT NOW!", but yeah if a single user has issues if there is not much other activity that's like doing CPR on a person that is screaming "stop, my foot is bleeding" and "hey man we will try this first, stop screaming" ;)