5 ms·

Encountered a challenge like this once. Infrastructure team kept telling us it wasn't anything on their end. I coordinated with the business unit VP to serve

Encountered a challenge like this once. Infrastructure team kept telling us it wasn't anything on their end. I coordinated with the business unit VP to serve their entire QA environment from four VM's on my laptop for a day, and performance went from slow as molasses, to purring like a kitten. A couple days later their infrastructure team finally identified storage latency issues on their multi-million dollar cluster and let me help them fix it.

pixl97 3y ago
I swear IOPS/request latency is one of the least understood things in these huge companies. "But we have 40bazillion GB and 100GB LAN", cool story bro, your disk queue latency is pegged at your depth limit and your fs response latency is over a second, everything is going to suck till you deal with that.
- RajT88 3y ago
  I have had customers say this when not getting desired throughput cross-continent. "But the pipes! They are fat!" Yes but also your window needs to scale... 6 figure network engineers not knowing about window scaling, who do not know how to analyze a packet trace. I recently had someone suggest that they needed 1ms latency cross-continent. I explained patiently that the laws of physics have to change for them to hit that number. I am not even a network engineer!
  - rescbr 3y ago
    Oh, this is too common! I can't even count how many times I had to blame Einstein for setting the speed of light too low in his law. (Nitpick: I know he's the wrong person to blame, but he's famous enough to make this joke understandable by everybody) Once I had to write a root cause analysis report and it basically explained how TCP works.
- NovemberWhiskey 3y ago
  Truth. I literally have a signal on one of my monitoring dashboards which indicates "database is currently undergoing online backup", because it is the single most important performance signal I have. This doesn't come to me as a signal from the database team; I have to poll the database for it myself. Noisy neighbor in inadequately-isolated, shared-tenancy models is just the worst.
- xelxebar 3y ago
  > your disk queue latency is pegged at your depth limit and your fs response latency is over a second You said words. And I totally, 100% understand them. But, like, for the plebs that are totally not me, could you elaborate on what you're talking about here? I^WThey would like to understand.
  - stavros 3y ago
    Basically, "the disk is always terribly busy with tiny data operations".
broast 3y ago
I find it very hard to tolerate teams that say an issue isn't on them if they aren't pointing to evidence that indicates where they think the issue is. If you think it's not on you but it affects something within your responsibility, you're still on the hook until you prove it.
- Scoundreller 3y ago
  Nothing more fun than troubleshooting a fax machine incompatibility. “It’s you, we’re receiving faxes from everyone else” (how would they know?) “Nooo, it’s you, everyone else is receiving our faxes”

pixl97 3y ago

I swear IOPS/request latency is one of the least understood things in these huge companies. "But we have 40bazillion GB and 100GB LAN", cool story bro, your disk queue latency is pegged at your depth limit and your fs response latency is over a second, everything is going to suck till you deal with that.

RajT88 3y ago
I have had customers say this when not getting desired throughput cross-continent. "But the pipes! They are fat!" Yes but also your window needs to scale... 6 figure network engineers not knowing about window scaling, who do not know how to analyze a packet trace. I recently had someone suggest that they needed 1ms latency cross-continent. I explained patiently that the laws of physics have to change for them to hit that number. I am not even a network engineer!
- rescbr 3y ago
  Oh, this is too common! I can't even count how many times I had to blame Einstein for setting the speed of light too low in his law. (Nitpick: I know he's the wrong person to blame, but he's famous enough to make this joke understandable by everybody) Once I had to write a root cause analysis report and it basically explained how TCP works.
NovemberWhiskey 3y ago
Truth. I literally have a signal on one of my monitoring dashboards which indicates "database is currently undergoing online backup", because it is the single most important performance signal I have. This doesn't come to me as a signal from the database team; I have to poll the database for it myself. Noisy neighbor in inadequately-isolated, shared-tenancy models is just the worst.
xelxebar 3y ago
> your disk queue latency is pegged at your depth limit and your fs response latency is over a second You said words. And I totally, 100% understand them. But, like, for the plebs that are totally not me, could you elaborate on what you're talking about here? I^WThey would like to understand.
- stavros 3y ago
  Basically, "the disk is always terribly busy with tiny data operations".

broast 3y ago

I find it very hard to tolerate teams that say an issue isn't on them if they aren't pointing to evidence that indicates where they think the issue is. If you think it's not on you but it affects something within your responsibility, you're still on the hook until you prove it.

Scoundreller 3y ago
Nothing more fun than troubleshooting a fax machine incompatibility. “It’s you, we’re receiving faxes from everyone else” (how would they know?) “Nooo, it’s you, everyone else is receiving our faxes”