3 ms·
I work with banks on, um issues, without disclosing too much. Occasionally we find issues with their software workflows, and this isn't related directly to mone
by pixl97 3y ago
I work with banks on, um issues, without disclosing too much. Occasionally we find issues with their software workflows, and this isn't related directly to monetary stuff. We're not allowed to have the end result change in any way most of the time. "But we're getting a wrong answer", or "This is costing XX person hours per day" isn't up for consideration. Nothing must change.
Ugh, and on the subject of banks computer operations. Every single department is in deep blame avoidance mode. It's not "find and identify problems" mode, it's "It wasn't me" mode. We had our application performance drop to almost zero (like we dropped to disk operations per minute IOPM) I spent hours telling the customer, this is your infrastructure. So we got infrastructure teams on the call trying to figure out where it was. Not a single one of them were helpful "Everything fine, it's not us" was the first thing out of their mouths and the second was "We didn't change anything".
It took 10 hours of sitting on a call over 2 days to get the NAS team to admit they turned on anti-virus on the NAS side and that the machines were in meltdown mode because the CPU was off the charts. The preceding people didn't even look at the metrics before coming back with a "it's not my problem, everything is fine" response.
- rkagerer 3y agoEncountered a challenge like this once. Infrastructure team kept telling us it wasn't anything on their end. I coordinated with the business unit VP to serve their entire QA environment from four VM's on my laptop for a day, and performance went from slow as molasses, to purring like a kitten. A couple days later their infrastructure team finally identified storage latency issues on their multi-million dollar cluster and let me help them fix it.
- pixl97 3y agoI swear IOPS/request latency is one of the least understood things in these huge companies. "But we have 40bazillion GB and 100GB LAN", cool story bro, your disk queue latency is pegged at your depth limit and your fs response latency is over a second, everything is going to suck till you deal with that.
- RajT88 3y agoI have had customers say this when not getting desired throughput cross-continent. "But the pipes! They are fat!" Yes but also your window needs to scale... 6 figure network engineers not knowing about window scaling, who do not know how to analyze a packet trace. I recently had someone suggest that they needed 1ms latency cross-continent. I explained patiently that the laws of physics have to change for them to hit that number. I am not even a network engineer!
- rescbr 3y agoOh, this is too common! I can't even count how many times I had to blame Einstein for setting the speed of light too low in his law. (Nitpick: I know he's the wrong person to blame, but he's famous enough to make this joke understandable by everybody) Once I had to write a root cause analysis report and it basically explained how TCP works.
- NovemberWhiskey 3y agoTruth. I literally have a signal on one of my monitoring dashboards which indicates "database is currently undergoing online backup", because it is the single most important performance signal I have. This doesn't come to me as a signal from the database team; I have to poll the database for it myself. Noisy neighbor in inadequately-isolated, shared-tenancy models is just the worst.
- xelxebar 3y ago> your disk queue latency is pegged at your depth limit and your fs response latency is over a second You said words. And I totally, 100% understand them. But, like, for the plebs that are totally not me, could you elaborate on what you're talking about here? I^WThey would like to understand.
- stavros 3y agoBasically, "the disk is always terribly busy with tiny data operations".
- broast 3y agoI find it very hard to tolerate teams that say an issue isn't on them if they aren't pointing to evidence that indicates where they think the issue is. If you think it's not on you but it affects something within your responsibility, you're still on the hook until you prove it.
- Scoundreller 3y agoNothing more fun than troubleshooting a fax machine incompatibility. “It’s you, we’re receiving faxes from everyone else” (how would they know?) “Nooo, it’s you, everyone else is receiving our faxes”
- aidenn0 3y ago> I work with banks on, um issues, without disclosing too much. Occasionally we find issues with their software workflows, and this isn't related directly to monetary stuff. We're not allowed to have the end result change in any way most of the time. I think this is a good constraint in general for banks to have, and its definitely harder to change a workflow under these constraints. > Every single department is in deep blame avoidance mode. It's not "find and identify problems" mode, it's "It wasn't me" mode. This tends to happen in large organizations, and is incredibly toxic to productivity. The most extreme form is when you get fired (or otherwise censured) for fixing something because "You were in charge of the thing that was causing all this trouble?!"
- WhiteOwlLion 3y agoYou sometimes have to treat adults like little children. The first thing I say is, "you're not in trouble". Kids assume if an adult talks to them, they are automatically in trouble. I've gotten a lot of helpful results by telling people no one is in trouble when you make your inquiry. Amazing how grown adults regress to children when a different department calls them.