6 ms·

> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction. I think part of t

by sbarre 5y ago

> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction. I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. A team is left behind to operate that infrastructure, but they may not necessarily understand it to the depth required to react properly when it fails or when it needs updating, etc.. There's a 20+ year old book called The Ingenuity Gap that addresses this exact challenge with complex systems that we aren't equipped to do much more than "keep the lights on".. It definitely feels like we are running up against similar situations in the software world, particularly with the huge amount of job churn (and corporate brain drain) that the pandemic produced.

ghaff 5y ago
You see this at all different levels of complexity. I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.
- syntheticnature 5y ago
  As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.
notyourday 5y ago
> I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )
hinkley 5y ago
> and then leave once the interesting/challenging work is done. Don't underestimate constructive dismissal. The courts have begun to take a somewhat dim view of executives being completely ignorant of Too Good to be True scenarios, which has been a viable form of plausible deniability forever and a day. I'm not treating the workers badly, George is treating the workers badly. George, who I brag about all the time and have promoted three times in five years. There's this sort of notion of 'what have you done for me lately?' that pushes us to underappreciate people who set up systems that usually run on autopilot. Bosses can't absorb the idea that just because nothing has broken doesn't mean it won't break, and these may be the easiest plates in the world to spin but they are in fact spinning, and so this person who "doesn't look busy" is doing so on purpose. It's what some clever person called "on-site on-call" to differentiate doing nothing from actively doing nothing. It's very reminiscent of the experience some people have buying someone a gift (a subscription, a car, etc) that has an ongoing cost and then they hear from the person how disappointed they are in the new/other gifts they got. "Last year they bought me a car and this year all I got was a new iPad." No, this year you got an iPad and your rich uncle is still paying for your car, you little shit. Steve may be maintaining systems that save us 4 headcount, but he looks like a cost center to the business. He's lucky we don't lay him off, so don't talk to me about giving him a 10% raise for boosting that to 4.5 headcount.

by sbarre 5y ago

ghaff 5y ago

You see this at all different levels of complexity. I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.

syntheticnature 5y ago
As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.

notyourday 5y ago

> I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done. Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )

hinkley 5y ago

> and then leave once the interesting/challenging work is done. Don't underestimate constructive dismissal. The courts have begun to take a somewhat dim view of executives being completely ignorant of Too Good to be True scenarios, which has been a viable form of plausible deniability forever and a day. I'm not treating the workers badly, George is treating the workers badly. George, who I brag about all the time and have promoted three times in five years. There's this sort of notion of 'what have you done for me lately?' that pushes us to underappreciate people who set up systems that usually run on autopilot. Bosses can't absorb the idea that just because nothing has broken doesn't mean it won't break, and these may be the easiest plates in the world to spin but they are in fact spinning, and so this person who "doesn't look busy" is doing so on purpose. It's what some clever person called "on-site on-call" to differentiate doing nothing from actively doing nothing. It's very reminiscent of the experience some people have buying someone a gift (a subscription, a car, etc) that has an ongoing cost and then they hear from the person how disappointed they are in the new/other gifts they got. "Last year they bought me a car and this year all I got was a new iPad." No, this year you got an iPad and your rich uncle is still paying for your car, you little shit. Steve may be maintaining systems that save us 4 headcount, but he looks like a cost center to the business. He's lucky we don't lay him off, so don't talk to me about giving him a 10% raise for boosting that to 4.5 headcount.