3 ms·
> Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lo
by hintymad 4mo ago
> Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts.
Not to be sarcastic but just to offer an observation: in a sufficiently large or bureaucratic organization, preventing an incident from happening can rarely get you any credit or visibility. Such achievement falls into the bucket of "what you're supposed to do". So, those who navigate company dynamics well would rather let the incident happen and then be loud on the follow-up action items. The trick is not to turn an incident into a diaster, so it's a dedicate act.
- tormeh 4mo agoI learned this early in a conservative org. Preventing things is risky. Just keep the solution ready for when things go wrong because then you'll get approval.
- cloche 4mo agoNever waste the opportunity a good crisis presents
- sergeym 4mo agoI've had this in my career also, I've had a solution that was deemed too risky to release by management but would have prevented an outage, but when an outage actually happened that was the first thing they wanted to try and it worked gloriously. I'm thinking that if it was released prior to the incident it would have not have had the same impact on my career.
- justonceokay 4mo agoAnecdotally I know of an engineer in the Excel team. They would keep around a list of low priority security bugs. When they wanted to do improvements on the system they would attach it to one of the security bugs “nearby“ because the change would become approved much more easily than just fixing the problem itself.
- nitwit005 4mo agoThe examples of high impact all seem like things unlikely to receive recognition. If you save a sales deal, they'll cheer the sales staff. And pay them a commission, which you will receive no part of.
- skmurphy 4mo agoAnd start to build a relationship with sales that, at least in a B2B firm, can be of significant benefit.
- CobrastanJorji 4mo agoAlso, disaster is a good signal to the higher ups that there are problems in your org. If you keep putting out every fire with heroics, your boss will know (maybe), but his boss's boss's boss will see that your org is doing great and everything is all code green. If you let a few things burn down, your boss's boss's boss will notice the fire, and things may improve. It's perhaps the easiest way you have of communicating with them.
- themaninthedark 4mo agoWhat if my boss's boss is the root cause of the fires and he just blames the "team" if his boss ever sees a hint the of issues?
- CobrastanJorji 4mo agoYour great grandboss will not look favorably upon your grandboss passing blame downhill. Disasters are (hopefully) always the fault of the person in charge, which is why you'll notice that your bosses are unusually involved with meetings about how to avoid more disasters.
- hintymad 4mo agoYeah, communicating upwards about potential issues and what you're doing about them is essential. When a disaster strikes, you'll look like a sage in the eyes of the higher ups.
- bauldursdev 4mo agoCan't solve a problem that doesn't exist yet
- mjklin 4mo agoIt’s about what people notice. In town governments I’ve heard of cutting popular programs that will provoke an outcry only to get credit for reinstating them, while possibly smuggling through other actions that are necessary but unpopular.
- johnnycool 4mo agoMany carears are made and bonuses paidout with heroesim trick.
- johnbcoughlin 4mo agoI read the section you quoted as being about saving the day in an incident that's already in progress.