7 ms·

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).

colmvp 6y ago
Thinking you're fucked is unhelpful, but it's pretty normal to go full Alien Hudson in a dire situation and need someone else to assure you that things will be okay.
4e530344963049 6y ago
If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things. Also, make sure everyone knows you are going on vacation once you resolve things.
- paulz_ 6y ago
  These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
  - 4e530344963049 6y ago
    A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!
    - paulz_ 6y ago
      I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10" If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
      - munchbunny 6y ago
        Blue teaming in big company cyber security teams will get that for you. Not everything is a true positive but it’s always urgent. Pay is decent too.
        meowface 6y ago
        I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk). However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports. An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching. But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
        htrp 6y ago
        I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents
      - insomniacity 6y ago
        I know Mandiant do this (disaster response) for security incidents. Don't know of a generalised service, but I imagine the big consultancies offer it.
  - Aeolun 6y ago
    Come work for our company! We have an infinite amount of fires to put out. To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.
    - sanedigital 6y ago
      I believe that implies that the fires are indeed infinite, just countably so.
  - m463 6y ago
    It would be fun just to be able to talk to management with the same sort of candor the "cleaner" in pulp fiction did. (harvey keitel)
  - mattlondon 6y ago
    Constant "emergencies" soon wear you down. Sure it is fun, but the stakes can be high and the stress gets piled on. It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off! Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed. Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
    - ido 6y ago
      I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.
  - papreclip 6y ago
    Being an SRE is a little more of this and a little less scheduled work. You still have standup and plenty of other meetings, though
- blhack 6y ago
  This is basically the reason I got out of doing IT and into development. "If you do things well enough, nobody will think you did anything at all."
batch12 6y ago
One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..
- amachefe 6y ago
  The answer to the question is there >We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
  - kwhitefoot 6y ago
    > Management said we don't have enough money to fix any of this. It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost. I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner. An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years. I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
    - benlivengood 6y ago
      > I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups. If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
danjac 6y ago
Looking through the comments, it appears concerns were raised for years and management refused to allow the necessary changes to save money. While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
- pydry 6y ago
  >it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.
- jart 6y ago
  Reading comments https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlgdw/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... it sounds like he ran into a kernel panic, maybe because cosmic radiation messed with their full disk encryption because they went cheap on the ecc ram, and then he got discouraged, he probably asked management for the resources to rewrite the whole system from scratch, they said no, and then he sat on his hands collecting paychecks for two years until the thing he knew would happen finally happened. How else can you explain an MSP not having a copy of their zone file? That's their core business. It's also no secret in the MSP world how frequently these DNS servers get hacked. I read about it all the time. Especially on Reddit where you can just sense the pervasive fear. So you'd think that'd give a sysadmin a moment's pause to think, OK, have I scp'd my zone file in case that happens to me?
hinkley 6y ago
The first manager I ever respected was this goofball who worked at a fast food restaurant. I don't think the other managers took him seriously. He didn't really take himself seriously either (told the worst jokes), but when there was a rush he was the one you wanted running the kitchen. The other 2-3 people were kind of assholes under stress (they were assholes all the time if I'm honest). He was a machine, and it didn't hurt that he good at predicting orders before they were even made. We always made too much food in these situations but more of his would sell and the wait times were lower. In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
- danaliv 6y ago
  "Are you Mr. Disorganized who shines in an emergency?" would be a not-terrible screening question for ADHD: https://www.additudemag.com/benefits-of-adhd-crisis/ https://www.additudemag.com/benefits-of-adhd-crisis/
- oehpr 6y ago
  I've heard this is a kind of symptom of people who have ADHD, some of them have grace under pressure. I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD. But maybe check this out and ask if it rings any bells: https://www.nhs.uk/conditions/attention-deficit-hyperactivity-disorder-adhd/symptoms/ https://www.nhs.uk/conditions/attention-deficit-hyperactivit...
  - mijoharas 6y ago
    Wow... I really should have connected the dots. I have ADHD and recently started an SRE squad in our company since I tend to be the go-to person to fix any production issues. I had never really thought of a connection between those two things until now.

colmvp 6y ago

Thinking you're fucked is unhelpful, but it's pretty normal to go full Alien Hudson in a dire situation and need someone else to assure you that things will be okay.

4e530344963049 6y ago

If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things. Also, make sure everyone knows you are going on vacation once you resolve things.

paulz_ 6y ago
These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
- 4e530344963049 6y ago
  A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!
  - paulz_ 6y ago
    I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10" If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
    - munchbunny 6y ago
      Blue teaming in big company cyber security teams will get that for you. Not everything is a true positive but it’s always urgent. Pay is decent too.
      - meowface 6y ago
        I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk). However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports. An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching. But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
      - htrp 6y ago
        I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents
    - insomniacity 6y ago
      I know Mandiant do this (disaster response) for security incidents. Don't know of a generalised service, but I imagine the big consultancies offer it.
- Aeolun 6y ago
  Come work for our company! We have an infinite amount of fires to put out. To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.
  - sanedigital 6y ago
    I believe that implies that the fires are indeed infinite, just countably so.
- m463 6y ago
  It would be fun just to be able to talk to management with the same sort of candor the "cleaner" in pulp fiction did. (harvey keitel)
- mattlondon 6y ago
  Constant "emergencies" soon wear you down. Sure it is fun, but the stakes can be high and the stress gets piled on. It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off! Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed. Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
  - ido 6y ago
    I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.
- papreclip 6y ago
  Being an SRE is a little more of this and a little less scheduled work. You still have standup and plenty of other meetings, though
blhack 6y ago
This is basically the reason I got out of doing IT and into development. "If you do things well enough, nobody will think you did anything at all."

batch12 6y ago

One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..

amachefe 6y ago
The answer to the question is there >We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
- kwhitefoot 6y ago
  > Management said we don't have enough money to fix any of this. It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost. I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner. An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years. I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
  - benlivengood 6y ago
    > I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups. If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.

danjac 6y ago

Looking through the comments, it appears concerns were raised for years and management refused to allow the necessary changes to save money. While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.

pydry 6y ago
>it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.
jart 6y ago
Reading comments https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlgdw/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... it sounds like he ran into a kernel panic, maybe because cosmic radiation messed with their full disk encryption because they went cheap on the ecc ram, and then he got discouraged, he probably asked management for the resources to rewrite the whole system from scratch, they said no, and then he sat on his hands collecting paychecks for two years until the thing he knew would happen finally happened. How else can you explain an MSP not having a copy of their zone file? That's their core business. It's also no secret in the MSP world how frequently these DNS servers get hacked. I read about it all the time. Especially on Reddit where you can just sense the pervasive fear. So you'd think that'd give a sysadmin a moment's pause to think, OK, have I scp'd my zone file in case that happens to me?

hinkley 6y ago

The first manager I ever respected was this goofball who worked at a fast food restaurant. I don't think the other managers took him seriously. He didn't really take himself seriously either (told the worst jokes), but when there was a rush he was the one you wanted running the kitchen. The other 2-3 people were kind of assholes under stress (they were assholes all the time if I'm honest). He was a machine, and it didn't hurt that he good at predicting orders before they were even made. We always made too much food in these situations but more of his would sell and the wait times were lower. In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.

danaliv 6y ago
"Are you Mr. Disorganized who shines in an emergency?" would be a not-terrible screening question for ADHD: https://www.additudemag.com/benefits-of-adhd-crisis/ https://www.additudemag.com/benefits-of-adhd-crisis/
oehpr 6y ago
I've heard this is a kind of symptom of people who have ADHD, some of them have grace under pressure. I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD. But maybe check this out and ask if it rings any bells: https://www.nhs.uk/conditions/attention-deficit-hyperactivity-disorder-adhd/symptoms/ https://www.nhs.uk/conditions/attention-deficit-hyperactivit...
- mijoharas 6y ago
  Wow... I really should have connected the dots. I have ADHD and recently started an SRE squad in our company since I tend to be the go-to person to fix any production issues. I had never really thought of a connection between those two things until now.