8 ms·

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a sweat, that still happens to me, just happened yesterday) is to take 5-10 minutes before doing any action to try to fix it and sketch it out, think about it as clearly as you can. Fear interferes with your ability to reason rationally. Mashing buttons in a panic can make your problems spiral even worse (seen that happen). Disrupting that fear circuit any way you can is important. Splashing my face and hands with extremely cold water is my trick. Then after you go through a few of these, you'll realize it really isn't too bad and you've dealt with bad situations before and you'll gain the confidence to know you can deal with it, even when there's no one you can reach out to for help.

H8crilA 2y ago
And ultimately you carry none of the risk. It's not your company, and the company can (and will) cut you off at a random time. Unless it is your company :)
- minkles 2y ago
  This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.
  - toomuchtodo 2y ago
    Was fired, worked out but only through luck (the firing was humane, so credit where credit due). Your advice is spot on: derisk financially, do your best work, but don't care when you get walked out. It's just a job, it doesn't matter. If it matters because you need the job, treat your financial situation like an emergency until you don't need the job. It will be a cold day in hell before I am ever on call or in a pager rotation again. "We're not saving lives, we're just building websites." as a wise old man once told me early in my career.
    - antisthenes 2y ago
      My manager told me this during the first "emergency" we had where the client was wildly over-reacting. We're not doing open-heart surgery or launching people into space. They can wait 1-2 hours extra. Earned a lot of my respect on that day :)
      - toomuchtodo 2y ago
        Good manager, rare manager. Worth their weight in gold.
  - mtnviewer 2y ago
    How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.
    - recursive 2y ago
      It's not net worth. It's how long you can abide. If you have $2M net worth and $1.99 is tied up in a way you can't spend on bills, that don't mean nothin'. If things got bad, I guess you could move. You could probably buy a house now in fly-over country and just retire, living on interest from conservative investments.
    - deleted 2y ago
      [deleted]
    - nine_zeros 2y ago
      At 2M net worth, the only thing that is making you fearful is your own poor assessment of risk/regret in life. You are more likely to regret taking large debts or missing out on life milestones than on losing the job or going hungry.
    - Muromec 2y ago
      Why do you even bother to work if you have 2M ?
      - supportengineer 2y ago
        Doesn't help you much if you are a family in Bay Area.
        deleted 2y ago
        [deleted]
        Muromec 2y ago
        Of course I'm a random person on the internet and you haven't asked for my advice. Just saying -- kids pick up languages real fast, friendship treaties and tax benefits are abundant and with 2M you can shop around for a better lifestyle if you really want to not have that stress and fear, unless you are determined to become filthy rich, I ofc don't know you.
sdenton4 2y ago
OTOH, a lot of people who really needed to be able to change their event pricing at 2am (+/- two timezones) on jumpcomedy.com were really let down. Probably some of them died. Imagine how much more the damage would have been if someone was telling this solo developer to stop trying to make 'fetch' happen.
agumonkey 2y ago
After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.
- caseyy 2y ago
  What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better. It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.
  - silverquiet 2y ago
    Action bias and it seems rather fundamental to human behavior. You don't want to be seen as doing nothing when there are problems, but of course with a complex system, it can be very difficult to diagnose an issue, so you tend to screw things up if you just rush in. I think my favorite saying in that instance is, "don't just do something, stand there".
    - caseyy 2y ago
      That’s a good one. I think my driving instructor told me something similar on my first lesson. I’m talking about the brain fog more than the action itself, though. I no longer have it since I’ve been trained in emergency field medicine. That training course gave me enough exposure to stress to no longer get into this state of confusion. But I want to help others deal with it and it’s hard to find info online.
  - fragmede 2y ago
    I've heard "catch a falling knife" for this.
    - samus 2y ago
      That's a very common term in stock trading as well! Prices are crashing, and you try to make use of that opportunity to buy stock and make a profit when the stock rebounds. Then the price crashes further, the rebound is not gonna happen in the near future, and your fingers are a bloody mess on the ground... Another panic reaction is selling off everything at the first sign of trouble. And some people hold on to doomed stocks for way longer than what makes sense (look up "Intel guy" on /r/wallstreetbets) Day trading requires due diligence and nerves like steel, else you are just making somebody else a little bit richer.
nine_zeros 2y ago
The easiest way to not panic is to always remind yourself, "You don't need to be a hero. It is not your company". If the company wants to derisk their business, they can hire more people and create systematic safeguards in their services over a long period of time. The return on this investment is theirs. If they don't make this investment, they don't get the returns - simple as that. If you are afraid of "losing your job" - remember that you might lose your job even if you were the hero because of reasons not really in your control. The only time to really panic is when your family is under crisis. To prevent your family from going in crisis, you should build safeguards in terms of health, income etc. Business owners can do the same if they want safeguards. If not, fine, things will just break. And it is NOT an individual engineers problem to fix all of their poor decisions.
bityard 2y ago
For me, in situations like this, the fear and anxiety comes from the part of my brain screaming, "but what if I can NEVER fix this?" Which is of course completely nonsense. There is ALWAYS a solution somewhere, even if it's VERY hard to find, or even if it's not the one you want. My first serious job as an adult was avionics repair. I knew how to do SOME component-level troubleshooting but I wasn't very good at it and always got lost in schematics pretty quickly. When I ran across a broken device that had a problem I couldn't solve within 30 minutes of investigation, my first instinct was to say, "welp, this is beyond my abilities" and ship the card or box off as permanently broken, or try to pawn it off to a more senior tech in the shop. Fortunately, I had a mentor who disabused me of that tendency. He taught me that EVERY fault has a cause. Every time you think you've exhausted all options, what it really means is that you've only exhausted all CONVENIENT options and you just have to suck it up and get cracking on the harder paths.
everdrive 2y ago
Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares enough to notice. - Useless meetings which waste collective employee hours daily. - Capability serves no purpose, but just exists to meet audit requirements. - Executives enrich themselves and waste company money constantly. - Etc. Downtime doesn't seem any worse than these problems, but it gets far more attention and panic. It feels a bit like terrorism vs. heart disease. It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. I'm not calling companies malevolent, but they're a bit like a bully in that regard: they push you as much as you yield.
- ekanes 2y ago
  > It feels a bit like terrorism vs. heart disease. Fantastic analogy! One is scary but practically non-existent, and the other will bring early death to many people you know.
- JohnMakin 2y ago
  One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time.
  - chasd00 2y ago
    > It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time. if no one noticed then can you really say it was broken?
lukan 2y ago
"Splashing my face and hands with extremely cold water is my trick." Yes, literally cooling down. Combined with conscious breathing. And then go on doing, what you can do.
- cma 2y ago
  It's the mammalian diving reflex: https://en.wikipedia.org/wiki/Diving_reflex https://en.wikipedia.org/wiki/Diving_reflex
  - lukan 2y ago
    [dead]
sameoldtune 2y ago
I was once the primary party responsible for an accounting bug that would have cost Amazon around 100 million dollars in unselable inventory. I stayed up for two nights writing the most heinous regexes to scrape logs(1) and fix the months-long discrepancies. Ended up fixing the problem by writing a script that re-submitted an event for every inbounded shipment to every Amazon warehouse for the last few months. I took the next week off to recover physically and mentally. Afterward I made a promise to myself to never get into such a panic again. But funny enough, when I feel like panic is immanent I just remind myself that whatever problem I have is absolute peanuts compared to that event. So for me at least having one truly bad event helps me put things into context. (1) the logs had JSON-escaped JSON and I am slightly proud of those regexes because it is the only time I hopefully ever have to use 8 backslashes in a row https://xkcd.com/1638/ https://xkcd.com/1638/
sb8244 2y ago
Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on something, but as I think back I'm not sure that was ever the case in the end. It was wild to see the top brass—normally very cool and composed—start suggesting arbitrary potential fixes during an incident.
- Vegenoid 2y ago
  It's a good reminder that if things get bad, people will just start burning things to try and appease the gods.
- JohnMakin 2y ago
  I have a similar mantra - "if you don't know why a fix worked, you may not have fixed it." I'm willing to throw shit at the wall early in the triaging process, but only when they are low-impact and "simple" things. stuff like - have we tried clearing cache? have we checked DNS resolver for errors? have we restarted the server? etc. I try to find the "dumb" problems before jumping to some wild fix. In one of the worst outages of my career, a team I was working for tried to do a full database restore, which had never been done in production, based on a guess. At 3am on a saturday. I push back really hard at stuff like that.
  - a_e_k 2y ago
    That mantra reminds me of "Any problem that goes away by itself can just as easily come back by itself."
- DylanSp 2y ago
  To quote Gene Kranz during the start of problems on Apollo 13: "Let's not make things worse by guessing."
- aPoCoMiLogin 2y ago
hibikir 2y ago
My proudest SRE moment was when, in a team that dealt with a key system where seconds meant millions, I was able to find a precursor to the problem, so instead of having to intervene on a page in 3 minutes before disaster, the alarm now had 20 minutes of headroom. We all slept so much better after that
pknomad 2y ago
One of my VP's (one of the coolest dude I know) would often say "slow is smooth and smooth is fast". You're absolutely right that fear interferes with one's ability to reason rationally but I'd also add that fear is highly infectious. If ICs sees their leaders/managers/colleagues panicking, they will often panic too. Thankfully my VP was always level-headed and prioritized clarity over action.
- DowagerDave 2y ago
  >> One of my VP's (one of the coolest dude I know) would often say "slow is smooth and smooth is fast". Apparently you worked for a Navy Seal, so yeah, likely a very cool dude :)
  - pknomad 2y ago
    How did you know?
    - c0balt 2y ago
      Well known quote, that is often promoted as being used by the Navy Seals. Enter it in your favorite search machine to learn more about the background of the phrase.
dmvdoug 2y ago
I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.
- deleted 2y ago
  [deleted]
iwontberude 2y ago
Same experience working for a major streaming provider as SRE. Even when it's a million dollars per minute, the stress never will help you to get the bleeding to stop. It's very helpful everywhere else in life. I can handle social, physical, and other stressors better after this exposure. Nothing in business is as bad as your biology can fool you to believe.

H8crilA 2y ago

And ultimately you carry none of the risk. It's not your company, and the company can (and will) cut you off at a random time. Unless it is your company :)

minkles 2y ago
This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.
- toomuchtodo 2y ago
  Was fired, worked out but only through luck (the firing was humane, so credit where credit due). Your advice is spot on: derisk financially, do your best work, but don't care when you get walked out. It's just a job, it doesn't matter. If it matters because you need the job, treat your financial situation like an emergency until you don't need the job. It will be a cold day in hell before I am ever on call or in a pager rotation again. "We're not saving lives, we're just building websites." as a wise old man once told me early in my career.
  - antisthenes 2y ago
    My manager told me this during the first "emergency" we had where the client was wildly over-reacting. We're not doing open-heart surgery or launching people into space. They can wait 1-2 hours extra. Earned a lot of my respect on that day :)
    - toomuchtodo 2y ago
      Good manager, rare manager. Worth their weight in gold.
- mtnviewer 2y ago
  How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.
  - recursive 2y ago
    It's not net worth. It's how long you can abide. If you have $2M net worth and $1.99 is tied up in a way you can't spend on bills, that don't mean nothin'. If things got bad, I guess you could move. You could probably buy a house now in fly-over country and just retire, living on interest from conservative investments.
  - deleted 2y ago
    [deleted]
  - nine_zeros 2y ago
    At 2M net worth, the only thing that is making you fearful is your own poor assessment of risk/regret in life. You are more likely to regret taking large debts or missing out on life milestones than on losing the job or going hungry.
  - Muromec 2y ago
    Why do you even bother to work if you have 2M ?
    - supportengineer 2y ago
      Doesn't help you much if you are a family in Bay Area.
      - deleted 2y ago
        [deleted]
      - Muromec 2y ago
        Of course I'm a random person on the internet and you haven't asked for my advice. Just saying -- kids pick up languages real fast, friendship treaties and tax benefits are abundant and with 2M you can shop around for a better lifestyle if you really want to not have that stress and fear, unless you are determined to become filthy rich, I ofc don't know you.

sdenton4 2y ago

OTOH, a lot of people who really needed to be able to change their event pricing at 2am (+/- two timezones) on jumpcomedy.com were really let down. Probably some of them died. Imagine how much more the damage would have been if someone was telling this solo developer to stop trying to make 'fetch' happen.

agumonkey 2y ago

After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.

caseyy 2y ago
What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better. It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.
- silverquiet 2y ago
  Action bias and it seems rather fundamental to human behavior. You don't want to be seen as doing nothing when there are problems, but of course with a complex system, it can be very difficult to diagnose an issue, so you tend to screw things up if you just rush in. I think my favorite saying in that instance is, "don't just do something, stand there".
  - caseyy 2y ago
    That’s a good one. I think my driving instructor told me something similar on my first lesson. I’m talking about the brain fog more than the action itself, though. I no longer have it since I’ve been trained in emergency field medicine. That training course gave me enough exposure to stress to no longer get into this state of confusion. But I want to help others deal with it and it’s hard to find info online.
- fragmede 2y ago
  I've heard "catch a falling knife" for this.
  - samus 2y ago
    That's a very common term in stock trading as well! Prices are crashing, and you try to make use of that opportunity to buy stock and make a profit when the stock rebounds. Then the price crashes further, the rebound is not gonna happen in the near future, and your fingers are a bloody mess on the ground... Another panic reaction is selling off everything at the first sign of trouble. And some people hold on to doomed stocks for way longer than what makes sense (look up "Intel guy" on /r/wallstreetbets) Day trading requires due diligence and nerves like steel, else you are just making somebody else a little bit richer.

nine_zeros 2y ago

The easiest way to not panic is to always remind yourself, "You don't need to be a hero. It is not your company". If the company wants to derisk their business, they can hire more people and create systematic safeguards in their services over a long period of time. The return on this investment is theirs. If they don't make this investment, they don't get the returns - simple as that. If you are afraid of "losing your job" - remember that you might lose your job even if you were the hero because of reasons not really in your control. The only time to really panic is when your family is under crisis. To prevent your family from going in crisis, you should build safeguards in terms of health, income etc. Business owners can do the same if they want safeguards. If not, fine, things will just break. And it is NOT an individual engineers problem to fix all of their poor decisions.

bityard 2y ago

For me, in situations like this, the fear and anxiety comes from the part of my brain screaming, "but what if I can NEVER fix this?" Which is of course completely nonsense. There is ALWAYS a solution somewhere, even if it's VERY hard to find, or even if it's not the one you want. My first serious job as an adult was avionics repair. I knew how to do SOME component-level troubleshooting but I wasn't very good at it and always got lost in schematics pretty quickly. When I ran across a broken device that had a problem I couldn't solve within 30 minutes of investigation, my first instinct was to say, "welp, this is beyond my abilities" and ship the card or box off as permanently broken, or try to pawn it off to a more senior tech in the shop. Fortunately, I had a mentor who disabused me of that tendency. He taught me that EVERY fault has a cause. Every time you think you've exhausted all options, what it really means is that you've only exhausted all CONVENIENT options and you just have to suck it up and get cracking on the harder paths.

everdrive 2y ago

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares enough to notice. - Useless meetings which waste collective employee hours daily. - Capability serves no purpose, but just exists to meet audit requirements. - Executives enrich themselves and waste company money constantly. - Etc. Downtime doesn't seem any worse than these problems, but it gets far more attention and panic. It feels a bit like terrorism vs. heart disease. It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. I'm not calling companies malevolent, but they're a bit like a bully in that regard: they push you as much as you yield.

ekanes 2y ago
> It feels a bit like terrorism vs. heart disease. Fantastic analogy! One is scary but practically non-existent, and the other will bring early death to many people you know.
JohnMakin 2y ago
One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time.
- chasd00 2y ago
  > It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time. if no one noticed then can you really say it was broken?

lukan 2y ago

"Splashing my face and hands with extremely cold water is my trick." Yes, literally cooling down. Combined with conscious breathing. And then go on doing, what you can do.

cma 2y ago
It's the mammalian diving reflex: https://en.wikipedia.org/wiki/Diving_reflex https://en.wikipedia.org/wiki/Diving_reflex
- lukan 2y ago
  [dead]

sameoldtune 2y ago

I was once the primary party responsible for an accounting bug that would have cost Amazon around 100 million dollars in unselable inventory. I stayed up for two nights writing the most heinous regexes to scrape logs(1) and fix the months-long discrepancies. Ended up fixing the problem by writing a script that re-submitted an event for every inbounded shipment to every Amazon warehouse for the last few months. I took the next week off to recover physically and mentally. Afterward I made a promise to myself to never get into such a panic again. But funny enough, when I feel like panic is immanent I just remind myself that whatever problem I have is absolute peanuts compared to that event. So for me at least having one truly bad event helps me put things into context. (1) the logs had JSON-escaped JSON and I am slightly proud of those regexes because it is the only time I hopefully ever have to use 8 backslashes in a row https://xkcd.com/1638/ https://xkcd.com/1638/

sb8244 2y ago

Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on something, but as I think back I'm not sure that was ever the case in the end. It was wild to see the top brass—normally very cool and composed—start suggesting arbitrary potential fixes during an incident.

Vegenoid 2y ago
It's a good reminder that if things get bad, people will just start burning things to try and appease the gods.
JohnMakin 2y ago
I have a similar mantra - "if you don't know why a fix worked, you may not have fixed it." I'm willing to throw shit at the wall early in the triaging process, but only when they are low-impact and "simple" things. stuff like - have we tried clearing cache? have we checked DNS resolver for errors? have we restarted the server? etc. I try to find the "dumb" problems before jumping to some wild fix. In one of the worst outages of my career, a team I was working for tried to do a full database restore, which had never been done in production, based on a guess. At 3am on a saturday. I push back really hard at stuff like that.
- a_e_k 2y ago
  That mantra reminds me of "Any problem that goes away by itself can just as easily come back by itself."
DylanSp 2y ago
To quote Gene Kranz during the start of problems on Apollo 13: "Let's not make things worse by guessing."
aPoCoMiLogin 2y ago

hibikir 2y ago

My proudest SRE moment was when, in a team that dealt with a key system where seconds meant millions, I was able to find a precursor to the problem, so instead of having to intervene on a page in 3 minutes before disaster, the alarm now had 20 minutes of headroom. We all slept so much better after that

pknomad 2y ago

One of my VP's (one of the coolest dude I know) would often say "slow is smooth and smooth is fast". You're absolutely right that fear interferes with one's ability to reason rationally but I'd also add that fear is highly infectious. If ICs sees their leaders/managers/colleagues panicking, they will often panic too. Thankfully my VP was always level-headed and prioritized clarity over action.

DowagerDave 2y ago
>> One of my VP's (one of the coolest dude I know) would often say "slow is smooth and smooth is fast". Apparently you worked for a Navy Seal, so yeah, likely a very cool dude :)
- pknomad 2y ago
  How did you know?
  - c0balt 2y ago
    Well known quote, that is often promoted as being used by the Navy Seals. Enter it in your favorite search machine to learn more about the background of the phrase.

dmvdoug 2y ago

I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.

deleted 2y ago
[deleted]

iwontberude 2y ago

Same experience working for a major streaming provider as SRE. Even when it's a million dollars per minute, the stress never will help you to get the bleeding to stop. It's very helpful everywhere else in life. I can handle social, physical, and other stressors better after this exposure. Nothing in business is as bad as your biology can fool you to believe.