4 ms·

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The so

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares enough to notice. - Useless meetings which waste collective employee hours daily. - Capability serves no purpose, but just exists to meet audit requirements. - Executives enrich themselves and waste company money constantly. - Etc. Downtime doesn't seem any worse than these problems, but it gets far more attention and panic. It feels a bit like terrorism vs. heart disease. It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. I'm not calling companies malevolent, but they're a bit like a bully in that regard: they push you as much as you yield.

ekanes 2y ago
> It feels a bit like terrorism vs. heart disease. Fantastic analogy! One is scary but practically non-existent, and the other will bring early death to many people you know.
JohnMakin 2y ago
One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time.
- chasd00 2y ago
  > It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time. if no one noticed then can you really say it was broken?
  - JohnMakin 2y ago
    Depends - IME yes, lol. Stuff like email campaigns being broken is super hard for the business to detect, as is a lot of random customer facing stuff that doesn't directly drive revenue. Stuff that isn't a total outage and a degradation will often be tolerated or unnoticed for quite a long time til it actually blows up. My favorite anti-pattern ever that happens here is when you notice a bug that's existed for a long time, you fix it, then some other part of the system that had adapted and expected the behavior over the years breaks because of how they were handling the bug no longer works. Then of course the business comes to you like "why did you break this?"
  - lolc 2y ago
    Errors can go unnoticed or ignored until they cause a fault. For example you may not notice missing bolts on the underside of the bridge.
lifeisstillgood 2y ago
That is a great perspective - thank you
caseyy 2y ago
> It feels a bit like terrorism [..] It’s important not to sweat it Hmm… interesting thought. It’s interesting how labeling something a terror attack makes the public lose critical thinking and act incoherently for a bit. It’s like a stunning shock response. The very same this author describes. Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode? People in customer service know clients definitely do. It is interesting how these concepts connect.
- everdrive 2y ago
  >Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode? I'm sure some do. Much more often I observe that the managers themselves are just reacting to the incentive system in their environment -- and that those incentive systems are complex, not-widely-understood, and often built "accidentally." ie, companies are often accidentally incentivizing the wrong behaviors and are often powerless to prevent this.
  - prerok 2y ago
    I am almost convinced most middle management are like small babies. That's not meant derogatively: there's a problem, they have been notified, yet they cannot do anything about it. They need someone else to fix it. So, they cry loud and clear that there is a need to fix this. It's difficult to understand sometimes, but they are pressured from above, and they don't have the means to fix it by themselves.
    - caseyy 2y ago
      Good management is a tool they have to fix the problem. But yes, sometimes they have mismanaged things so much that they can only kick up a fuss or pester people. It causes a lot of stress and is not very effective from my exp. On a more positive note, other managers will have a rapport with their teams where if the team only learns of a problem, it will be addressed. They only need to act as liaisons — stick something on a kanban board somewhere, and it will get resolved. No loud crying necessary, just business as usual.
vlunkr 2y ago
The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.
- blastro 2y ago
  Every time the stock market shits itself, the large brokerages seem to "go down"
- Alupis 2y ago
  The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer forever. Customer loyalty is often fickle - why would they wait around for you to fix your stuff when they can simply place their order on Amazon or another retail website?
  - vasco 2y ago
    You are of course right, but in practice I've seen many more incidents created by doing changes to make things more robust than from simpler incremental product changes that usually are feature flagged and so on. At deeper levels usually there's less ability to do such containment (or is too expensive or takes too long or people are lazy) and so many times I wonder if it's better to do the trade-off or just keep things simple and eat only the "simple" sources of downtime to fix. For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.
fragmede 2y ago
There are even times to prefer to site being down to being up. If you've been hacked and the site is serving CSAM/malware/leaking PII maliciously, as the SRE on deck, your job is to keep it down!
Keyframe 2y ago
not saying you're wrong, because you absolutely aren't, but there's one key difference - usually freakouts are when it's client-facing related in combination with no control and/or information on what's going on and 'they' have to answer customer calls. That's when panic mode is activated.
cgh 2y ago
> It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. One time a few years ago, a coworker called me to tell me the stress we were under was causing her hair to fall out. I mentioned I'd lost significant weight and wasn't sleeping well, especially since another coworker had quit and now I was shouldering his responsibilities (programming and managing a junior). She said she was going to quit. I wished her the best. Around six months later, I had to quit too. It took around a year for me to get back to normal sleep patterns and for my heart to stop racing in the night. Oh yeah, another guy from that team had a "cardiac event" due to stress and had to take a month off.
- ryandrake 2y ago
  This is such an insane mentality for a rank-and-file employee who does not have significant upside tied to the business. If you're the sole shareholder of the company and there's some technical problem causing it to stop making money, then I can see it having a stressful effect on you and your health. If you're a worker bee who got 0.00000001% of the company's equity (if you're lucky), why oh why would you let the company's problems cause you to miss sleep and have a literal heart attack?
burningChrome 2y ago
>> Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems THIS. Worked for a huge corporation when I first started contracting as a developer. This was a daily occurrence. Someone would push some stuff to prod and it would break their very large e-commerce site. Sometimes it was minor stuff, other times it would cripple the entire site. Most of the time, nobody GAF at all. I remember sitting at my desk one day and my senior dev walking by my desk like Lumbergh in Office Space. Coffee in hand and just casually said to me, "Looks like someone broke the build, most of the video game pages are throwing 404's right now, going to lunch, see you soon." and then just walked away. I also found out one day one of the CDN folks that was running the Akamai account quit and then a month later, we found out one of our secondary content servers had been down since the day he left. When dude left, he never transferred any of his knowledge to the other team members so once they found out the server was down, it took another two days to get in contact with someone at Akamai who was handling our account. At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""?
- burntwater 2y ago
  >> At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""? That's funny, because reading your first sentence I was thinking that was the dysfunctional place. I had a boss that would take problems from 0 to 10 in a flash, when the problem was really a 4 or 5, and it really was not a great environment.
  - ryandrake 2y ago
    Yea, nobody seems to be able to handle problems that require solid, but moderate, non-urgent effort. The problem is either a 0, where nobody even knows, let alone cares that it's happening or a 10, where it's treated as though everyone's chair is on fire and the company is losing $10M per millisecond.

ekanes 2y ago

> It feels a bit like terrorism vs. heart disease. Fantastic analogy! One is scary but practically non-existent, and the other will bring early death to many people you know.

JohnMakin 2y ago

One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time.

chasd00 2y ago
> It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time. if no one noticed then can you really say it was broken?
- JohnMakin 2y ago
  Depends - IME yes, lol. Stuff like email campaigns being broken is super hard for the business to detect, as is a lot of random customer facing stuff that doesn't directly drive revenue. Stuff that isn't a total outage and a degradation will often be tolerated or unnoticed for quite a long time til it actually blows up. My favorite anti-pattern ever that happens here is when you notice a bug that's existed for a long time, you fix it, then some other part of the system that had adapted and expected the behavior over the years breaks because of how they were handling the bug no longer works. Then of course the business comes to you like "why did you break this?"
- lolc 2y ago
  Errors can go unnoticed or ignored until they cause a fault. For example you may not notice missing bolts on the underside of the bridge.

lifeisstillgood 2y ago

That is a great perspective - thank you

caseyy 2y ago

> It feels a bit like terrorism [..] It’s important not to sweat it Hmm… interesting thought. It’s interesting how labeling something a terror attack makes the public lose critical thinking and act incoherently for a bit. It’s like a stunning shock response. The very same this author describes. Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode? People in customer service know clients definitely do. It is interesting how these concepts connect.

everdrive 2y ago
>Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode? I'm sure some do. Much more often I observe that the managers themselves are just reacting to the incentive system in their environment -- and that those incentive systems are complex, not-widely-understood, and often built "accidentally." ie, companies are often accidentally incentivizing the wrong behaviors and are often powerless to prevent this.
- prerok 2y ago
  I am almost convinced most middle management are like small babies. That's not meant derogatively: there's a problem, they have been notified, yet they cannot do anything about it. They need someone else to fix it. So, they cry loud and clear that there is a need to fix this. It's difficult to understand sometimes, but they are pressured from above, and they don't have the means to fix it by themselves.
  - caseyy 2y ago
    Good management is a tool they have to fix the problem. But yes, sometimes they have mismanaged things so much that they can only kick up a fuss or pester people. It causes a lot of stress and is not very effective from my exp. On a more positive note, other managers will have a rapport with their teams where if the team only learns of a problem, it will be addressed. They only need to act as liaisons — stick something on a kanban board somewhere, and it will get resolved. No loud crying necessary, just business as usual.

vlunkr 2y ago

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

blastro 2y ago
Every time the stock market shits itself, the large brokerages seem to "go down"
Alupis 2y ago
The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer forever. Customer loyalty is often fickle - why would they wait around for you to fix your stuff when they can simply place their order on Amazon or another retail website?
- vasco 2y ago
  You are of course right, but in practice I've seen many more incidents created by doing changes to make things more robust than from simpler incremental product changes that usually are feature flagged and so on. At deeper levels usually there's less ability to do such containment (or is too expensive or takes too long or people are lazy) and so many times I wonder if it's better to do the trade-off or just keep things simple and eat only the "simple" sources of downtime to fix. For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.

fragmede 2y ago

There are even times to prefer to site being down to being up. If you've been hacked and the site is serving CSAM/malware/leaking PII maliciously, as the SRE on deck, your job is to keep it down!

Keyframe 2y ago

not saying you're wrong, because you absolutely aren't, but there's one key difference - usually freakouts are when it's client-facing related in combination with no control and/or information on what's going on and 'they' have to answer customer calls. That's when panic mode is activated.

cgh 2y ago

> It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. One time a few years ago, a coworker called me to tell me the stress we were under was causing her hair to fall out. I mentioned I'd lost significant weight and wasn't sleeping well, especially since another coworker had quit and now I was shouldering his responsibilities (programming and managing a junior). She said she was going to quit. I wished her the best. Around six months later, I had to quit too. It took around a year for me to get back to normal sleep patterns and for my heart to stop racing in the night. Oh yeah, another guy from that team had a "cardiac event" due to stress and had to take a month off.

ryandrake 2y ago
This is such an insane mentality for a rank-and-file employee who does not have significant upside tied to the business. If you're the sole shareholder of the company and there's some technical problem causing it to stop making money, then I can see it having a stressful effect on you and your health. If you're a worker bee who got 0.00000001% of the company's equity (if you're lucky), why oh why would you let the company's problems cause you to miss sleep and have a literal heart attack?

burningChrome 2y ago

>> Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems THIS. Worked for a huge corporation when I first started contracting as a developer. This was a daily occurrence. Someone would push some stuff to prod and it would break their very large e-commerce site. Sometimes it was minor stuff, other times it would cripple the entire site. Most of the time, nobody GAF at all. I remember sitting at my desk one day and my senior dev walking by my desk like Lumbergh in Office Space. Coffee in hand and just casually said to me, "Looks like someone broke the build, most of the video game pages are throwing 404's right now, going to lunch, see you soon." and then just walked away. I also found out one day one of the CDN folks that was running the Akamai account quit and then a month later, we found out one of our secondary content servers had been down since the day he left. When dude left, he never transferred any of his knowledge to the other team members so once they found out the server was down, it took another two days to get in contact with someone at Akamai who was handling our account. At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""?

burntwater 2y ago
>> At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""? That's funny, because reading your first sentence I was thinking that was the dysfunctional place. I had a boss that would take problems from 0 to 10 in a flash, when the problem was really a 4 or 5, and it really was not a great environment.
- ryandrake 2y ago
  Yea, nobody seems to be able to handle problems that require solid, but moderate, non-urgent effort. The problem is either a 0, where nobody even knows, let alone cares that it's happening or a 10, where it's treated as though everyone's chair is on fire and the company is losing $10M per millisecond.