5 ms·

My experience there (15 years ago) was that on-call was terrible because line management was unable or unwilling to invest in fixing root causes of operational

by jp57 2y ago

My experience there (15 years ago) was that on-call was terrible because line management was unable or unwilling to invest in fixing root causes of operational issues. When I started I lucked into a situation where I was one engineer a "team" of two. We didn't have a manager and were reporting to the director of our department. He only had about an hour a week to meet with us. We spent a lot of time fixing broken stuff that we'd inherited (a task that I actually found kind of fun), and soon our ops load started going down. We eventually got another engineer and a manager who was willing to prioritize fixing the root causes of our on-call tickets. During black-friday-week of my second year there we had essentially no operational issues and spent our time brainstorming future work while we kept an eye our performance dashboards. We got semi-scolded by a senior engineer from a neighboring team because we didn't "seem very busy". Our manager called that a win. Even back then Amazon had the reputation for being a brutal place to work and for burning out engineers, but I rather liked it. I ultimately left because my wife hated living in Seattle.

hypeatei 2y ago
> We got semi-scolded by a senior engineer from a neighboring team because we didn't "seem very busy" What the hell? Hope you told him off, not his job or his business. Weird.
- goostavos 2y ago
  Stuff like that can almost always be traced back to that senior being told to "be visible." Show up! Have opinion on things (loudly)! "Scale yourself!" Other mumbo jumbo. It often leads to these weird misguided drive-bys where everyone is left confused.
  - kevinventullo 2y ago
    Yet here we are talking about the person… they sound pretty influential to me. (I’m only half-joking)
- jp57 2y ago
  Well, he rolled up to the same director and was the most senior engineer under that director, so it was a little bit his business. And, like I said it was only semi scolded: he came to me and quietly said something like, "you guys don't seem very busy", which I took to mean "why are you loudly brainstorming future work when the guys in the next row of cubes over haven't slept in 36 hours?" My answer was, "All our stuff is working."
  - ethbr1 2y ago
    I heard it expressed years ago, when the role was still call sysops, that it was the one job where the better you were then the less work you did. It was attached to a similar anecdote about someone being yelled at for crafting a well-oiled system.
Twirrim 2y ago
> line management was unable or unwilling to invest in fixing root causes of operational issues. Sorry for an obligatory: there is no such thing as a root cause. That said, that matches my general experience too (I left about 9 years ago). Unless the S-team specifically calls them out for any particular metric, it's not going to get touched. Even then they'll try and game the metric. Sev2 rate is too high, let's find some alarms that are behind lots of false positives, and just make them sev3 instead, rather than investigate why. No way it can backfire... wait what do you mean I had an outage and didn't know, because the alarm used to fire legitimately too? That major S3 collapse several years ago was caused by a component that engineers had warned leadership about for at least 4-5 years when I was there. They'd carefully gathered data, written reports, written up remediation plans that weren't particularly painful. Engineers knew it was in an increasingly fragile state. It took the outage for leadership to recognise that maybe, just maybe, it was time to follow the plan laid out by engineering. I can't talk about the actual what/why of that component, but if I did it'd have you face palming, because it was painfully obvious before the incident that an incident was inevitable. Unfortunately, it seems like an unwillingness to invest in operations just pervades the tech industry. So many folks I speak to across a wide variety of tech companies are constantly having to fight to get operations considered any kind of a priority. No one gets promoted for performing miracles keeping stuff running.
- akulbe 2y ago
  I'm curious why you say "there is no such thing as a root cause". Is this because that's what you genuinely believe, or was this just Amazon culture?
  - willthames 2y ago
    Perhaps more accurate is that there is no one root cause, but a bunch of contributing factors. I had a quick search to find my favourite article on this one, it might be https://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallacy/ https://willgallego.com/2018/04/02/no-seriously-root-cause-i... as it demolishes Five Whys but https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... is a classic

by jp57 2y ago

hypeatei 2y ago

> We got semi-scolded by a senior engineer from a neighboring team because we didn't "seem very busy" What the hell? Hope you told him off, not his job or his business. Weird.

goostavos 2y ago
Stuff like that can almost always be traced back to that senior being told to "be visible." Show up! Have opinion on things (loudly)! "Scale yourself!" Other mumbo jumbo. It often leads to these weird misguided drive-bys where everyone is left confused.
- kevinventullo 2y ago
  Yet here we are talking about the person… they sound pretty influential to me. (I’m only half-joking)
jp57 2y ago
Well, he rolled up to the same director and was the most senior engineer under that director, so it was a little bit his business. And, like I said it was only semi scolded: he came to me and quietly said something like, "you guys don't seem very busy", which I took to mean "why are you loudly brainstorming future work when the guys in the next row of cubes over haven't slept in 36 hours?" My answer was, "All our stuff is working."
- ethbr1 2y ago
  I heard it expressed years ago, when the role was still call sysops, that it was the one job where the better you were then the less work you did. It was attached to a similar anecdote about someone being yelled at for crafting a well-oiled system.

Twirrim 2y ago

> line management was unable or unwilling to invest in fixing root causes of operational issues. Sorry for an obligatory: there is no such thing as a root cause. That said, that matches my general experience too (I left about 9 years ago). Unless the S-team specifically calls them out for any particular metric, it's not going to get touched. Even then they'll try and game the metric. Sev2 rate is too high, let's find some alarms that are behind lots of false positives, and just make them sev3 instead, rather than investigate why. No way it can backfire... wait what do you mean I had an outage and didn't know, because the alarm used to fire legitimately too? That major S3 collapse several years ago was caused by a component that engineers had warned leadership about for at least 4-5 years when I was there. They'd carefully gathered data, written reports, written up remediation plans that weren't particularly painful. Engineers knew it was in an increasingly fragile state. It took the outage for leadership to recognise that maybe, just maybe, it was time to follow the plan laid out by engineering. I can't talk about the actual what/why of that component, but if I did it'd have you face palming, because it was painfully obvious before the incident that an incident was inevitable. Unfortunately, it seems like an unwillingness to invest in operations just pervades the tech industry. So many folks I speak to across a wide variety of tech companies are constantly having to fight to get operations considered any kind of a priority. No one gets promoted for performing miracles keeping stuff running.

akulbe 2y ago
I'm curious why you say "there is no such thing as a root cause". Is this because that's what you genuinely believe, or was this just Amazon culture?
- willthames 2y ago
  Perhaps more accurate is that there is no one root cause, but a bunch of contributing factors. I had a quick search to find my favourite article on this one, it might be https://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallacy/ https://willgallego.com/2018/04/02/no-seriously-root-cause-i... as it demolishes Five Whys but https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... is a classic