5 ms·

>Our on-call was pretty bad (40-60 tickets a week) and there was very little investment being put in to improve it. We had a lot of little scripts here and ther

>Our on-call was pretty bad (40-60 tickets a week) and there was very little investment being put in to improve it. We had a lot of little scripts here and there which would solve extremely specific situations but no focus was ever put on in building a general framework or trying to reduce the ticket count. AWS engineer here and I confirm everything you say, but this quote really struck home with me. The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues. I've worked across multiple teams and there is an "institutional ego" at Amazon where everyone, especially L7s+, think that Amazon is the best/smartest company in the world and have an attitude of "if Amazon, the best company ever, hasn't already figured out a way to solve this problem, then it must be an unsolvable problem and we won't even try". The thing is, a lot of these problems are in no way unique to Amazon and many other companies across the world have already found fantastic solutions to reduce things like on-call load. But adopting those solutions would require admitting that other companies were able to solve something that Amazon hasn't, which would hurt the ego. This all applies to the very issue being talked about in the OP, too. Even managers will vent to you about how their team goes through 50% attrition every year, and how everyone is overloaded and finding new engineers is hard. They just accept 50% attrition as "something that just happens every year" as if having such a shitty team is normal, and there is no movement at all to fix it.

kablow 5y ago
> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues. In my case, direct management seems interested in these issues and understand there are problems we need to fix, but ultimately the feature/product launches always make it into the sprint and the larger bug fixes never do. It's very much "actions speak louder than words".
EliRivers 5y ago
Related to part of what you said. I've worked in a half dozen different industries; I've worked in little companies with a half-dozen employees and globe spanning companies with tens of thousands and employees. They all think that they have special unique problems that nobody else has - 90% of it is the same problem I've seen in other companies in different industries, with different industry specific acronyms and words. Every damn job, the same basic problems over and over, with the insistence that these problems are specific to the industry and usually to that specific company and that specific product.
- mirker 5y ago
  Same is true in academic research (though old-timers catch it often). The common pattern is: * approach “A” was invented in 1970 or so and didn’t work * “B” extends “A” in multiple ways and now works * noobs assume “B” invented “A” and treat “B” as the root of modern knowledge. “B” often has more market presence so noobs (without deep understanding) don’t see the relationship to prior attempts. Examples: * AlexNet/deep learning/ML in general * MapReduce/databases/functional programming primitives * Docker/chroot “Jail” Containers * Bitcoin/90s coins/blockchains Ego and ignorance are rarely a great combination :)
  - doggodaddo78 5y ago
    The Emperor's new clothes (TENC). Most people don't do history, especially the Dunning-Kruger afflicted. Docker (Linux containers) is, like jails: awful, leaky isolation pretending to be virtualization. If you want real resource and security containment, use virtualization. Docker is insecure in so many ways; it's like using PHP to write a TLS library.
- doggodaddo78 5y ago
  "NIH" duplication of effort with "IH" bugs. SMH. Will they ever learn?
vlovich123 5y ago
Can you share your knowledge or reading materials for how to reduce on-call load? I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things. If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.
- awsthro00945 5y ago
  I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people. On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis. This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different. It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.
  - YZF 5y ago
    There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.
throwaway210620 5y ago
The lack of investment into decreasing on-call pain is a real factor. I work on Oracle cloud (OCI) and at least some of the orgs (VP-down) have figured out that this is something worth focusing on, and the on-call gets better and better as a result. My original team had an average of something like 50 pager-worthy (sev2) events per week until we got moved into a new org that had the right philosophy and we relentlessly drove that down because management realized that engineers made miserable by mundane ops fake-emergencies would eventually get fed up and leave, and that's not what they wanted (afaik, OCI has no such forced attrition). So we got put on a program of relentlessly tracking and categorizing the sev2 counts and committing to improving those numbers over a period of time. 25% of dev sprints were dedicated to improving ops (tools, better alarms, fixing long-backlogged bugs that led to pages), and now that team's ops are pretty easy and they are free to work on new features, which everyone prefers. I've since moved to another team whose ops had already had this optimization done, and I've never experienced a bad week of on call there. I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.
taway_zonian257 5y ago
> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues. SDE1 here. IMMV of course but on my corner of the org I've seen a bunch of team members raising concerns regarding tech debt for the L7 to shut it down as it got in the way of delivering the features he wanted to deliver. Also the elephant in the room is how the company relies on trial by fire as a form of performance evaluation, which involves inexperienced SDEs being pushed to deliver alone chunks of major projects in spite of lack of experience or insight.
rodgerd 5y ago
> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Jeff Bezos can afford a private space program because - in part - his Amazon retail model is based on working people until they are mentally and physically broken, while paying them a pittance, discarding them, and moving onto the next group of people. He would rather spend money on crying booths and astroturf campaigns and buying newspapers than change that. Why would he think about the expendable meat-units in AWS any differently?
- fellellor 5y ago
  Because We are knowledge workers, Not manual labourers! /s

kablow 5y ago

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues. In my case, direct management seems interested in these issues and understand there are problems we need to fix, but ultimately the feature/product launches always make it into the sprint and the larger bug fixes never do. It's very much "actions speak louder than words".

EliRivers 5y ago

Related to part of what you said. I've worked in a half dozen different industries; I've worked in little companies with a half-dozen employees and globe spanning companies with tens of thousands and employees. They all think that they have special unique problems that nobody else has - 90% of it is the same problem I've seen in other companies in different industries, with different industry specific acronyms and words. Every damn job, the same basic problems over and over, with the insistence that these problems are specific to the industry and usually to that specific company and that specific product.

mirker 5y ago
Same is true in academic research (though old-timers catch it often). The common pattern is: * approach “A” was invented in 1970 or so and didn’t work * “B” extends “A” in multiple ways and now works * noobs assume “B” invented “A” and treat “B” as the root of modern knowledge. “B” often has more market presence so noobs (without deep understanding) don’t see the relationship to prior attempts. Examples: * AlexNet/deep learning/ML in general * MapReduce/databases/functional programming primitives * Docker/chroot “Jail” Containers * Bitcoin/90s coins/blockchains Ego and ignorance are rarely a great combination :)
- doggodaddo78 5y ago
  The Emperor's new clothes (TENC). Most people don't do history, especially the Dunning-Kruger afflicted. Docker (Linux containers) is, like jails: awful, leaky isolation pretending to be virtualization. If you want real resource and security containment, use virtualization. Docker is insecure in so many ways; it's like using PHP to write a TLS library.
doggodaddo78 5y ago
"NIH" duplication of effort with "IH" bugs. SMH. Will they ever learn?

vlovich123 5y ago

Can you share your knowledge or reading materials for how to reduce on-call load? I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things. If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.

awsthro00945 5y ago
I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people. On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis. This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different. It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.
- YZF 5y ago
  There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.

throwaway210620 5y ago

The lack of investment into decreasing on-call pain is a real factor. I work on Oracle cloud (OCI) and at least some of the orgs (VP-down) have figured out that this is something worth focusing on, and the on-call gets better and better as a result. My original team had an average of something like 50 pager-worthy (sev2) events per week until we got moved into a new org that had the right philosophy and we relentlessly drove that down because management realized that engineers made miserable by mundane ops fake-emergencies would eventually get fed up and leave, and that's not what they wanted (afaik, OCI has no such forced attrition). So we got put on a program of relentlessly tracking and categorizing the sev2 counts and committing to improving those numbers over a period of time. 25% of dev sprints were dedicated to improving ops (tools, better alarms, fixing long-backlogged bugs that led to pages), and now that team's ops are pretty easy and they are free to work on new features, which everyone prefers. I've since moved to another team whose ops had already had this optimization done, and I've never experienced a bad week of on call there. I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.

taway_zonian257 5y ago

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues. SDE1 here. IMMV of course but on my corner of the org I've seen a bunch of team members raising concerns regarding tech debt for the L7 to shut it down as it got in the way of delivering the features he wanted to deliver. Also the elephant in the room is how the company relies on trial by fire as a form of performance evaluation, which involves inexperienced SDEs being pushed to deliver alone chunks of major projects in spite of lack of experience or insight.

rodgerd 5y ago

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Jeff Bezos can afford a private space program because - in part - his Amazon retail model is based on working people until they are mentally and physically broken, while paying them a pittance, discarding them, and moving onto the next group of people. He would rather spend money on crying booths and astroturf campaigns and buying newspapers than change that. Why would he think about the expendable meat-units in AWS any differently?

fellellor 5y ago
Because We are knowledge workers, Not manual labourers! /s