42 ms·

Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative abo

Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.

alpb 1y ago
Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
- andrewflnr 1y ago
  > Organizations like this survive these brain brains. True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
  - dredmorbius 1y ago
    There's no argument that the end is neigh. There is an argument, specifically, that "This is a tipping point moment." Not the end, and possibly not even the beginning of the end. But the start of what is likely to be a long and persistent slide.
- dredmorbius 1y ago
  Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." From TFA.
  - losvedir 1y ago
    I read that as "of 100 people who quit voluntarily, we wish 69-81 of them hadn't". But that number is meaningless without the context of how many people are quitting out of how many are there, not to mention onboarding processes and how fast new hires get up to speed.
    - inemesitaffia 1y ago
      Not to mention Amazon vs AWS
strunz 1y ago
The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.
- ed_elliott_asc 1y ago
  It’s a bit of a guess though isn’t it?
  - HelloNurse 1y ago
    It's the most plausible, fact-based guess, beating other competing theories. Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty. An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place. AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
    - almostgotcaught 1y ago
      > It's the most plausible, fact-based guess, beating other competing theories. "My wildly conjectural and self-serving theory is not only correct, it is the most correct". Lol perfectly represents the arrogance of hn.
      - HelloNurse 1y ago
        The article describes evidence for a concrete, straightforward organizational decay pattern that can explain a large part of this miserable failure. What's "self-serving" about such a theory? My personal "guess" is that failing to retain knowledge and talent is only one of many components of a well-rounded crisis of bad management and bad company culture that has been eroding Amazon on more fronts than AWS reliability. What's your theory? Conspiracy within Amazon? Formidable hostile hackers? Epic bad luck? Something even more movie-plot-like? Do you care about making sense of events in general?

alpb 1y ago

Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.

andrewflnr 1y ago
> Organizations like this survive these brain brains. True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
- dredmorbius 1y ago
  There's no argument that the end is neigh. There is an argument, specifically, that "This is a tipping point moment." Not the end, and possibly not even the beginning of the end. But the start of what is likely to be a long and persistent slide.
dredmorbius 1y ago
Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." From TFA.
- losvedir 1y ago
  I read that as "of 100 people who quit voluntarily, we wish 69-81 of them hadn't". But that number is meaningless without the context of how many people are quitting out of how many are there, not to mention onboarding processes and how fast new hires get up to speed.
  - inemesitaffia 1y ago
    Not to mention Amazon vs AWS

strunz 1y ago

The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.

ed_elliott_asc 1y ago
It’s a bit of a guess though isn’t it?
- HelloNurse 1y ago
  It's the most plausible, fact-based guess, beating other competing theories. Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty. An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place. AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
  - almostgotcaught 1y ago
    > It's the most plausible, fact-based guess, beating other competing theories. "My wildly conjectural and self-serving theory is not only correct, it is the most correct". Lol perfectly represents the arrogance of hn.
    - HelloNurse 1y ago
      The article describes evidence for a concrete, straightforward organizational decay pattern that can explain a large part of this miserable failure. What's "self-serving" about such a theory? My personal "guess" is that failing to retain knowledge and talent is only one of many components of a well-rounded crisis of bad management and bad company culture that has been eroding Amazon on more fronts than AWS reliability. What's your theory? Conspiracy within Amazon? Formidable hostile hackers? Epic bad luck? Something even more movie-plot-like? Do you care about making sense of events in general?