4 ms·

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

by akdev1l 3mo ago

qurren 3mo ago
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
- tyre 3mo ago
  Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
  - geodel 3mo ago
    "Former" seems to an important detail here.
    - gleenn 3mo ago
      If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
      - FabCH 3mo ago
        Not the OP, but: The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
    - switchbak 3mo ago
      If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else. Or in my case, actively ignore any and all recruiting from that sesspool.
  - zelphirkalt 3mo ago
    Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
    - nullsanity 3mo ago
      [dead]
    - tmpz22 3mo ago
      AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this. Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
      - embedding-shape 3mo ago
        > Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect. The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
  - mendigou 3mo ago
    I am former AWS and this is pretty accurate. The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
    - akdev1l 3mo ago
      Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways
      - NBJack 3mo ago
        And many of which have been shipped across the pond to be someone else's problem.
    - NBJack 3mo ago
      [dead]
  - qurren 3mo ago
    Yes, I am a former AWS employee. I got put on Focus because my "contributions were not coming through" to leadership.
- nullorempty 3mo ago
  This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption". There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job. And no one is shooting for better than OK.
  - hoppp 3mo ago
    Yeah but this also negates the argument that people gotta use it or they get left behind. If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
- mightyham 3mo ago
  Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience. COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
  - eventualcomp 3mo ago
    Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org. Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
    - dlenski 3mo ago
      This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries. There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
  - awakeasleep 3mo ago
    Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve. If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations. That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
- thewebguyd 3mo ago
  > take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
  - qurren 3mo ago
    The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
    - rootsudo 3mo ago
      This, exactly 100%
  - dice 3mo ago
    How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?
    - thewebguyd 3mo ago
      Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.
01284a7e 3mo ago
If only there was some way to get anomaly detection services [1] inside of AWS... 1: https://aws.amazon.com/what-is/anomaly-detection/ https://aws.amazon.com/what-is/anomaly-detection/
mcpherrinm 3mo ago
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills. I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_and_tsunami https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped. I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure) It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
- oenton 3mo ago
  > there's definitely this sort of monitoring in place. Surely somebody was paged. Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
Groxx 3mo ago
They've already got anomaly detection: their users.
el1s7 3mo ago
I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
dclowd9901 3mo ago
Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

by akdev1l 3mo ago

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

qurren 3mo ago

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).

tyre 3mo ago
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
- geodel 3mo ago
  "Former" seems to an important detail here.
  - gleenn 3mo ago
    If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
    - FabCH 3mo ago
      Not the OP, but: The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
  - switchbak 3mo ago
    If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else. Or in my case, actively ignore any and all recruiting from that sesspool.
- zelphirkalt 3mo ago
  Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
  - nullsanity 3mo ago
    [dead]
  - tmpz22 3mo ago
    AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this. Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
    - embedding-shape 3mo ago
      > Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect. The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
- mendigou 3mo ago
  I am former AWS and this is pretty accurate. The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
  - akdev1l 3mo ago
    Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways
    - NBJack 3mo ago
      And many of which have been shipped across the pond to be someone else's problem.
  - NBJack 3mo ago
    [dead]
- qurren 3mo ago
  Yes, I am a former AWS employee. I got put on Focus because my "contributions were not coming through" to leadership.
nullorempty 3mo ago
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption". There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job. And no one is shooting for better than OK.
- hoppp 3mo ago
  Yeah but this also negates the argument that people gotta use it or they get left behind. If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
mightyham 3mo ago
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience. COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
- eventualcomp 3mo ago
  Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org. Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
  - dlenski 3mo ago
    This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries. There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
- awakeasleep 3mo ago
  Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve. If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations. That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
thewebguyd 3mo ago
> take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
- qurren 3mo ago
  The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
  - rootsudo 3mo ago
    This, exactly 100%
- dice 3mo ago
  How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?
  - thewebguyd 3mo ago
    Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.

01284a7e 3mo ago

If only there was some way to get anomaly detection services [1] inside of AWS... 1: https://aws.amazon.com/what-is/anomaly-detection/ https://aws.amazon.com/what-is/anomaly-detection/

mcpherrinm 3mo ago

While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills. I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_and_tsunami https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped. I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure) It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.

oenton 3mo ago
> there's definitely this sort of monitoring in place. Surely somebody was paged. Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.

Groxx 3mo ago

They've already got anomaly detection: their users.

el1s7 3mo ago

I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.

dclowd9901 3mo ago

Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.