5 ms·

This seems like a major global outage. Good luck to those poor sysadmins over at Amazon, this must easily be in the millions lost per minute. Let's see whether

This seems like a major global outage. Good luck to those poor sysadmins over at Amazon, this must easily be in the millions lost per minute. Let's see whether this outage takes out HN with it once again ;-)

leddt 5y ago
More likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.
- xwolfi 5y ago
  I d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on
  - ROARosen 5y ago
    It's back
- lamontcg 5y ago
  I think peak Christmas time shopping was a couple million per hour back in 2006. I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).
  - oblio 5y ago
    Yeah, but it's been 15 years. Amazon traffic has probably grown since then, most likely a lot.
    - lamontcg 5y ago
      Do you think that might have been why I specifically cited the year in my comment?
icedistilled 5y ago
Well they can't blame AWS. AWS status pages shows no issues. (Although it never does even when there are)
- irjustin 5y ago
  AWS and Amazon run on completely separate systems/hardware. Amazon's internal, last i read, was years behind AWS's infrastructure and they're generally jealous of not having the same tools available. I tried finding the article, but maybe someone else can help me remember?
  - sudhirj 5y ago
    There was apparently always some kind of mix (AWS did start from internal requirements, though it quickly diverged). The OG services like S3 and DynamoDB power a lot of Amazon for sure. There’s been more bed of Amazon moving to AWS, most recently talk about shifting their main DBs.
  - rejectedandsad 5y ago
    That hasn't been 100% true for like four years. Most new things in Retail are built on AWS.
    - busymom0 5y ago
      Are you certain? As of last year, Amazon was switching to Fastly for their retain images: https://www.streamingmediablog.com/2020/05/fastly-amazon-homepage.html https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
      - deleted 5y ago
        [deleted]
      - pjbeam 5y ago
        Some retail is definitely on AWS. Source: I work at AWS.
  - 5y ago
ronnier 5y ago
An interesting thing about Amazon outages like this is that most people wait and come back and make the purchase.
- MattGaiser 5y ago
  Everyone else will still make me wait several days or make me go to a store.
- paxys 5y ago
  A lot of people who would have made impulse purchases don't though, once they have had time to reconsider.
  - MetaWhirledPeas 5y ago
    Right but when they return how do you distinguish between a new impulse and a continuation of the old? This is one of those cases where marketers and businesses love to act like they know everything when in fact it's a huge info blind spot.
  - 6c696e7578 5y ago
    There's probably a lot less going to landfill today. That said, maybe more getting thrashed too: https://www.itv.com/news/2021-06-21/amazon-destroying-millions-of-items-of-unsold-stock-in-one-of-its-uk-warehouses-every-year-itv-news-investigation-finds https://www.itv.com/news/2021-06-21/amazon-destroying-millio...
- colanderman 5y ago
  Not too surprising I think... for myself and I suspect many/most, the calculus is something like: A) check back in half an hour when Amazon will probably be working again or B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up. Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.
varenc 5y ago
The one Amazon property that seems unaffected is their China site (https://amazon.cn https://amazon.cn). No doubt its running entirely separate infrastructure, unlike every other country's local Amazon site. Example: https://www.amazon.cn/any-random-string-works-here/dp/0321992784 https://www.amazon.cn/any-random-string-works-here/dp/032199...
- sp332 5y ago
  Either everything is back, or amazon.com.au is also an exception.
  - fridif 5y ago
    Wouldn't be surprised if they serve .au from China
    - indemnity 5y ago
      Why? They have a lot of infrastructure in Sydney, that would make no sense.
rantwasp 5y ago
yeah, Amazon is past the stage where they had sysadmins. They now have armies of Operations People with SDEs being the fallback in case a runbook does not cover the current failure
- kazen44 5y ago
  what? Amazon definetly has operational people. whatever fancy name they give them.
  - rantwasp 5y ago
    my point is that you no longer have “classic” sysadmins and a lot of work is already done automatically by machines
dang 5y ago
I don't think Amazon outages have ever taken HN down (HN doesn't run on AWS). We've done it to ourselves a few times though.
- Sebb767 5y ago
  This was meant as a friendly nod to the fact that FAANG outages usually lead to a lot of people checking HN and generating insane amounts of load, like the time a few weeks ago when YouTube/Google Auth was down and took HN right with it :) But overall this site is extremely well run, so please don't take this as more than a joke.
  - dang 5y ago
    No worries! But I don't think this happened? > YouTube/Google Auth was down and took HN right with it Have I forgotten something obvious?
    - Sebb767 5y ago
      I think it was this [0] event. Some people commented on HN being overloaded [1], including me [2] :-) It was not completely down (iirc), but there were a lot of "Sorry, we cannot serve your request right now". [0] https://news.ycombinator.com/item?id=25415989 https://news.ycombinator.com/item?id=25415989 [1] https://news.ycombinator.com/item?id=25416390 https://news.ycombinator.com/item?id=25416390 [2] https://news.ycombinator.com/item?id=25416144 https://news.ycombinator.com/item?id=25416144

leddt 5y ago

More likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.

xwolfi 5y ago
I d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on
- ROARosen 5y ago
  It's back
lamontcg 5y ago
I think peak Christmas time shopping was a couple million per hour back in 2006. I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).
- oblio 5y ago
  Yeah, but it's been 15 years. Amazon traffic has probably grown since then, most likely a lot.
  - lamontcg 5y ago
    Do you think that might have been why I specifically cited the year in my comment?

icedistilled 5y ago

Well they can't blame AWS. AWS status pages shows no issues. (Although it never does even when there are)

irjustin 5y ago
AWS and Amazon run on completely separate systems/hardware. Amazon's internal, last i read, was years behind AWS's infrastructure and they're generally jealous of not having the same tools available. I tried finding the article, but maybe someone else can help me remember?
- sudhirj 5y ago
  There was apparently always some kind of mix (AWS did start from internal requirements, though it quickly diverged). The OG services like S3 and DynamoDB power a lot of Amazon for sure. There’s been more bed of Amazon moving to AWS, most recently talk about shifting their main DBs.
- rejectedandsad 5y ago
  That hasn't been 100% true for like four years. Most new things in Retail are built on AWS.
  - busymom0 5y ago
    Are you certain? As of last year, Amazon was switching to Fastly for their retain images: https://www.streamingmediablog.com/2020/05/fastly-amazon-homepage.html https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
    - deleted 5y ago
      [deleted]
    - pjbeam 5y ago
      Some retail is definitely on AWS. Source: I work at AWS.
- 5y ago

ronnier 5y ago

An interesting thing about Amazon outages like this is that most people wait and come back and make the purchase.

MattGaiser 5y ago
Everyone else will still make me wait several days or make me go to a store.
paxys 5y ago
A lot of people who would have made impulse purchases don't though, once they have had time to reconsider.
- MetaWhirledPeas 5y ago
  Right but when they return how do you distinguish between a new impulse and a continuation of the old? This is one of those cases where marketers and businesses love to act like they know everything when in fact it's a huge info blind spot.
- 6c696e7578 5y ago
  There's probably a lot less going to landfill today. That said, maybe more getting thrashed too: https://www.itv.com/news/2021-06-21/amazon-destroying-millions-of-items-of-unsold-stock-in-one-of-its-uk-warehouses-every-year-itv-news-investigation-finds https://www.itv.com/news/2021-06-21/amazon-destroying-millio...
colanderman 5y ago
Not too surprising I think... for myself and I suspect many/most, the calculus is something like: A) check back in half an hour when Amazon will probably be working again or B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up. Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.

varenc 5y ago

The one Amazon property that seems unaffected is their China site (https://amazon.cn https://amazon.cn). No doubt its running entirely separate infrastructure, unlike every other country's local Amazon site. Example: https://www.amazon.cn/any-random-string-works-here/dp/0321992784 https://www.amazon.cn/any-random-string-works-here/dp/032199...

sp332 5y ago
Either everything is back, or amazon.com.au is also an exception.
- fridif 5y ago
  Wouldn't be surprised if they serve .au from China
  - indemnity 5y ago
    Why? They have a lot of infrastructure in Sydney, that would make no sense.

rantwasp 5y ago

yeah, Amazon is past the stage where they had sysadmins. They now have armies of Operations People with SDEs being the fallback in case a runbook does not cover the current failure

kazen44 5y ago
what? Amazon definetly has operational people. whatever fancy name they give them.
- rantwasp 5y ago
  my point is that you no longer have “classic” sysadmins and a lot of work is already done automatically by machines

dang 5y ago

I don't think Amazon outages have ever taken HN down (HN doesn't run on AWS). We've done it to ourselves a few times though.

Sebb767 5y ago
This was meant as a friendly nod to the fact that FAANG outages usually lead to a lot of people checking HN and generating insane amounts of load, like the time a few weeks ago when YouTube/Google Auth was down and took HN right with it :) But overall this site is extremely well run, so please don't take this as more than a joke.
- dang 5y ago
  No worries! But I don't think this happened? > YouTube/Google Auth was down and took HN right with it Have I forgotten something obvious?
  - Sebb767 5y ago
    I think it was this [0] event. Some people commented on HN being overloaded [1], including me [2] :-) It was not completely down (iirc), but there were a lot of "Sorry, we cannot serve your request right now". [0] https://news.ycombinator.com/item?id=25415989 https://news.ycombinator.com/item?id=25415989 [1] https://news.ycombinator.com/item?id=25416390 https://news.ycombinator.com/item?id=25416390 [2] https://news.ycombinator.com/item?id=25416144 https://news.ycombinator.com/item?id=25416144