6 ms·

Amazon.com was down

68 points by gregsadetsky 6y ago

gregsadetsky 6y ago
I checked and checked again before submitting. https://downforeveryoneorjustme.com/amazon https://downforeveryoneorjustme.com/amazon All green here -- https://status.aws.amazon.com/ https://status.aws.amazon.com/ -- although I cannot login to AWS as one of the redirects in the flow is through an amazon.com url.
- dmlittle 6y ago
  I know the AWS status page is known for delayed updates but AWS itself is up. Amazon.com (retail) is what is down.
michaelmior 6y ago
Down for me as well.
eappleby 6y ago
seems to be up on mobile
- gregsadetsky 6y ago
  To which URL are you connecting? amazon.ca is up for instance.
  - eappleby 6y ago
    amazon.com and aws.amazon.com
    - gregsadetsky 6y ago
      aws.amazon.com works here while amazon.com does not, over both my home internet connection and mobile (via my carrier's LTE connection). Which continent are you on out of curiosity?
      - eappleby 6y ago
        I'm in the US (New York), but Amazon.com just went down for me on mobile (and the app). AWS is accessible though.
        leesalminen 6y ago
        DNS Lookups to www.amazon.com are what's failing.
leesalminen 6y ago
% nslookup www.amazon.com Server: 8.8.8.8 Address: 8.8.8.8#53 server can't find www.amazon.com: SERVFAIL Colorado, USA
- deleted 6y ago
  [deleted]
- gregsadetsky 6y ago
  When I do `dig amazon.com @8.8.8.8` I get a number of round-robin IP addresses. But `nslookup` returns SERVFAIL. Why are they showing different results?
  - leesalminen 6y ago
    looks like `amazon.com` is just a redirect to `www.amazon.com`. `dig www.amazon.com @8.8.8.8` returns nada. % curl -v https://amazon.com https://amazon.com HTTP/1.1 301 Moved Permanently Location: https://www.amazon.com/ https://www.amazon.com/ Could also be that your default DNS isn't 8.8.8.8?
    - gregsadetsky 6y ago
      You're right, thanks. `dig www.amazon.com` for both `@8.8.8.8` and `@8.8.4.4` return no IP addresses. Same for CloudFlare's `1.1.1.1` ... ouch.
      - leesalminen 6y ago
        Ouch indeed! :)
- jlkinsel 6y ago
  Actually, the www was working. It's a cname to tp.47cf2c8c9-frontier.amazon.com, which should have been a cname to d3ag4hukkh62yn.cloudfront.net, but that record was broken for about 20 minutes. Update: that frontier cname seems to be bouncing between Cloudfront and Akamai now, so I suspect they're fighting off either an attack or internal issues. I see issues to other route53 servers as well.
taf2 6y ago
It's down for me too...
kgraves 6y ago
so much for high availability... edit: I'm don't understand these swarm of downvotes? If I was a VP/CXO of a $1.2TN company's website being down for even an hour, I would be in a fit of rage and would request a full investigation of how this happened.
- dmlittle 6y ago
  Mistakes happen, highly available systems fail, being enraged and demanding heads isn't going to make anyone happy but just scared. A full investigation is probably going to happen and steps will be taken to make sure this doesn't happen again but it's hard to blame it on a particular individual. Ideally you'd have steps such that one person alone can't screw up and it also requires other people to let that happen.
  - kgraves 6y ago
    hmmm, maybe you're right, could have been a gremlin that had chewed on one of the cables.
    - dmlittle 6y ago
      It's not a gremlin and at the end of the day it's probably a misconfiguration or bug somewhere but it doesn't help anyone to point fingers and blame. A system probably went through a design phase, why didn't the reviewers of the design didn't catch this? The implementation was probably reviewed by peers, why didn't they catch this? It might have gone through QA, why didn't they catch this? And the list can go on and on. At the end of the day a mistake was made and it should be fixed but being enraged doesn't help anyone. Figure out what went wrong, how to make sure it doesn't happen again and do it.
  - downerending 6y ago
    Shit happens, and rage is pointless. Nonetheless, it's worth noting that even the best HA just isn't that good. Certainly I wouldn't care to fly in a plane that critically depended on any web provider, even Amazon.
Xavdidtheshadow 6y ago
https://smile.amazon.com/ https://smile.amazon.com/ is working for me. I've got a chrome extension that redirects, so I didn't even notice the issue.
- dalfonso 6y ago
  Same here. Smile Always for Chrome.
rubyron 6y ago
Wow, down for me too (DNS doesn't resolve, using google dns). Wonder how many $$millions they're losing every minute.
- leesalminen 6y ago
  Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
  - michaelmior 6y ago
    The calculation is accurate in one sense, but I don't think it represents actual revenue loss for two reasons. Firstly, spending is non-uniform and peaks a lot on Black Friday, near Christmas, etc. Second, assuming the site is down for only a few minutes (it's back now), a significant fraction of people will probably just try again later. I still imagine this is a significant loss.
- Reedx 6y ago
  Impossible to say really, since most will just come back later. Especially given that it'll likely be brief. In any case, must be like a 5 alarm fire at Amazon right now!
- flatiron 6y ago
  guess who just spent 10 minutes debugging pi-hole, this guy
  - hackeraccount 6y ago
    Thank you!
- aficiomaquinas 6y ago
  Specifically the revenue that won't come back once the website is back online.
mdorazio 6y ago
Down from LA also. Really curious what their revenue loss per minute is in this kind of situation. Edit: based on product-only revenue for Q1 2020 (since AWS was unaffected), it looks like around $325K per minute if Amazon is down for all of North America. Naively assuming that people wouldn't just wait to order, of course. That's actually lower than I thought.
- leesalminen 6y ago
  Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
  - jumbopapa 6y ago
    Yeah, but how many people come back to make a purchase after the site is back up? I don't think you can just extrapolate that 1 minute down equals $500k lost revenue.
    - leesalminen 6y ago
      For sure, just napkin math for fun. Also, I believe that number includes revenue from non-amazon.com sources (like AWS).
- gregsadetsky 6y ago
  AMZN's revenue is about 10k / second.
- Simulacra 6y ago
  Sorry, it's a Virginia data center thing. We're working on it.
  - warrentr 6y ago
    Can you elaborate on the issue/fix?
- INTPenis 6y ago
Erlangen 6y ago
Cannot log into aws console at the moment.
quaffapint 6y ago
Amazing how many times simple configuration issues can kill you.
aficiomaquinas 6y ago
Aaand it's back.
- rubyron 6y ago
  Yep, back up for me, too. Lucky them - DNS outtages can sometimes take a lot longer to resolve (heh).
amaajemyfren 6y ago
Seems to work now. Nairobi, Kenya.
confiq 6y ago
GCPL Do you need help? :troll:
tly_alex 6y ago
DNS doesn't resolve somehow. works now.
wglb 6y ago
No it is not. Let's not post this generally incorrect type of post.
- outworlder 6y ago
  You waited too long to post.

68 points by gregsadetsky 6y ago

I checked and checked again before submitting. https://downforeveryoneorjustme.com/amazon https://downforeveryoneorjustme.com/amazon All green here -- https://status.aws.amazon.com/ https://status.aws.amazon.com/ -- although I cannot login to AWS as one of the redirects in the flow is through an amazon.com url.

dmlittle 6y ago
I know the AWS status page is known for delayed updates but AWS itself is up. Amazon.com (retail) is what is down.

michaelmior 6y ago

Down for me as well.

eappleby 6y ago

seems to be up on mobile

gregsadetsky 6y ago
To which URL are you connecting? amazon.ca is up for instance.
- eappleby 6y ago
  amazon.com and aws.amazon.com
  - gregsadetsky 6y ago
    aws.amazon.com works here while amazon.com does not, over both my home internet connection and mobile (via my carrier's LTE connection). Which continent are you on out of curiosity?
    - eappleby 6y ago
      I'm in the US (New York), but Amazon.com just went down for me on mobile (and the app). AWS is accessible though.
      - leesalminen 6y ago
        DNS Lookups to www.amazon.com are what's failing.

leesalminen 6y ago

% nslookup www.amazon.com Server: 8.8.8.8 Address: 8.8.8.8#53 server can't find www.amazon.com: SERVFAIL Colorado, USA

deleted 6y ago
[deleted]
gregsadetsky 6y ago
When I do `dig amazon.com @8.8.8.8` I get a number of round-robin IP addresses. But `nslookup` returns SERVFAIL. Why are they showing different results?
- leesalminen 6y ago
  looks like `amazon.com` is just a redirect to `www.amazon.com`. `dig www.amazon.com @8.8.8.8` returns nada. % curl -v https://amazon.com https://amazon.com HTTP/1.1 301 Moved Permanently Location: https://www.amazon.com/ https://www.amazon.com/ Could also be that your default DNS isn't 8.8.8.8?
  - gregsadetsky 6y ago
    You're right, thanks. `dig www.amazon.com` for both `@8.8.8.8` and `@8.8.4.4` return no IP addresses. Same for CloudFlare's `1.1.1.1` ... ouch.
    - leesalminen 6y ago
      Ouch indeed! :)
jlkinsel 6y ago
Actually, the www was working. It's a cname to tp.47cf2c8c9-frontier.amazon.com, which should have been a cname to d3ag4hukkh62yn.cloudfront.net, but that record was broken for about 20 minutes. Update: that frontier cname seems to be bouncing between Cloudfront and Akamai now, so I suspect they're fighting off either an attack or internal issues. I see issues to other route53 servers as well.

taf2 6y ago

It's down for me too...

kgraves 6y ago

so much for high availability... edit: I'm don't understand these swarm of downvotes? If I was a VP/CXO of a $1.2TN company's website being down for even an hour, I would be in a fit of rage and would request a full investigation of how this happened.

dmlittle 6y ago
Mistakes happen, highly available systems fail, being enraged and demanding heads isn't going to make anyone happy but just scared. A full investigation is probably going to happen and steps will be taken to make sure this doesn't happen again but it's hard to blame it on a particular individual. Ideally you'd have steps such that one person alone can't screw up and it also requires other people to let that happen.
- kgraves 6y ago
  hmmm, maybe you're right, could have been a gremlin that had chewed on one of the cables.
  - dmlittle 6y ago
    It's not a gremlin and at the end of the day it's probably a misconfiguration or bug somewhere but it doesn't help anyone to point fingers and blame. A system probably went through a design phase, why didn't the reviewers of the design didn't catch this? The implementation was probably reviewed by peers, why didn't they catch this? It might have gone through QA, why didn't they catch this? And the list can go on and on. At the end of the day a mistake was made and it should be fixed but being enraged doesn't help anyone. Figure out what went wrong, how to make sure it doesn't happen again and do it.
- downerending 6y ago
  Shit happens, and rage is pointless. Nonetheless, it's worth noting that even the best HA just isn't that good. Certainly I wouldn't care to fly in a plane that critically depended on any web provider, even Amazon.

Xavdidtheshadow 6y ago

https://smile.amazon.com/ https://smile.amazon.com/ is working for me. I've got a chrome extension that redirects, so I didn't even notice the issue.

dalfonso 6y ago
Same here. Smile Always for Chrome.

rubyron 6y ago

Wow, down for me too (DNS doesn't resolve, using google dns). Wonder how many $$millions they're losing every minute.

leesalminen 6y ago
Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
- michaelmior 6y ago
  The calculation is accurate in one sense, but I don't think it represents actual revenue loss for two reasons. Firstly, spending is non-uniform and peaks a lot on Black Friday, near Christmas, etc. Second, assuming the site is down for only a few minutes (it's back now), a significant fraction of people will probably just try again later. I still imagine this is a significant loss.
Reedx 6y ago
Impossible to say really, since most will just come back later. Especially given that it'll likely be brief. In any case, must be like a 5 alarm fire at Amazon right now!
flatiron 6y ago
guess who just spent 10 minutes debugging pi-hole, this guy
- hackeraccount 6y ago
  Thank you!
aficiomaquinas 6y ago
Specifically the revenue that won't come back once the website is back online.

mdorazio 6y ago

Down from LA also. Really curious what their revenue loss per minute is in this kind of situation. Edit: based on product-only revenue for Q1 2020 (since AWS was unaffected), it looks like around $325K per minute if Amazon is down for all of North America. Naively assuming that people wouldn't just wait to order, of course. That's actually lower than I thought.

leesalminen 6y ago
Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
- jumbopapa 6y ago
  Yeah, but how many people come back to make a purchase after the site is back up? I don't think you can just extrapolate that 1 minute down equals $500k lost revenue.
  - leesalminen 6y ago
    For sure, just napkin math for fun. Also, I believe that number includes revenue from non-amazon.com sources (like AWS).
gregsadetsky 6y ago
AMZN's revenue is about 10k / second.
Simulacra 6y ago
Sorry, it's a Virginia data center thing. We're working on it.
- warrentr 6y ago
  Can you elaborate on the issue/fix?
INTPenis 6y ago

Erlangen 6y ago

Cannot log into aws console at the moment.

quaffapint 6y ago

Amazing how many times simple configuration issues can kill you.

aficiomaquinas 6y ago

Aaand it's back.

rubyron 6y ago
Yep, back up for me, too. Lucky them - DNS outtages can sometimes take a lot longer to resolve (heh).

amaajemyfren 6y ago

Seems to work now. Nairobi, Kenya.

confiq 6y ago

GCPL Do you need help? :troll:

tly_alex 6y ago

DNS doesn't resolve somehow. works now.

wglb 6y ago

No it is not. Let's not post this generally incorrect type of post.

outworlder 6y ago
You waited too long to post.