7 ms·

Global Outage of AWS CloudFront CDN on Nov 26 2014

89 points by sajal83 12y ago

ceejayoz 12y ago
I'm really freaking tired of the AWS Status page using the green "info" marker for major outages.
lxfontes 12y ago
bottom line: always CNAME your asset domain.
- mikejholly 12y ago
  Totally. We were able to side step the CDN in less than a minute. +1
  - Nagyman 12y ago
    That helps! I guess it assumes your origin servers have the capacity to serve the traffic. Or did you switch CDNs? Just curious as there are services like Cedexis (or TurboBytes; the author of the article) for load balancing CDNs
- publicprivacy 12y ago
  What exactly is an asset domain? The one you setup to point to the CDN?
  - ceejayoz 12y ago
    Yes. Instead of using pqowieurpqoiwer.cloudfront.net, you can point one of your own subdomains at it. Leaves you in control. The only complexity is SSL certificates - you either have to pay $600/month for a custom cert, or use SNI and ditch some older IEs.
    - sbarre 12y ago
      $600 per month for what exactly? (honestly asking here)
      - ceejayoz 12y ago
        Without SNI, you need a dedicated IP per certificate. For CloudFront, this means dozens of dedicated IPs around the world just for your distribution.
        sbarre 12y ago
        Ah, makes sense! Thanks
      - deleted 12y ago
        [deleted]
    - micro-ram 12y ago
      Could you add a global variable to your site code with the fqdn of your assets to point to the origin for times like this?
    - Xorlev 12y ago
      They do offer free SSL with SNI. But that's not viable for most people given the number of popular older browsers missing SNI support.
      - kbuck 12y ago
        Browsers that do support SNI include: IE7+ (unless running on Windows XP) Firefox 2.0 and later Opera 8.0 and later Chrome 6 and later (unless running on Windows XP) Safari 3.0 and later Android default browser on Android 3.0 and later (this is probably the biggest chunk of users) Windows Phone 7 So, realistically, you're looking at people who still use Windows XP (unless they're using Firefox) and people with really old Android phones that'll never receive a manufacturer firmware update.
- cbsmith 12y ago
  Meh. Adds an extra DNS lookup. Just have a short TTL and let people who ignore it suffer. ;-)
  - deleted 12y ago
    [deleted]
- eknkc 12y ago
  And burn down your backend servers. Assuming you are not swithing to another CDN that fast.
aidos 12y ago
It's the second time in less than a week that AWS have failed to update their status page during service disruption and then filed it under 'increased error rates'. Don't get me wrong, I love AWS and use it every day, but it's a little insulting to pretend that things haven't gone wrong when they quite obviously have. I know Amazon aren't the most transparent company in the world but why even bother having a status page if everyone has to turn to twitter to validate the outage? Last night while sitting around waiting for cloudfront to come back so I could get on with my work I turned to HN to give me a bit of reading material. My word, sooooo many of the articles linked to on the front page were missing their static assets because of the AWS outage. Would be really interesting to know what percentage of the internet was affected.
- jbrooksuk 12y ago
  1&1 are guilty of this too.
  - kbar13 12y ago
    1&1 is the scum of the internet.
    - paromi 12y ago
      care to explain this ? whats wrong with 1&1 ?
      - iancarroll 12y ago
        Bad services. Someone even social engineered our 1&1 account a few years back...
      - jbrooksuk 12y ago
        Terrible technical setup, we have a dedicated server that can't be guaranteed an upgrade without losing the absolute server paths.
- deleted 12y ago
  [deleted]
EGreg 12y ago
Is there some way to do a fallback if the DNS server is down? Also if my host is down can I "quickly" try another host? I know that it's possible in my own app, but can I tell a browser to do it, as a web publisher?
stevenh 12y ago
This is incredibly upsetting. Amazon publishing false status reports for public consumption is directly equivalent to libeling every single customer they have, simultaneously. I was trying to register a new domain during this outage. When my registrar's site wasn't loading correctly, I dug around and discovered they use AWS. I then checked Amazon's AWS status page to see if they were the culprit. Nothing but green icons across the board. Amazon having the audacity to tell such a gargantuan lie seemed inconceivable, so I was absolutely convinced that the problem was the registrar's fault. I decided the registrar must have become shoddy at some point without me noticing, and so I made a "to-do" note for myself to switch to another registrar this weekend. If I hadn't happened to stumble across this HN submission, the registrar I use would have lost a long-time customer due to Amazon publishing a maliciously false status page for AWS.

89 points by sajal83 12y ago

ceejayoz 12y ago

I'm really freaking tired of the AWS Status page using the green "info" marker for major outages.

lxfontes 12y ago

bottom line: always CNAME your asset domain.

mikejholly 12y ago
Totally. We were able to side step the CDN in less than a minute. +1
- Nagyman 12y ago
  That helps! I guess it assumes your origin servers have the capacity to serve the traffic. Or did you switch CDNs? Just curious as there are services like Cedexis (or TurboBytes; the author of the article) for load balancing CDNs
publicprivacy 12y ago
What exactly is an asset domain? The one you setup to point to the CDN?
- ceejayoz 12y ago
  Yes. Instead of using pqowieurpqoiwer.cloudfront.net, you can point one of your own subdomains at it. Leaves you in control. The only complexity is SSL certificates - you either have to pay $600/month for a custom cert, or use SNI and ditch some older IEs.
  - sbarre 12y ago
    $600 per month for what exactly? (honestly asking here)
    - ceejayoz 12y ago
      Without SNI, you need a dedicated IP per certificate. For CloudFront, this means dozens of dedicated IPs around the world just for your distribution.
      - sbarre 12y ago
        Ah, makes sense! Thanks
    - deleted 12y ago
      [deleted]
  - micro-ram 12y ago
    Could you add a global variable to your site code with the fqdn of your assets to point to the origin for times like this?
  - Xorlev 12y ago
    They do offer free SSL with SNI. But that's not viable for most people given the number of popular older browsers missing SNI support.
    - kbuck 12y ago
      Browsers that do support SNI include: IE7+ (unless running on Windows XP) Firefox 2.0 and later Opera 8.0 and later Chrome 6 and later (unless running on Windows XP) Safari 3.0 and later Android default browser on Android 3.0 and later (this is probably the biggest chunk of users) Windows Phone 7 So, realistically, you're looking at people who still use Windows XP (unless they're using Firefox) and people with really old Android phones that'll never receive a manufacturer firmware update.
cbsmith 12y ago
Meh. Adds an extra DNS lookup. Just have a short TTL and let people who ignore it suffer. ;-)
- deleted 12y ago
  [deleted]
eknkc 12y ago
And burn down your backend servers. Assuming you are not swithing to another CDN that fast.

aidos 12y ago

It's the second time in less than a week that AWS have failed to update their status page during service disruption and then filed it under 'increased error rates'. Don't get me wrong, I love AWS and use it every day, but it's a little insulting to pretend that things haven't gone wrong when they quite obviously have. I know Amazon aren't the most transparent company in the world but why even bother having a status page if everyone has to turn to twitter to validate the outage? Last night while sitting around waiting for cloudfront to come back so I could get on with my work I turned to HN to give me a bit of reading material. My word, sooooo many of the articles linked to on the front page were missing their static assets because of the AWS outage. Would be really interesting to know what percentage of the internet was affected.

jbrooksuk 12y ago
1&1 are guilty of this too.
- kbar13 12y ago
  1&1 is the scum of the internet.
  - paromi 12y ago
    care to explain this ? whats wrong with 1&1 ?
    - iancarroll 12y ago
      Bad services. Someone even social engineered our 1&1 account a few years back...
    - jbrooksuk 12y ago
      Terrible technical setup, we have a dedicated server that can't be guaranteed an upgrade without losing the absolute server paths.
deleted 12y ago
[deleted]

EGreg 12y ago

Is there some way to do a fallback if the DNS server is down? Also if my host is down can I "quickly" try another host? I know that it's possible in my own app, but can I tell a browser to do it, as a web publisher?

stevenh 12y ago

This is incredibly upsetting. Amazon publishing false status reports for public consumption is directly equivalent to libeling every single customer they have, simultaneously. I was trying to register a new domain during this outage. When my registrar's site wasn't loading correctly, I dug around and discovered they use AWS. I then checked Amazon's AWS status page to see if they were the culprit. Nothing but green icons across the board. Amazon having the audacity to tell such a gargantuan lie seemed inconceivable, so I was absolutely convinced that the problem was the registrar's fault. I decided the registrar must have become shoddy at some point without me noticing, and so I made a "to-do" note for myself to switch to another registrar this weekend. If I hadn't happened to stumble across this HN submission, the registrar I use would have lost a long-time customer due to Amazon publishing a maliciously false status page for AWS.