12 ms·

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

602 points by fredoliveira 14y ago

deleted 14y ago
[deleted]
- ChrisClark 14y ago
  Did you try their status page? https://status.heroku.com/ https://status.heroku.com/
- nixme 14y ago
  There have been multiple tweets mirroring the status page: http://twitter.com/herokustatus http://twitter.com/herokustatus
- TallboyOne 14y ago
  They've given both status updates and tweets. So much for actually reading first.
orourkek 14y ago
It's degraded performance for some EBS volumes in a single availability zone - isn't this title a bit sensationalistic?
- thomaslutz 14y ago
  No. Amazon just understates the issues. Reddit, Heroku etc. all have problems.
  - orourkek 14y ago
    Even if they do understate the issue, It's limited to US-EAST 1, and is an EBS issue. Saying that EC2 is "down" because of this is totally off the mark - I've got dozens of EBS volmes in EAST 1 that are unaffected, plus all of the other zones that are operating normally...
    - thomaslutz 14y ago
      EC2 instances are affected by the EBS issues. But you're right, it's not correct that EC2 is "down".
      - onenine 14y ago
        Not all instances are EBS backed.
  - evan_ 14y ago
    EC2 is not down. My EC2 sites are all functioning.
    - enjo 14y ago
      All my instances across 3 zones are functioning normally.
- trimbo 14y ago
  From what I'm seeing, if your root disk is on EBS and your SSH keys are there, you cannot SSH into those hosts right now. Also, the availability zones are disparate in terms of what they can support. A great number of my instances are in 1d because of unavailability in others.
  - acdha 14y ago
    If you're relying on individual servers to have drives which never fail you're doing reliability wrong.
  - ConstantineXVI 14y ago
    Note that the region-1[a b c d & such] designations are randomized per account; my us-east-1d won't (necessarily) be the same as yours or anyone else's.
    - trimbo 14y ago
      Ah, I didn't know that. Thanks!
- drone 14y ago
  "Degraded performance," is a fairly off-handed way of putting what they're experiencing. First RDS connectivity went down the tube, and then EBS followed, finally the EC2 console is failing to operate properly (for me, in US-EAST region at least). Of course, as soon as I read the report that the issue was confined to one AZ, I looked to move my server over to another AZ. Oh yes, two were full and refused new instances, and then surprisingly, new requests for the other AZs never were received or operated on - and now the console is failing. It's a bit more than just "slow EBS" if that's what you were thinking. - edit: said ec2 twice in the second sentence, corrected to say ebs.
ojiikun 14y ago
Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.
- milkshakes 14y ago
  too bad their own dashboard and management interface don't fail over... http://db.tt/BcuoSnPu http://db.tt/BcuoSnPu
- driverdan 14y ago
  Amazon is downplaying the problem. It is affecting many large sites.
  - res0nat0r 14y ago
    Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.
    - mey 14y ago
      http://downrightnow.com/netflix http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.
      - res0nat0r 14y ago
        The last Netflix post mortem mentioned they had a bug in their configuration where they kept sending traffic to already down ELB instances, which was the cause of the last outage for them if I remember correctly.
        Uchikoma 14y ago
        AirBnB is also down.
pcorsaro 14y ago
My site is currently down because of this.
lgleason 14y ago
Now is a good time to stroke your dev pair :) http://news.ycombinator.com/item?id=4680178 http://news.ycombinator.com/item?id=4680178
thehigherlife 14y ago
I jokingly said the internet was broken this morning as I've been having weird connectivity issues all morning. Funny that I was sort of right.
halis 14y ago
Thanks I have a friend whose site is down from this and this helps shed some light on it.
jaysonelliot 14y ago
I suppose this explains why Airbnb is down now, as well: http://aws.amazon.com/solutions/case-studies/airbnb/ http://aws.amazon.com/solutions/case-studies/airbnb/ I was just in the middle of booking a stay in a Palo Alto startup embassy for this week, too!
- taigeair 14y ago
  seems they should have a back up
- mikeevans 14y ago
  They've been down for several hours actually. I tried to get to their site this morning to check out the jedburg talk, and got the "We're on the case" message.
losvedir 14y ago
Sigh, my heroku site is down.
Revex 14y ago
I just recently signed up for a EC2 (3 weeks ago) and my small site is also down. :-(
HannibalLecter 14y ago
Reddit had a big story that criticized Amazon's role in deleting a woman's kindle library and basically telling her to go fuck herself.
jaequery 14y ago
isn't it about time they supported a multi-regional failover system?
gailees 14y ago
WAT!?
bas 14y ago
Some of our sites are affected. Good times!
playhard 14y ago
As of now , I don't have a problem with my site. Elastic beanstalk with ec2 small instances.
jmvoodoo 14y ago
All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting a single zone is bs.
- Banduin 14y ago
  Our multi-zoned RDS instance was able to fail over to another zone with minimal downtime. It took about 2 minutes.
  - jmvoodoo 14y ago
    Lucky you? :)
- modsearch 14y ago
  Our multi-AZ RDS instance did not failover correctly this time although it has in the past..
  - rynop 14y ago
    same here, no fail-over
    - jmccree 14y ago
      I had one multi-az failover correctly, however the security group was refusing connections to the web servers ec2 security group. I had to manually add in the private ips of the ec2 instances. It appears the API issue is affecting security group to ip lookups.
- abailin 14y ago
  Same here, our servers are still up, but our RDS instance is down.
- 14y ago
benwerd 14y ago
This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?
- petejansson 14y ago
  The reference architectures and white papers here are helpful: http://aws.amazon.com/architecture/ http://aws.amazon.com/architecture/
- caseysoftware 14y ago
  We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastructure-in-the-cloud-evan-cooke-web-20-expo-nyc-2011 http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasnt-affected-by-todays-aws-issues/ http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.
  - caseysoftware 14y ago
    And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
    - wyck 14y ago
      Yet netflix is currently down, so maybe there is a problem with the chimp army?
      - bermanoid 14y ago
        The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered. There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not robust against "single AZ outages", given that they're involved in pretty much every one of these incidents and make them affect people on the entire cloud...
diego 14y ago
The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.
- merlincorey 14y ago
  I believe this is because North Virginia is also their historical first facility.
  - ceejayoz 14y ago
    And largest, and busiest.
- acangiano 14y ago
  Realistically, it's at least in part because everyone defaults to the East region. So it's the most crowded and demanding on the system.
  - jey 14y ago
    Yeah, it's got to be much larger than the other regions, so it makes sense that we see more errors. Since error_rate = machines * error_rate_per_machine.
    - nasmorn 14y ago
      The whole region is down, you just calculated the chance of at least one machine having an error.
      - jey 14y ago
        No, I calculated the error rate for the region. If us-east-1 has 5 times the machines (or availability zones, or routers, or EBS backplanes, or other thing-that-can-fail) as us-west-1, we would expect to see us-east-1 have each type of error occur about 5 times as often as us-west-1.
Codhisattva 14y ago
Worst infrastructure ever.
LaSombra 14y ago
Coursera is also down.
ceejayoz 14y ago
My dashboard says "Degraded EBS performance in a single Availability Zone". It then lists each of the five zones as "Availability zone is operating normally." http://cl.ly/image/202F3B0I371g http://cl.ly/image/202F3B0I371g
- dirtae 14y ago
  I was seeing that too, but it looks like Amazon has now updated the availability zone status. When I run ec2-describe-availability-zones from the command line, it's telling me that us-east-1b is impaired. (Availability zone mapping is different for each account, so my 1b may be some other availability zone for you.)
- ulope 14y ago
  CloudApp seems to have trouble too
gtaylor 14y ago
Does anyone know if this really is just one AZ? Seems like an awful lot of larger sites are down. I'd expect at least some of them to be multi-AZ by now.
- rynop 14y ago
  I'm multi-az and am being impacted. I also use multi-az RDS adn its being impacted. So calling BS on the 1 az impact.
  - ejdyksen 14y ago
    One of our Multi-AZ RDS instances failed over successfully. It was down for about 5-10 minutes.
- colinhowe 14y ago
  Looks like multi-AZ to us too
- dsl 14y ago
  Multiple AZs here.
lr 14y ago
pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."
- Terretta 14y ago
  That's editorializing. For many sites, a single server in a single zone (e.g., a non redundant server, an instance, a slice, a VM, whatever) is the right decision for ROI. For many sites, the money spent on redundancy could be better spent on, say, Google Adwords, until they're big enough that a couple hours downtime has irreplaceable costs higher than the added costs of redundancy (dev, hosting, admin) for a year.
  - lr 14y ago
    Yes, it is editorializing. My point is, but I guess too subtle, that the current link text is very much an editorial comment, especially since the content at that link location has nothing to do with the sites mentioned in the link text.
- lgsilver 14y ago
  This issue is affecting both an EC2 Zone and Amazon's RDS servers, which are technically multi-zone. There are a ton of well-architected apps and sites that have been affected. Unconstructive...
- bdcravens 14y ago
  Tons has been written about Netflix's architecture, and they're down as well. I believe Amazon isn't being totally transparent.
  - grumps 14y ago
    Personally, starting to believe the Amazon is quickly becoming public enemy #2.
    - 14y ago
Ramonaxvh 14y ago
This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.
- gfodor 14y ago
  If you go through the pains of architecting your system to span multiple AZs, or you avoid using EBS, then you probably dodge most of the EC2 outages. (Remains to be seen if that is the case here.) That said, I don't think most people think using the cloud means that downtime is a thing of the past. I think the more attractive proposition is when hardware breaks, or meteors hit the datacenter, etc, it is their problem, not yours. You still have to deal with software-level operations, but hardware-level operations is effectively outsourced. The question is if you think you can do a better job than Amazon -- some companies think they can, most startups know they can't.
  - w_t_payne 14y ago
    Yeah. Even with this, they still do better than I would. My record: misconfigured air-conditioning unit alarm leading to servers being baked at high temperature over a weekend, leading to much wailing and gnashing of teeth. I now know to be really careful to set up air conditioning units properly, but what other lessons am I still waiting to learn? The main lesson that I took from this is that I should stick to what I am good at: cutting code & chewing data. :-)
    - gfodor 14y ago
      Yeah this is another important point. Part of the cost of AWS is also a bit of an insurance policy against you physically breaking your servers :)
    - gfodor 14y ago
      Yeah this is another important point. Part of the cost of AWS is also a bit of an insurance policy against you physically breaking your servers :)
Ramonaxvh 14y ago
This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.
lgsilver 14y ago
Two of our four EC2 servers are down. Both have RDS connections and EBS storage. Looks like the RDS connection has something to do with it...
- spartango 14y ago
  RDS is built on EBS, so if EBS has problems you can bet that RDS will feel the effects.
interro 14y ago
just checked my site and it is up now!
fsckin 14y ago
My instances just came back up.
tyw 14y ago
I reacted quickly enough to get a new instance spun up and added it to my site's load balancer, but the load balancer is failing to recognize the new instance and send traffic to it... yay. Console is totally unresponsive on the load balancer's "instances" tab. If I point my browser at the ec2 public DNS of the new instance it seems to be running just fine. So much for the load balancer being helpful.
gailees 14y ago
At least HN isn't down! :D
taigeair 14y ago
back up :)
- SquareWheel 14y ago
  As far as I can see Reddit is still down.
dave1619 14y ago
Our app is down because it's hosted on Heroku and it's frustrating because it seems like N Virginia is the least reliable Amazon datacenter. Every year it seems to go down a couple times for at least a couple hours. Heroku should offer a choice between N Virginia and Oregon hosting (I think they're almost comparable in price nowadays). That way people who want more uptime/reliability can choose Oregon. Sure it will be further from Europe (but then it will be closer to Asia) and people can make that choice on their own. But basing an entire hosting service on N Virginia doesn't make sense anymore, considering the history of major downtime in that region.
- nlh 14y ago
  Or better yet, Heroku should offer an add-on "instant failover" service that, for a premium of course, offers a package for multi-site (or, knowing they're 100% AWS, multi-datacenter) deployment with all of the best practices, etc. Seems like a logical next step for them (or a competitor) given the recent spate of outages.
gorkemcetin 14y ago
Countly mobile analytics platform servers are also affected from this issue (http://count.ly http://count.ly). Waiting and waiting. :) @caseysoftware thanks for the links, we now have something to read and implement this week.
Naushad 14y ago
The Console itself is down, <html><body>Http/1.1 Service Unavailable</body> </html> Its high time AWS does something now.
jeaguilar 14y ago
This is affecting more than a single Availability Zone, but probably for reasons that have been seen before. One reason might be that an EBS failure in one AZ triggers a spike in EBS activity in other AZs which overwhelms EBS. (I believe this is what happened in April 2011). Does anybody have any experience with migrating to Oregon or N. California in terms of speed and latency?
btilly 14y ago
As https://twitter.com/DEVOPS_BORAT https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)
- engtech 14y ago
  so true: "In devops is turtle all way down but at bottom is perl script. " https://twitter.com/DEVOPS_BORAT/status/248770195580125185 https://twitter.com/DEVOPS_BORAT/status/248770195580125185
- eloisant 14y ago
  To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
  - dennisgorelik 14y ago
    SoftLayer that we use seems to be much more reliable than Amazon. At least more reliable than that particular Amazon's datacenter in Virginia.
    - x3sphere 14y ago
      I agree, been with SL for 3 years, never had an outage apart from a drive failure one time but it was fixed within an hour.
  - apendleton 14y ago
    But this is part of the problem: we have multiple web properties, and the fact that AWS issues can affect all of them at once is a huge downside. Certainly, if we ran on metal, we would have hardware fail, but failures would be likely to be better-isolated than at Amazon.
kellyhclay 14y ago
Can't update my list fast enough, but other major services experiencing problems are Netflix and Pinterest. Lots of other (smaller) sites are starting to fail too. http://www.forbes.com/sites/kellyclay/2012/10/22/amazon-aws-goes-down-again-takes-reddit-with-it/ http://www.forbes.com/sites/kellyclay/2012/10/22/amazon-aws-...
run4yourlives 14y ago
Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)
- IheartApplesDix 14y ago
  Because reddit and ycombinator are the center of the tech sociosphere?
  - farinasa 14y ago
    For the less specialized groups, kind of, yeah.
- UnoriginalGuy 14y ago
  Well if that was their goal then it has been a success as they have taken out a lot of the forums and discussion sites which were critical of them. Except HN.
- macrael 14y ago
  This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts because of a conspiracy, let's belittle their efforts because of bad systems design.
  - Angostura 14y ago
    It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
    - tuple 14y ago
      actually, there's probably some few idiots who will believe it.
espeed 14y ago
When is Google Compute Engine supposed to be open to everyone?
- quotemstr 14y ago
  There's Azure.
blaze33 14y ago
Hosted on heroku with a hosted heroku postgres dev instance, I observed a drop in the db response time : - from a consistent average of 10ms over the last week - to a new consistent average of ~2.5ms beetwen 17:31 and 17:35 UTC. AWS started to report the current issue at 17:38 UTC. My app then experienced some intermittent issues (reported by newrelic pinging it every 30s). Don't know if it's related, could it be some sort of hw upgrade that went wrong ? I did a push affecting my most used queries but that was one and a half hour sooner, at 15:56 UTC, so probably unrelated.
jasonkester 14y ago
For added fun, their EC2 console is down. I got this for a while: <html><body>Http/1.1 Service Unavailable</body> </html> ... then an empty console saying "loading" for the last 20 minutes. Then recently it upgraded to saying "Request limit exceeded." in place of the loading message (because hey, I'd refreshed the page four times over the course of 20 minutes). On the upside, their status page shows all green lights.
- hornbaker 14y ago
 They've acknowledged that at http://status.aws.amazon.com/ http://status.aws.amazon.com/ for awhile now with a tiny "i" status icon (I can't load my instances pane): 12:07 PM PDT We are experiencing elevated error rates with the EC2 Management Console.
 - mrb 14y ago
 Amazon misrepresents reality on this status page! They have standardized icons to represent various levels of issues (orange = perf issue, red = disruption). But they don't even use them. Instead they add [i] to the green icons to indicate perf issues (Amazon Elastic Compute Cloud - N. Virginia) and disruptions of service (Amazon Relational Database Service - N. Virginia). Maybe this status page is controlled by marketing bozos who want to pretend the situation is not so bad.
- adrianbg 14y ago
 Does Amazon explain anywhere what regions, availability zones or whatever have single points of failure for each of their services? I guess that's supposed to be what an "availability zone" is but somehow that doesn't quite seem to capture it. It's pretty hard to build reliable apps without knowing where the single points of failure are in the underlying infrastructure.
bdcravens 14y ago
Noticed that Rackspace was down for a bit - I presumed they're getting hammered on their WWW as people are considering an infrastructure switch?
zenwheels 14y ago
Perfectly timed with our product launch, now our site is down too - www.zenwheels.com
Yrlec 14y ago
Does this just affect normal EBS or EBS with provisioned IOPS as well?
timmythebest 14y ago
a ton of other stuff is down too... www.reddit.com, netflix, www.freewebsite.com and a bunch more
mtgx 14y ago
Time to switch to Google's Compute Engine.
rsync 14y ago
This reminds me of an ad campaign we sometimes run on reddit: http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws_and_your_offsite_backups/ http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws... ... which you'll be able to click as soon as reddit is back up :)
incision 14y ago
Unfortunate. I have to deal with a number of folks who will be overjoyed to read this news when their tech cartel vendor of choice forwards it this evening. There's a huge contingent of currently endangered infrastructure folks (and vendors who feed off them) out there who throw a party every time AWS has a visible outage.
- druiid 14y ago
  I'd question if they are truly wrong to send out news like this? Basically you have to weigh the realities of the 'cloud' with the perception. The perception that many people has is that somehow the 'cloud' is a magical up-time device that will save you money in droves. The reality is that for many companies with mid-level traffic and aren't a start-up with a billion users, the 'old' style tech might very well be your best option.
- rdl 14y ago
  AWS sucking at availability (and especially specific parts like EBS, and then services built on top of EBS and on top of AWS) doesn't mean the correct option is to mine your own sand and dig up your own coal to run servers in your own datacenters in the basement. Even if you're totally sold on the cloud, you can still have a requirement that things be transparent all the way down. AWS is one of the least transparent hosting options around. If you're a customer of a regular colo, or even a managed hosting provider themselves based at a colo, it's pretty easy to dig into how the infrastructure is set up, identify areas where you need your own redundancy, etc. Essentially impossible within AWS -- there is no reason intelligible to me that ELB in multiple AZs should depend on EBS in a single AZ, but that's how they have it set up.
- tisme 14y ago
  > who throw a party every time AWS has a visible outage. At the going rate they'll be AA members before long.
fredoliveira 14y ago
One of the most frustrating issues here is that we have to deal with Amazon's status page for information. It's a complex page, divided by continent instead of region, which means at least 5 or 6 clicks to figure out progress. They should learn from these issues about how people want to be informed - to date, they haven't. Also, they have a twitter account, which would be the perfect fit to keep everyone up-to-date with what's going on; to at least show a human side to these issues. Alas, they're not updating that either. I've been working with AWS since early 2006 when they first launched - I was lucky to be granted a VIP invite to try out EC2 before everyone else, and ended up launching the first public app on EC2. This might be the first time when frustration has overcome my love for these guys.
Latricerzj 14y ago
Single point of failure?
johansch 14y ago
The emperor has no clothes!
gavinlynch 14y ago
Disappointing, but I guess not totally unexpected. Our prod site was down for a bit with many others.
patrickgzill 14y ago
You have a virtualized platform, on top of which are many pieces like load-balancing, EBS, RDS, the control-plane itself, etc. You have burstable network connections which by their nature, will have hard limits (you can't burst above 10Gbps on a 10Gbps pipe, for example; even assuming the host machine is connected to a 10Gbps port). Burstable (meaning quite frankly, over-provisioned) disk and CPU resources. And if any piece fails, you may well have downtime... It is always surprising to me, that people feel that layering complexity upon complexity, will result in greater reliability.
gourneau 14y ago
These semi-regular outages are why I am going to use the Google Cloud Platform for my next project.
1SaltwaterC 14y ago
ELB is also fucked. Seen that nobody mentions it. Some of our load balancers are completely unreachable. Just one multi-AZ RDS instance claimed the automatic failover. However, the 200+ alerts over the automatic failover due to internal DNS changes to point to the new master shows that things aren't as easily described by the RDS DB Events log. Some instances reported high disk I/O (EBS backed) via the New Relic agent (the console still has some issues). So far, this is what I see from my side.
orph 14y ago
Adding insult to injury: when trying to view instances in the console I now only see "Rate limit exceeded".
rdl 14y ago
Why the fuck do the two most critical services (ELB and Console) have depends on their historically most unreliable pile of shit (EBS)?
- taligent 14y ago
  Seriously THIS has to be addressed. I can tolerate EC2/EBS going down but why on earth is ELB/Console always going down at the same time ?
sigkill 14y ago
Isn't it worrying that there are so many services/sites that are so dependent on Amazon EC?
zenwheels 14y ago
Woot, Back Online www.zenwheels.com
cupcake_death 14y ago
Maybe I'm being daft. Why does this sh*t never seem to affect www.amazon.com
- mvelie 14y ago
  Amazon doesn't run their site off the same set of EC2 servers and systems that everyone else does. They just used their experience to build it.
  - cupcake_death 14y ago
    That would be the ultimate testament, maybe the only stunt they could pull to PR themselves out of this mess.. if www.amazon.com ran on the same set of EC2 servers and systems as us muggles
  - yap 14y ago
    Amazon runs all of the Amazon.com web servers on EC2 hosts (and has since 2010: https://www.youtube.com/watch?feature=player_detailpage&v=dxk8b9rSKOo#t=393s https://www.youtube.com/watch?feature=player_detailpage&...). They've just made sure that they have enough hosts in each of the other AZs to withstand an outage.
lucb1e 14y ago
Dropbox runs on S3, they are unaffected? And Heroku's website seems okay. Only reddit is down. That's because of this?
rynop 14y ago
EIP remapping is all messed up as well. api errors. console for it does not work either.
blhack 14y ago
I'm sorry, am I in a time machine? Didn't this exact thing happen last year, and didn't these exact people explain exactly how they were going to make sure it never happens again? What the hell is happening here?
valhallarecords 14y ago
This is why I trust Google's data centers http://www.google.com/about/datacenters/gallery/#/ http://www.google.com/about/datacenters/gallery/#/
- taligent 14y ago
  HN really needs to hold a "dumbest comments of the year" award. This has to be a strong contender.
  - JackWebbHeller 14y ago
    HN really needs to hold a "snarkiest response of the year" award.
    - pardner 14y ago
      Snap!
j45 14y ago
How does Twilio manage to stay up when AWS goes down, and how much of that can the average developer reasonably do?
- driverdan 14y ago
  See Keith's (of Twilio) response here: http://news.ycombinator.com/item?id=4684766 http://news.ycombinator.com/item?id=4684766
  - j45 14y ago
    Nice, thanks!
hornbaker 14y ago
Update from http://status.aws.amazon.com/ http://status.aws.amazon.com/: 2:20 PM PDT We've now restored performance for about half of the volumes that experienced issues. Instances that were attached to these recovered volumes are recovering. We're continuing to work on restoring availability and performance for the volumes that are still degraded. We also want to add some detail around what customers using ELB may have experienced. Customers with ELBs running in only the affected Availability Zone may be experiencing elevated error rates and customers may not be able to create new ELBs in the affected Availability Zone. For customers with multi-AZ ELBs, traffic was shifted away from the affected Availability Zone early in this event and they should not be seeing impact at this time.
- nkohari 14y ago
  This is infuriating. Our instances aren't ELB-backed, so we have an entire AZ sitting idle while the rest of our instances are overwhelmed with traffic. Why did they make this decision for us? I'm normally an Amazon apologist when outages happen, but this is absolutely ridiculous.
Lashawndazqd 14y ago
I wonder if this is going to be an attack or hardware/software failure..
johnmurray_io 14y ago
Here comes another round of Hacker News posts about how you should never host your app in just one availability zone...
pg 14y ago
Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.
- Splines 14y ago
  I'm not familiar with HN's technical stack (other than arc), how has it scaled as the community grew over the years?
  - rdl 14y ago
    I think it's a case of "well optimized code which does as little as possible can handle an awful lot of users on a single beefy box".
    - adrianbg 14y ago
      Until it can't. Then five years later you get Spanner.
      - rdl 14y ago
        You can buy some seriously big boxes, and easily split off a lot of services onto multiple boxes. The big problem with the "single big box" strategy is being able to do upgrades -- I see hn go down frequently for 5-10 min at a time in the middle of the night, which I assume is upgrades/reboots. The happy medium is probably splitting database (master/slave at least) and cdn (if needed) and some other services (AAA? logging?) out, and then having 2+ front end servers with load balancing for availability.
        ukd1 14y ago
        hn doesn't have a database capable of master/slave as such...so I think this will be harder if it ever becomes popular enough. I don't think it gets enough traffic it's ever likely to exceed what you can fit in a single box, from what I know.
Cordiapxq 14y ago
Workplace productivity is going to skyrocket today
potomak 14y ago
This is why you should think about a multi region architecture to prevent these types of service interruptions.
mattbillenstein 14y ago
I thought at least Reddit had learned the "don't use EBS" rule from past outages - I was bitten by the April/2011 outage and switched everything over to instance storage shortly thereafter. For most applications I think architecting EBS out should be straightforward - instance storage doesn't put a huge single-point-of-failure in the middle of your app if you're doing replication, failover, and backups properly. And EBS seems to be the biggest component of the recent AWS failures upon which they've built a lot of their other systems.
justplay 14y ago
i read in my book that amazon is the best cloud service provider and seeing this thing can't really prove it. I am still confused which cloud service i use ? Amazon is only one which Indians are currently rely on ,since latency is bit smaller for amazon servers as compared to other providers.
jbarham 14y ago
FWIW here's Google App Engine status page: http://code.google.com/status/appengine http://code.google.com/status/appengine
robbiet480 14y ago
I made this to express my outrage and hope to help others easily express theirs too! http://robbiet480.github.com/StopLyingToUsAmazon/ http://robbiet480.github.com/StopLyingToUsAmazon/
d0m 14y ago
Ironically, on the first hacking health event, our server was running heroku.. and heroku went down. 3 months later, we have the second hacking health in Toronto and all AWS is going down.
foxylad 14y ago
Not wanting to shill for Google, but we've been very happy with Appengine's reliability - particularly since moving to the HR datastore. Have other Appengine users had any significant downtime apart from MS datastore issues?
- valhallarecords 14y ago
  What site are you running on App Engine?
DavidThi808 14y ago
We came up with a way to get better reliability than AWS, and for less money. We list the 3 providers we combined to get it at http://go.windward.net/TLWAugust8_BasicWebinarFormLP.html http://go.windward.net/TLWAugust8_BasicWebinarFormLP.html Thanks Dave
cryptoz 14y ago
nell 14y ago
The tech industry is manufacturing its version of "Too big to fail" entity.
traderd65 14y ago
Building infrastructure on top of Amazon with global replication across multiple availability zones can sidestep such failures and guarantee uninterrupted operations to customers - www.pubnub.com is unaffected by AWS being down.
stinky613 14y ago
Somewhere Steve Wozniak is saying "I told you so"
briandear 14y ago
I fixed EC2 using this one weird trick..
GigabyteCoin 14y ago
It's back up now.
jasonlingx 14y ago
I am frankly surprised so many still use EC2 considering how frequently it breaks. It's not cheap, so the only reasons to use it would be reliability or scale right? Why not just get lots of boxes at Hetzner and OVH (40 EUR a month for 32GB of RAM and 4x2x3.4GHz cores) and scale up / redundansize that way?
deleted 14y ago
[deleted]
davidjnelson 14y ago
I have a few questions about HA AWS deployments. Anyone know the answers? It seems obvious that multi region for HA is ideal, but the reference architecture for web apps on the aws architecture page only uses multi AZ for HA. How do you bridge that disconnect? Route53 with super low ttl? Wouldn’t route53 then be a single POF? Also, is dynamodb multi region? If you have to use MySQL, how do you replicate cross region in a time and cost efficient manner? Isn’t S3 also only multi az, is there even a way to make s3 cross region? Or do you just “hope” dynamodb, route53, and s3 don’t fail? Thanks, Best, David

602 points by fredoliveira 14y ago

deleted 14y ago

[deleted]

ChrisClark 14y ago
Did you try their status page? https://status.heroku.com/ https://status.heroku.com/
nixme 14y ago
There have been multiple tweets mirroring the status page: http://twitter.com/herokustatus http://twitter.com/herokustatus
TallboyOne 14y ago
They've given both status updates and tweets. So much for actually reading first.

orourkek 14y ago

It's degraded performance for some EBS volumes in a single availability zone - isn't this title a bit sensationalistic?

thomaslutz 14y ago
No. Amazon just understates the issues. Reddit, Heroku etc. all have problems.
- orourkek 14y ago
  Even if they do understate the issue, It's limited to US-EAST 1, and is an EBS issue. Saying that EC2 is "down" because of this is totally off the mark - I've got dozens of EBS volmes in EAST 1 that are unaffected, plus all of the other zones that are operating normally...
  - thomaslutz 14y ago
    EC2 instances are affected by the EBS issues. But you're right, it's not correct that EC2 is "down".
    - onenine 14y ago
      Not all instances are EBS backed.
- evan_ 14y ago
  EC2 is not down. My EC2 sites are all functioning.
  - enjo 14y ago
    All my instances across 3 zones are functioning normally.
trimbo 14y ago
From what I'm seeing, if your root disk is on EBS and your SSH keys are there, you cannot SSH into those hosts right now. Also, the availability zones are disparate in terms of what they can support. A great number of my instances are in 1d because of unavailability in others.
- acdha 14y ago
  If you're relying on individual servers to have drives which never fail you're doing reliability wrong.
- ConstantineXVI 14y ago
  Note that the region-1[a b c d & such] designations are randomized per account; my us-east-1d won't (necessarily) be the same as yours or anyone else's.
  - trimbo 14y ago
    Ah, I didn't know that. Thanks!
drone 14y ago
"Degraded performance," is a fairly off-handed way of putting what they're experiencing. First RDS connectivity went down the tube, and then EBS followed, finally the EC2 console is failing to operate properly (for me, in US-EAST region at least). Of course, as soon as I read the report that the issue was confined to one AZ, I looked to move my server over to another AZ. Oh yes, two were full and refused new instances, and then surprisingly, new requests for the other AZs never were received or operated on - and now the console is failing. It's a bit more than just "slow EBS" if that's what you were thinking. - edit: said ec2 twice in the second sentence, corrected to say ebs.

ojiikun 14y ago

Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.

milkshakes 14y ago
too bad their own dashboard and management interface don't fail over... http://db.tt/BcuoSnPu http://db.tt/BcuoSnPu
driverdan 14y ago
Amazon is downplaying the problem. It is affecting many large sites.
- res0nat0r 14y ago
  Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.
  - mey 14y ago
    http://downrightnow.com/netflix http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.
    - res0nat0r 14y ago
      The last Netflix post mortem mentioned they had a bug in their configuration where they kept sending traffic to already down ELB instances, which was the cause of the last outage for them if I remember correctly.
      - Uchikoma 14y ago
        AirBnB is also down.

pcorsaro 14y ago

My site is currently down because of this.

lgleason 14y ago

Now is a good time to stroke your dev pair :) http://news.ycombinator.com/item?id=4680178 http://news.ycombinator.com/item?id=4680178

thehigherlife 14y ago

I jokingly said the internet was broken this morning as I've been having weird connectivity issues all morning. Funny that I was sort of right.

halis 14y ago

Thanks I have a friend whose site is down from this and this helps shed some light on it.

jaysonelliot 14y ago

I suppose this explains why Airbnb is down now, as well: http://aws.amazon.com/solutions/case-studies/airbnb/ http://aws.amazon.com/solutions/case-studies/airbnb/ I was just in the middle of booking a stay in a Palo Alto startup embassy for this week, too!

taigeair 14y ago
seems they should have a back up
mikeevans 14y ago
They've been down for several hours actually. I tried to get to their site this morning to check out the jedburg talk, and got the "We're on the case" message.

losvedir 14y ago

Sigh, my heroku site is down.

Revex 14y ago

I just recently signed up for a EC2 (3 weeks ago) and my small site is also down. :-(

HannibalLecter 14y ago

Reddit had a big story that criticized Amazon's role in deleting a woman's kindle library and basically telling her to go fuck herself.

jaequery 14y ago

isn't it about time they supported a multi-regional failover system?

gailees 14y ago

WAT!?

bas 14y ago

Some of our sites are affected. Good times!

playhard 14y ago

As of now , I don't have a problem with my site. Elastic beanstalk with ec2 small instances.

jmvoodoo 14y ago

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting a single zone is bs.

Banduin 14y ago
Our multi-zoned RDS instance was able to fail over to another zone with minimal downtime. It took about 2 minutes.
- jmvoodoo 14y ago
  Lucky you? :)
modsearch 14y ago
Our multi-AZ RDS instance did not failover correctly this time although it has in the past..
- rynop 14y ago
  same here, no fail-over
  - jmccree 14y ago
    I had one multi-az failover correctly, however the security group was refusing connections to the web servers ec2 security group. I had to manually add in the private ips of the ec2 instances. It appears the API issue is affecting security group to ip lookups.
abailin 14y ago
Same here, our servers are still up, but our RDS instance is down.
14y ago

benwerd 14y ago

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

petejansson 14y ago
The reference architectures and white papers here are helpful: http://aws.amazon.com/architecture/ http://aws.amazon.com/architecture/
caseysoftware 14y ago
We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastructure-in-the-cloud-evan-cooke-web-20-expo-nyc-2011 http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasnt-affected-by-todays-aws-issues/ http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.
- caseysoftware 14y ago
  And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
  - wyck 14y ago
    Yet netflix is currently down, so maybe there is a problem with the chimp army?
    - bermanoid 14y ago
      The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered. There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not robust against "single AZ outages", given that they're involved in pretty much every one of these incidents and make them affect people on the entire cloud...

diego 14y ago

The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.

merlincorey 14y ago
I believe this is because North Virginia is also their historical first facility.
- ceejayoz 14y ago
  And largest, and busiest.
acangiano 14y ago
Realistically, it's at least in part because everyone defaults to the East region. So it's the most crowded and demanding on the system.
- jey 14y ago
  Yeah, it's got to be much larger than the other regions, so it makes sense that we see more errors. Since error_rate = machines * error_rate_per_machine.
  - nasmorn 14y ago
    The whole region is down, you just calculated the chance of at least one machine having an error.
    - jey 14y ago
      No, I calculated the error rate for the region. If us-east-1 has 5 times the machines (or availability zones, or routers, or EBS backplanes, or other thing-that-can-fail) as us-west-1, we would expect to see us-east-1 have each type of error occur about 5 times as often as us-west-1.

Codhisattva 14y ago

Worst infrastructure ever.

LaSombra 14y ago

Coursera is also down.

ceejayoz 14y ago

My dashboard says "Degraded EBS performance in a single Availability Zone". It then lists each of the five zones as "Availability zone is operating normally." http://cl.ly/image/202F3B0I371g http://cl.ly/image/202F3B0I371g

dirtae 14y ago
I was seeing that too, but it looks like Amazon has now updated the availability zone status. When I run ec2-describe-availability-zones from the command line, it's telling me that us-east-1b is impaired. (Availability zone mapping is different for each account, so my 1b may be some other availability zone for you.)
ulope 14y ago
CloudApp seems to have trouble too

gtaylor 14y ago

Does anyone know if this really is just one AZ? Seems like an awful lot of larger sites are down. I'd expect at least some of them to be multi-AZ by now.

rynop 14y ago
I'm multi-az and am being impacted. I also use multi-az RDS adn its being impacted. So calling BS on the 1 az impact.
- ejdyksen 14y ago
  One of our Multi-AZ RDS instances failed over successfully. It was down for about 5-10 minutes.
colinhowe 14y ago
Looks like multi-AZ to us too
dsl 14y ago
Multiple AZs here.

lr 14y ago

pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."

Terretta 14y ago
That's editorializing. For many sites, a single server in a single zone (e.g., a non redundant server, an instance, a slice, a VM, whatever) is the right decision for ROI. For many sites, the money spent on redundancy could be better spent on, say, Google Adwords, until they're big enough that a couple hours downtime has irreplaceable costs higher than the added costs of redundancy (dev, hosting, admin) for a year.
- lr 14y ago
  Yes, it is editorializing. My point is, but I guess too subtle, that the current link text is very much an editorial comment, especially since the content at that link location has nothing to do with the sites mentioned in the link text.
lgsilver 14y ago
This issue is affecting both an EC2 Zone and Amazon's RDS servers, which are technically multi-zone. There are a ton of well-architected apps and sites that have been affected. Unconstructive...
bdcravens 14y ago
Tons has been written about Netflix's architecture, and they're down as well. I believe Amazon isn't being totally transparent.
- grumps 14y ago
  Personally, starting to believe the Amazon is quickly becoming public enemy #2.
  - 14y ago

Ramonaxvh 14y ago

This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.

gfodor 14y ago
If you go through the pains of architecting your system to span multiple AZs, or you avoid using EBS, then you probably dodge most of the EC2 outages. (Remains to be seen if that is the case here.) That said, I don't think most people think using the cloud means that downtime is a thing of the past. I think the more attractive proposition is when hardware breaks, or meteors hit the datacenter, etc, it is their problem, not yours. You still have to deal with software-level operations, but hardware-level operations is effectively outsourced. The question is if you think you can do a better job than Amazon -- some companies think they can, most startups know they can't.
- w_t_payne 14y ago
  Yeah. Even with this, they still do better than I would. My record: misconfigured air-conditioning unit alarm leading to servers being baked at high temperature over a weekend, leading to much wailing and gnashing of teeth. I now know to be really careful to set up air conditioning units properly, but what other lessons am I still waiting to learn? The main lesson that I took from this is that I should stick to what I am good at: cutting code & chewing data. :-)
  - gfodor 14y ago
    Yeah this is another important point. Part of the cost of AWS is also a bit of an insurance policy against you physically breaking your servers :)
  - gfodor 14y ago
    Yeah this is another important point. Part of the cost of AWS is also a bit of an insurance policy against you physically breaking your servers :)

Ramonaxvh 14y ago

lgsilver 14y ago

Two of our four EC2 servers are down. Both have RDS connections and EBS storage. Looks like the RDS connection has something to do with it...

spartango 14y ago
RDS is built on EBS, so if EBS has problems you can bet that RDS will feel the effects.

interro 14y ago

just checked my site and it is up now!

fsckin 14y ago

My instances just came back up.

tyw 14y ago

I reacted quickly enough to get a new instance spun up and added it to my site's load balancer, but the load balancer is failing to recognize the new instance and send traffic to it... yay. Console is totally unresponsive on the load balancer's "instances" tab. If I point my browser at the ec2 public DNS of the new instance it seems to be running just fine. So much for the load balancer being helpful.

gailees 14y ago

At least HN isn't down! :D

taigeair 14y ago

back up :)

SquareWheel 14y ago
As far as I can see Reddit is still down.

dave1619 14y ago

Our app is down because it's hosted on Heroku and it's frustrating because it seems like N Virginia is the least reliable Amazon datacenter. Every year it seems to go down a couple times for at least a couple hours. Heroku should offer a choice between N Virginia and Oregon hosting (I think they're almost comparable in price nowadays). That way people who want more uptime/reliability can choose Oregon. Sure it will be further from Europe (but then it will be closer to Asia) and people can make that choice on their own. But basing an entire hosting service on N Virginia doesn't make sense anymore, considering the history of major downtime in that region.

nlh 14y ago
Or better yet, Heroku should offer an add-on "instant failover" service that, for a premium of course, offers a package for multi-site (or, knowing they're 100% AWS, multi-datacenter) deployment with all of the best practices, etc. Seems like a logical next step for them (or a competitor) given the recent spate of outages.

gorkemcetin 14y ago

Countly mobile analytics platform servers are also affected from this issue (http://count.ly http://count.ly). Waiting and waiting. :) @caseysoftware thanks for the links, we now have something to read and implement this week.

Naushad 14y ago

The Console itself is down, <html><body>Http/1.1 Service Unavailable</body> </html> Its high time AWS does something now.

jeaguilar 14y ago

This is affecting more than a single Availability Zone, but probably for reasons that have been seen before. One reason might be that an EBS failure in one AZ triggers a spike in EBS activity in other AZs which overwhelms EBS. (I believe this is what happened in April 2011). Does anybody have any experience with migrating to Oregon or N. California in terms of speed and latency?

btilly 14y ago

As https://twitter.com/DEVOPS_BORAT https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)

engtech 14y ago
so true: "In devops is turtle all way down but at bottom is perl script. " https://twitter.com/DEVOPS_BORAT/status/248770195580125185 https://twitter.com/DEVOPS_BORAT/status/248770195580125185
eloisant 14y ago
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
- dennisgorelik 14y ago
  SoftLayer that we use seems to be much more reliable than Amazon. At least more reliable than that particular Amazon's datacenter in Virginia.
  - x3sphere 14y ago
    I agree, been with SL for 3 years, never had an outage apart from a drive failure one time but it was fixed within an hour.
- apendleton 14y ago
  But this is part of the problem: we have multiple web properties, and the fact that AWS issues can affect all of them at once is a huge downside. Certainly, if we ran on metal, we would have hardware fail, but failures would be likely to be better-isolated than at Amazon.

kellyhclay 14y ago

Can't update my list fast enough, but other major services experiencing problems are Netflix and Pinterest. Lots of other (smaller) sites are starting to fail too. http://www.forbes.com/sites/kellyclay/2012/10/22/amazon-aws-goes-down-again-takes-reddit-with-it/ http://www.forbes.com/sites/kellyclay/2012/10/22/amazon-aws-...

run4yourlives 14y ago

Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)

IheartApplesDix 14y ago
Because reddit and ycombinator are the center of the tech sociosphere?
- farinasa 14y ago
  For the less specialized groups, kind of, yeah.
UnoriginalGuy 14y ago
Well if that was their goal then it has been a success as they have taken out a lot of the forums and discussion sites which were critical of them. Except HN.
macrael 14y ago
This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts because of a conspiracy, let's belittle their efforts because of bad systems design.
- Angostura 14y ago
  It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
  - tuple 14y ago
    actually, there's probably some few idiots who will believe it.

espeed 14y ago

When is Google Compute Engine supposed to be open to everyone?

quotemstr 14y ago
There's Azure.

blaze33 14y ago

Hosted on heroku with a hosted heroku postgres dev instance, I observed a drop in the db response time : - from a consistent average of 10ms over the last week - to a new consistent average of ~2.5ms beetwen 17:31 and 17:35 UTC. AWS started to report the current issue at 17:38 UTC. My app then experienced some intermittent issues (reported by newrelic pinging it every 30s). Don't know if it's related, could it be some sort of hw upgrade that went wrong ? I did a push affecting my most used queries but that was one and a half hour sooner, at 15:56 UTC, so probably unrelated.

jasonkester 14y ago

For added fun, their EC2 console is down. I got this for a while: <html><body>Http/1.1 Service Unavailable</body> </html> ... then an empty console saying "loading" for the last 20 minutes. Then recently it upgraded to saying "Request limit exceeded." in place of the loading message (because hey, I'd refreshed the page four times over the course of 20 minutes). On the upside, their status page shows all green lights.

hornbaker 14y ago
They've acknowledged that at http://status.aws.amazon.com/ http://status.aws.amazon.com/ for awhile now with a tiny "i" status icon (I can't load my instances pane): 12:07 PM PDT We are experiencing elevated error rates with the EC2 Management Console.
- mrb 14y ago
  Amazon misrepresents reality on this status page! They have standardized icons to represent various levels of issues (orange = perf issue, red = disruption). But they don't even use them. Instead they add [i] to the green icons to indicate perf issues (Amazon Elastic Compute Cloud - N. Virginia) and disruptions of service (Amazon Relational Database Service - N. Virginia). Maybe this status page is controlled by marketing bozos who want to pretend the situation is not so bad.
adrianbg 14y ago
Does Amazon explain anywhere what regions, availability zones or whatever have single points of failure for each of their services? I guess that's supposed to be what an "availability zone" is but somehow that doesn't quite seem to capture it. It's pretty hard to build reliable apps without knowing where the single points of failure are in the underlying infrastructure.

bdcravens 14y ago

Noticed that Rackspace was down for a bit - I presumed they're getting hammered on their WWW as people are considering an infrastructure switch?

zenwheels 14y ago

Perfectly timed with our product launch, now our site is down too - www.zenwheels.com

Yrlec 14y ago

Does this just affect normal EBS or EBS with provisioned IOPS as well?

timmythebest 14y ago

a ton of other stuff is down too... www.reddit.com, netflix, www.freewebsite.com and a bunch more

mtgx 14y ago

Time to switch to Google's Compute Engine.

rsync 14y ago

This reminds me of an ad campaign we sometimes run on reddit: http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws_and_your_offsite_backups/ http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws... ... which you'll be able to click as soon as reddit is back up :)

incision 14y ago

Unfortunate. I have to deal with a number of folks who will be overjoyed to read this news when their tech cartel vendor of choice forwards it this evening. There's a huge contingent of currently endangered infrastructure folks (and vendors who feed off them) out there who throw a party every time AWS has a visible outage.

druiid 14y ago
I'd question if they are truly wrong to send out news like this? Basically you have to weigh the realities of the 'cloud' with the perception. The perception that many people has is that somehow the 'cloud' is a magical up-time device that will save you money in droves. The reality is that for many companies with mid-level traffic and aren't a start-up with a billion users, the 'old' style tech might very well be your best option.
rdl 14y ago
AWS sucking at availability (and especially specific parts like EBS, and then services built on top of EBS and on top of AWS) doesn't mean the correct option is to mine your own sand and dig up your own coal to run servers in your own datacenters in the basement. Even if you're totally sold on the cloud, you can still have a requirement that things be transparent all the way down. AWS is one of the least transparent hosting options around. If you're a customer of a regular colo, or even a managed hosting provider themselves based at a colo, it's pretty easy to dig into how the infrastructure is set up, identify areas where you need your own redundancy, etc. Essentially impossible within AWS -- there is no reason intelligible to me that ELB in multiple AZs should depend on EBS in a single AZ, but that's how they have it set up.
tisme 14y ago
> who throw a party every time AWS has a visible outage. At the going rate they'll be AA members before long.

fredoliveira 14y ago

One of the most frustrating issues here is that we have to deal with Amazon's status page for information. It's a complex page, divided by continent instead of region, which means at least 5 or 6 clicks to figure out progress. They should learn from these issues about how people want to be informed - to date, they haven't. Also, they have a twitter account, which would be the perfect fit to keep everyone up-to-date with what's going on; to at least show a human side to these issues. Alas, they're not updating that either. I've been working with AWS since early 2006 when they first launched - I was lucky to be granted a VIP invite to try out EC2 before everyone else, and ended up launching the first public app on EC2. This might be the first time when frustration has overcome my love for these guys.

Latricerzj 14y ago

Single point of failure?

johansch 14y ago

The emperor has no clothes!

gavinlynch 14y ago

Disappointing, but I guess not totally unexpected. Our prod site was down for a bit with many others.

patrickgzill 14y ago

You have a virtualized platform, on top of which are many pieces like load-balancing, EBS, RDS, the control-plane itself, etc. You have burstable network connections which by their nature, will have hard limits (you can't burst above 10Gbps on a 10Gbps pipe, for example; even assuming the host machine is connected to a 10Gbps port). Burstable (meaning quite frankly, over-provisioned) disk and CPU resources. And if any piece fails, you may well have downtime... It is always surprising to me, that people feel that layering complexity upon complexity, will result in greater reliability.

gourneau 14y ago

These semi-regular outages are why I am going to use the Google Cloud Platform for my next project.

1SaltwaterC 14y ago

ELB is also fucked. Seen that nobody mentions it. Some of our load balancers are completely unreachable. Just one multi-AZ RDS instance claimed the automatic failover. However, the 200+ alerts over the automatic failover due to internal DNS changes to point to the new master shows that things aren't as easily described by the RDS DB Events log. Some instances reported high disk I/O (EBS backed) via the New Relic agent (the console still has some issues). So far, this is what I see from my side.

orph 14y ago

Adding insult to injury: when trying to view instances in the console I now only see "Rate limit exceeded".

rdl 14y ago

Why the fuck do the two most critical services (ELB and Console) have depends on their historically most unreliable pile of shit (EBS)?

taligent 14y ago
Seriously THIS has to be addressed. I can tolerate EC2/EBS going down but why on earth is ELB/Console always going down at the same time ?

sigkill 14y ago

Isn't it worrying that there are so many services/sites that are so dependent on Amazon EC?

zenwheels 14y ago

Woot, Back Online www.zenwheels.com

cupcake_death 14y ago

Maybe I'm being daft. Why does this sh*t never seem to affect www.amazon.com

mvelie 14y ago
Amazon doesn't run their site off the same set of EC2 servers and systems that everyone else does. They just used their experience to build it.
- cupcake_death 14y ago
  That would be the ultimate testament, maybe the only stunt they could pull to PR themselves out of this mess.. if www.amazon.com ran on the same set of EC2 servers and systems as us muggles
- yap 14y ago
  Amazon runs all of the Amazon.com web servers on EC2 hosts (and has since 2010: https://www.youtube.com/watch?feature=player_detailpage&v=dxk8b9rSKOo#t=393s https://www.youtube.com/watch?feature=player_detailpage&...). They've just made sure that they have enough hosts in each of the other AZs to withstand an outage.

lucb1e 14y ago

Dropbox runs on S3, they are unaffected? And Heroku's website seems okay. Only reddit is down. That's because of this?

rynop 14y ago

EIP remapping is all messed up as well. api errors. console for it does not work either.

blhack 14y ago

I'm sorry, am I in a time machine? Didn't this exact thing happen last year, and didn't these exact people explain exactly how they were going to make sure it never happens again? What the hell is happening here?

valhallarecords 14y ago

This is why I trust Google's data centers http://www.google.com/about/datacenters/gallery/#/ http://www.google.com/about/datacenters/gallery/#/

taligent 14y ago
HN really needs to hold a "dumbest comments of the year" award. This has to be a strong contender.
- JackWebbHeller 14y ago
  HN really needs to hold a "snarkiest response of the year" award.
  - pardner 14y ago
    Snap!

j45 14y ago

How does Twilio manage to stay up when AWS goes down, and how much of that can the average developer reasonably do?

driverdan 14y ago
See Keith's (of Twilio) response here: http://news.ycombinator.com/item?id=4684766 http://news.ycombinator.com/item?id=4684766
- j45 14y ago
  Nice, thanks!

hornbaker 14y ago

Update from http://status.aws.amazon.com/ http://status.aws.amazon.com/: 2:20 PM PDT We've now restored performance for about half of the volumes that experienced issues. Instances that were attached to these recovered volumes are recovering. We're continuing to work on restoring availability and performance for the volumes that are still degraded. We also want to add some detail around what customers using ELB may have experienced. Customers with ELBs running in only the affected Availability Zone may be experiencing elevated error rates and customers may not be able to create new ELBs in the affected Availability Zone. For customers with multi-AZ ELBs, traffic was shifted away from the affected Availability Zone early in this event and they should not be seeing impact at this time.

nkohari 14y ago
This is infuriating. Our instances aren't ELB-backed, so we have an entire AZ sitting idle while the rest of our instances are overwhelmed with traffic. Why did they make this decision for us? I'm normally an Amazon apologist when outages happen, but this is absolutely ridiculous.

Lashawndazqd 14y ago

I wonder if this is going to be an attack or hardware/software failure..

johnmurray_io 14y ago

Here comes another round of Hacker News posts about how you should never host your app in just one availability zone...

pg 14y ago

Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.

Splines 14y ago
I'm not familiar with HN's technical stack (other than arc), how has it scaled as the community grew over the years?
- rdl 14y ago
  I think it's a case of "well optimized code which does as little as possible can handle an awful lot of users on a single beefy box".
  - adrianbg 14y ago
    Until it can't. Then five years later you get Spanner.
    - rdl 14y ago
      You can buy some seriously big boxes, and easily split off a lot of services onto multiple boxes. The big problem with the "single big box" strategy is being able to do upgrades -- I see hn go down frequently for 5-10 min at a time in the middle of the night, which I assume is upgrades/reboots. The happy medium is probably splitting database (master/slave at least) and cdn (if needed) and some other services (AAA? logging?) out, and then having 2+ front end servers with load balancing for availability.
      - ukd1 14y ago
        hn doesn't have a database capable of master/slave as such...so I think this will be harder if it ever becomes popular enough. I don't think it gets enough traffic it's ever likely to exceed what you can fit in a single box, from what I know.

Cordiapxq 14y ago

Workplace productivity is going to skyrocket today

potomak 14y ago

This is why you should think about a multi region architecture to prevent these types of service interruptions.

mattbillenstein 14y ago

I thought at least Reddit had learned the "don't use EBS" rule from past outages - I was bitten by the April/2011 outage and switched everything over to instance storage shortly thereafter. For most applications I think architecting EBS out should be straightforward - instance storage doesn't put a huge single-point-of-failure in the middle of your app if you're doing replication, failover, and backups properly. And EBS seems to be the biggest component of the recent AWS failures upon which they've built a lot of their other systems.

justplay 14y ago

i read in my book that amazon is the best cloud service provider and seeing this thing can't really prove it. I am still confused which cloud service i use ? Amazon is only one which Indians are currently rely on ,since latency is bit smaller for amazon servers as compared to other providers.

jbarham 14y ago

FWIW here's Google App Engine status page: http://code.google.com/status/appengine http://code.google.com/status/appengine

robbiet480 14y ago

I made this to express my outrage and hope to help others easily express theirs too! http://robbiet480.github.com/StopLyingToUsAmazon/ http://robbiet480.github.com/StopLyingToUsAmazon/

d0m 14y ago

Ironically, on the first hacking health event, our server was running heroku.. and heroku went down. 3 months later, we have the second hacking health in Toronto and all AWS is going down.

foxylad 14y ago

Not wanting to shill for Google, but we've been very happy with Appengine's reliability - particularly since moving to the HR datastore. Have other Appengine users had any significant downtime apart from MS datastore issues?

valhallarecords 14y ago
What site are you running on App Engine?

DavidThi808 14y ago

We came up with a way to get better reliability than AWS, and for less money. We list the 3 providers we combined to get it at http://go.windward.net/TLWAugust8_BasicWebinarFormLP.html http://go.windward.net/TLWAugust8_BasicWebinarFormLP.html Thanks Dave

cryptoz 14y ago

nell 14y ago

The tech industry is manufacturing its version of "Too big to fail" entity.

traderd65 14y ago

Building infrastructure on top of Amazon with global replication across multiple availability zones can sidestep such failures and guarantee uninterrupted operations to customers - www.pubnub.com is unaffected by AWS being down.

stinky613 14y ago

Somewhere Steve Wozniak is saying "I told you so"

briandear 14y ago

I fixed EC2 using this one weird trick..

GigabyteCoin 14y ago

It's back up now.

jasonlingx 14y ago

I am frankly surprised so many still use EC2 considering how frequently it breaks. It's not cheap, so the only reasons to use it would be reliability or scale right? Why not just get lots of boxes at Hetzner and OVH (40 EUR a month for 32GB of RAM and 4x2x3.4GHz cores) and scale up / redundansize that way?

deleted 14y ago

[deleted]

davidjnelson 14y ago

I have a few questions about HA AWS deployments. Anyone know the answers? It seems obvious that multi region for HA is ideal, but the reference architecture for web apps on the aws architecture page only uses multi AZ for HA. How do you bridge that disconnect? Route53 with super low ttl? Wouldn’t route53 then be a single POF? Also, is dynamodb multi region? If you have to use MySQL, how do you replicate cross region in a time and cost efficient manner? Isn’t S3 also only multi az, is there even a way to make s3 cross region? Or do you just “hope” dynamodb, route53, and s3 don’t fail? Thanks, Best, David