10 ms·
Large-scale Amazon EC2 Outage
- randomanonymous 15y agoWhy don't people just be redundant and use multiple locations, or have a backup server at your business that can at least somewhat take over? It seems silly all these people rely solely on Ec2 and not have their own hardware anymore.
- acdha 15y agoThey could get the same thing by simply reading Amazon's recommendations and having EC2 servers in multiple regions as well. Most people either don't need reliability or failed to plan for it - the only difference with EC2 is that it's easier to add redundancy when you don't have to deal with physical hardware, network connections, etc.
- reustle 15y ago100% green for me
- captaincrunch 15y agoscroll down.
- deleted 15y ago[deleted]
- mgkimsal 15y agois this related to the one from the other day? DUH - doesn't seem to be - that was west coast, IIRC, and the only issue I see now is in Virginia.
- elemenohpee 15y agoIs this why Reddit is down?
- ntkachov 15y agoyup.
- bgentry 15y ago3 out of 4 us-east-1 AZs are unavailable from the outside. The AZ that is working is the one that has chronic availability issues, so there is no way to recover service in that zone.
- jbellis 15y agoThere's an AZ with "chronic" availability issues? Where do I find out more about this?
- bgentry 15y agoI should have been more specific, I was in a rush =) The original us-east-1b has chronic capacity issues, meaning that it is always at or near its capacity. AWS refuses to sell instance reservations for this AZ, and it's often difficult/impossible to launch new instances in this zone. Keep in mind that AZs are remapped for all accounts newer than a certain point in time (say, 1 yr ago). Your 1b may not be my 1b.
- Hisoka 15y agoImagine launching your startup on the day Amazon AWS goes down? On another note.. why don't these power outages ever affect Amazon.com from going down? Come'n, eat your own dog food!!!
- cbs 15y agoWhat? You mean without a backup plan to substitute in for the platform with known reliably issues? Yeah I guess that would suck. Edit re your edit on dogfooding: supposedly they use the same tech, just not the same servers, so their AZs are probably separate form ec2s. Also, they're in all of their geographical locations, not just us east.
- justinsb 15y agoDo you have a source for Amazon.com running on AWS or AWS technology? As far as I know, they've never gone on the record with that.
- cbs 15y agoI'm on my celly so I can't, but if I remember after I land ill try and dig up where I read about it.
- deleted 15y ago[deleted]
- amock 15y agoAccording to http://en.wikipedia.org/wiki/Amazon_Cloud http://en.wikipedia.org/wiki/Amazon_Cloud the Amazon.com retail website has been running on AWS since November 2010.
- justinsb 15y agoWow - looks like you're right! I dug into the presentation (link below, important slide is #33) and Amazon are now claiming that all their web servers run on AWS. It's only their webservers (not their DBs, load-balancers or "services" which I presume means anything stateful). Of course, the co-location option for 'difficult' services is only available to Amazon, and before the switch-over in November they presumably weren't using AWS nearly so heavily, but this is at least a step in the right direction. http://www.slideshare.net/AmazonWebServices/2011-aws-tour-australia-closing-keynote-how-amazoncom-migrated-to-aws-by-jon-jenkins http://www.slideshare.net/AmazonWebServices/2011-aws-tour-au...
- hysterix 15y agoReddit gives: An error occurred while processing your request. Reference #97.374a7b5c.1312858006.1a17105 Must be this.
- azth 15y agoThat's why I came here after I found out reddit was down :P
- zrail 15y agoHuh. My piddly little website hosted on a micro instance is down. This kind of sucks, not that I get a lot of hits, but I have a few things hosted there that local stuff depends on.
- doubaokun 15y agoMy servers are down in east us.
- skennedy 15y agoTwilio is down. Bummer
- taf2 15y agodamn, http://status.twilio.com/ http://status.twilio.com/ - calls are still going through, just a few extra rings before my service receives the http request from twilio... gotta love twilio they rock - looks like no call recordings...
- CamperBob 15y agoDropbox is up but stored content seems to be inaccessible. Yay for single points of failure!
- mtogo 15y agoUsing dropbox is a bad idea overall in my opinion. They lied to their customers about what they can and cannot view in your account. They lied to you. They intentionally told you that they could not view your files when in fact they can. If that's not enough, they had a critical security vulnerability (log in with no password) for four hours, proving that their systems fail open. Finally, if all this wasn't bad enough yet, they do not encrypt your files when they store them[1]. [1] Technically they do encrypt the files, but the keys are right next to them on the same infrastructure. Doesn't do any more good than not encrypting them.
- blackhole 15y agoThis really isn't a problem if you aren't storing anything sensitive, especially if you aren't even paying for it.
- jules 15y agoFor the bits that need to be protected there is Tarsnap (http://www.tarsnap.com/ http://www.tarsnap.com/). The client is open source, so you can check for yourself that things are encrypted before going on the wire.
- cbs 15y agoFrom a security conscious perspective it is a bad idea, but too many people undervalue noncritical data and don't secure it like they should (That's part of why infosec is such big buisness). A prevelant lack of security sense is basically the only reason dropbox is still in business. But from a HN perspective they've received YC cash so your post is currently voted <= 0.
- Dylan16807 15y agoI'm not convinced they were intentionally misleading. I was never 'lied' to; I understood what their system delivered.
- rmrm 15y agonetflix down on my PC and their app is down on my TV.
- tricolon 15y agoNetflix also seems to be affected.
- blantonl 15y agoMy business is on 15+ AWS instances and we are completely down and all hosts are unreachable.
- deleted 15y ago[deleted]
- blantonl 15y agoas of 9:59 PM CST it appears we are back up. Additionally, all our instances are running normal. It appears to be a network connectivity issue for AWS.
- dennisgorelik 15y agoNetflix is down.
- presidentender 15y agoOur minecraft server on linode is down.
- dbingham 15y agoLinode relys on amazon? My linode's still up...
- presidentender 15y agoI'm a fool, it's mojang's authentication.
- jcsalterego 15y agohttp://status.aws.amazon.com/rss/EC2.rss http://status.aws.amazon.com/rss/EC2.rss http://reports.panopta.com/cloudharmony-borderless/server/57481 http://reports.panopta.com/cloudharmony-borderless/server/57... various ec2 clients affected: - dotcloud - dropbox - engine yard - foursquare - heroku - hootsuite - instagram - kicksend - netflix - pagerduty - reddit - twilio
- rednaught 15y agoReddit is intermittent too.
- santi 15y agopagerduty down?! I don't know if I should be sarcastic here and laugh, or just worry...
- gshakhn 15y agoIs it just their user facing website that's hosted on EC2? Or do the pages originate from EC2 as well?
- shazow 15y agoTheir website might have gone down but I'm told that their underlying service is redundant across multiple availability zones. Perhaps they should do the same for the website just to instill extra confidence, though.
- AngryParsley 15y agoI didn't get any SMSes or calls from PagerDuty even though I see 4 incidents triggered in the PD web UI. Although that could be due to Twilio going down.
- alexsolo 15y agoWe (PagerDuty) were down for a bit. We will move off of AWS asap once the dust settles.
- enoptix 15y agoDotcloud is down
- deleted 15y ago[deleted]
- mtogo 15y agoNetflix is down as well.
- Pewpewarrows 15y agoTwo large, lengthy outages within a few months of each other? I hate to piggy-back on Amazon's follies, but it's times like these that make me love my Linode boxes. Total downtime over the last 5 years? 5 hours. I really couldn't ask for a better hosting company. Edit: The 5 hours are what I've noticed from either personal servers or those of friends/acquaintances. YMMV.
- Derferman 15y agoI must have picked a bad time to become a Linode customer. My node in Fremont has been up for two days and has already seen two network outages.
- techiferous 15y agoJust a fluke. I've been a Linode customer since 2008 and have managed multiple Linodes (on the East coast) and as far as I can tell, Linode has given me more than four nines.
- rednaught 15y agoOverall they have great service. Only their Fremont location has been a bit touchy this year. I've been with them since the days they used UML before Xen and if it's any consolation, I have experienced at least one downtime in each DC(but this was over the course of many years now). Unless you plan for distributed systems, expect some failures from time to time no matter who is the provider.
- jwilliams 15y agoFrom what I've encountered, and what I've read, Fremont has been the most patchy Linode DC - they've had two outages there recently. One just a day or two ago, and another in early May. Apparently there was a significant outage late last year too. p.s. The last two weren't network outages - they were total power failures. Your 'node would have been hard-booted as a result.
- jccodez 15y agoI noticed the same outage. I signed up a week ago and was moving some stuff to the freemont site. Maybe I should have picked Dallas.
- necubi 15y agoAppears to be back up, at least for us.
- ksdsh 15y agoYes, my site is up now too.
- bgentry 15y agoBack up now.
- shriphani 15y agoLooks like it is up now. (7:58 pacific). I can access netflix.
- dennisgorelik 15y ago11:17 pm EST - netflix.com does not work (though shows some minor signs of life).
- teeray 15y agoThis will likely kick Heroku into high gear with jumping off EC2.
- ahhrrr 15y agoMy servers (on Engine Yard) were down for a few minutes but appear to be back up again.
- chetan51 15y agoIt's back up now.
- agotterer 15y agoAll my boxes just came back up
- bane 15y agoI wonder how many 9's Amazon is down to? It's really a shame because it's such a good service in theory. But it really is time for Amazon to start thinking about some kind of automatic failover system.
- dbingham 15y agoLooks like we're back.
- andymoe 15y agoI think amazon can fix this problem simply by raising us-east prices to be on par with us-west. At least then there would not be quite the incentive to make the foolish choice of running all your instaces out of one region. (I suppose they could lower prices in us-west to match as well) But in any case, I expect the PAAS people like heroku to at least start to step up their game. My apps on Google App Engine are up and I can run clojure and ruby there too.
- xal 15y agoLooks like Pagerduty is hosted on ec2? Really?
- teeray 15y agoI would like to run my app on whatever AWS's status board is running on :-P
- kenneth_chau 15y agoWhy are these big name companies still relying on exactly ONE web host? These companies are centralizing failures to just one cloud host. They definitely need to abstract cloud deployment with something like libcloud and jcloud APIs and make use of multiple cloud deployments on a dime. As for storage, they should have replications set up on other cloud hosts as well.
- atambo 15y agoAnd for rubyists: https://github.com/geemus/fog https://github.com/geemus/fog
- dstein 15y agoI thought it was silly how everyone was going apeshit the last time this happened. Even here on HN people were falling over themselves proclaiming the end of the cloud. But you know what, I kind of like seeing outages like this because it means Amazon is (hopefully) going to reduce the risk of a bug like this ever happening again which in the end just makes the platform more reliable in the future.
- sjs 15y agoTo be fair the last one was a lot longer than a 1/2 hour and even though it was a small percentage there was permanent data loss. 0.06% of 100 PB is 60 TB which is still a lot of data. I have no idea how much data they store in one DC, maybe somebody could hazard a real guess.
- cperciva 15y agothe last one was a lot longer than a 1/2 hour and even though it was a small percentage there was permanent data loss There was permanent data loss on EBS volumes, which are explicitly stated to be vulnerable to data loss.
- rmc 15y agothe last one was a lot longer than a 1/2 hour About 25% of EU west instances have been down for about 36 hours now...
- sjs 15y agoOh! I did not know that.
- grandalf 15y agoMy site (hosted on Heroku and RDS) was inaccessible, as was heroku.com ... but the AWS status website said everything was OK. Heroku status also said everything was OK. What is the point of those status dashboards if they are not actually monitoring the health of the cloud in real time? Google App Engine's status dashboard quickly returns to green the minute an outage goes away, hiding the overall unreliability from view.
- bradly 15y agoI've found http://search.twitter.com http://search.twitter.com far more reliable than any status dashboard for finding out if there is an issue or not.
- itsnotvalid 15y agoone should build a crowdsourced status checker to really know what happens.
- kalleboo 15y agohttp://downrightnow.com/ http://downrightnow.com/ monitors twitter, their own user reports, official status RSS feeds, etc for a probability calculation of if a site is down or not.
- NathanKP 15y agoIt looks like the outage was only 30 minutes long. At least that is Pingdom's measurement of the downtime on my EC2 East 1 servers.
- vilda 15y agoIt depends on what you mean by outage. Instances with EBS volumes attached have problems even 24 hours later.
- m0nastic 15y agoI'm moderately embarrassed to admit that I learned about this outage when both the Facebook games I play went down at the same time. Both back up now though.
- deleted 15y ago[deleted]
- bch 15y agoIs this pure network issue (ie: strictly connectivity), or were instances rebooted?
- petedoyle 15y agoSeems like network only. I have an instance that was unavailable via SSH. When things came back up I still had 81 days of uptime.
- Aloisius 15y agoI pay the extra money for US-west which seems to have a much better track record.
- rtrunck 15y agoI hear all these things about the need to run in multiple regions, etc. Why do I have to worry about this? Why isn't there an easy off the self solution? Honestly, isn't there an easier way to have redundancy?
- cloudwalking 15y agoData synchronization between data centers is tough. There's A LOT of data moving in those data centers all once--it isn't easy to mirror that across the country in real time.
- diolpah 15y agoThat's why the good lord invented eventually consistent data synchronization models. If your use case can tolerate it, use it.
- acdha 15y agoIt's a hard problem because there are many different things which can cause downtime: S3 is simple and redundant but also limited so they constrain the engineering problem to something manageable. If you need anything more complex than delivering static content, you have to spend the effort thinking about what's important to you and how you're comfortable spending and/or compromising to meet that reliability goal. The answers for this tend to be pretty specific to your business needs and tech stack.
- Strom 15y agoThere is an off the shelf solution - Google App Engine.
- NARKOZ 15y agortomayko: Everyone hates AWS when it fails in exactly the way they clearly state it will in the fine print. It's not free. You have to engineer for it.
- rmoriz 15y agohttp://i.imgur.com/ZIqmo.jpg http://i.imgur.com/ZIqmo.jpg
- ars 15y agoThis type of comment is not desired here. Take it to reddit.
- hamburgersushi 15y agoThis site is already, effectively, /r/technology combined with /r/politics. I like this comment.
- rmc 15y agoRemember there is large EC2 EU West Outage that isn't fully fixed. About 25% of EC2 instances have been down for about 36 hours now...
- adamt 15y agoIt gets better - I hope nobody was relying on their AWS DB snapshots for backups - I just had a note from Amazon to say that one or more of EU-West database snapshots had missing blocks (due to an EBS software errors) and had been removed.
- pdaddyo 15y agoYep just had the same email - I've lost all bar one EBS snapshot. Don't think I can trust them anymore, I wish I had more time to move our infrastructure elsewhere! They've essentially swiss-cheesed all our backups. Email copy follows.... Hello, We've discovered an error in the Amazon EBS software that cleans up unused snapshots. This has affected at least one of your snapshots in the EU-West Region. During a recent run of this EBS software in the EU-West Region, one or more blocks in a number of EBS snapshots were incorrectly deleted. The root cause was a software error that caused the snapshot references to a subset of blocks to be missed during the reference counting process. This process compares the blocks scheduled for deletion to the blocks referenced in customer snapshots. As a result of the software error, the EBS snapshot management system in the EU-West Region incorrectly thought some of the blocks were no longer being used and deleted them. We've addressed the error in the EBS snapshot system to prevent it from recurring. We have now disabled all of your snapshots that contain these missing blocks. You can determine which of your snapshots were affected via the AWS Management Console or the DescribeSnapshots API call. The status for any affected snapshots will be shown as "error." We have created copies of your affected snapshots where we've replaced the missing blocks with empty blocks. You can create a new volume from these snapshot copies and run a recovery tool on it (e.g. a file system recovery tool like fsck); in some cases this may restore normal volume operation. These snapshots can be identified via the snapshot Description field which you can see on the AWS Management Console or via the DescribeSnapshots API call. The Description field contains "Recovery Snapshot snap-xxxx" where snap-xxx is the id of the affected snapshot. Alternately, if you have any older or more recent snapshots that were unaffected, you will be able to create a volume from those snapshots without error. For additional questions, you may open a case in our Support Center: https://aws.amazon.com/support/createCase https://aws.amazon.com/support/createCase We apologize for any potential impact this might have on your applications. Sincerely, AWS Developer Support
- rjurney 15y agoUS-East is the Big Lots of EC2 zones.
- nirvana 15y agoI'm sure Amazon will fix this issue, but the question it brings to my mind is- why are so many people using ECC? Most startups need hosting, not an elastic compute cloud. ECC makes sense for someone who needs to spin up 1,000 workers, pull data out of S3, process it, store the results in S3 and then spin down the workers when it is done. But startups need hosts that are up 24/7. ECC doesn't give you any guarantee of uptime, and if it goes down the local (fast) disk is ephemeral. Yet, the EBS alternative which is backed up, is very slow. The basic VPS offerings, that Linode, Rackspace, and just about everyone else offers, isn't available at all from Amazon (near as I can see.) Yet this is what startups need- local disks, a small monthly fee, and up all the time. So, Amazon requires extra engineering--to account for nodes going down more often and to make ephemeral disks reliable or EBS performant. It also puts you on the path to lock-in, since so many of amazons services have their own unique APIs. It isn't exactly cheap, near as I can tell, when compared to, say, dedicated hosting in germany. Its advantages exist elsewhere- if you need to spin up a bunch of machines with an API you can do this at rackspace cloud. And... that's about the only advantage I'm able to think of. Personally, I'm architecting to have some overcapacity built in and to survive a spike, because I use that excess capacity in the off hours for heavy lifting. I wouldn't try to bring up extra nodes in the morning and shut them down in the evening anyway... and I doubt that many startups are really doing that. Possibly, I'm missing something. I tend to forget about features that aren't compelling to me, but are compelling to others. So maybe there's something that's important to these startups.
- petervandijck 15y agoS3 is a big part of it. If you're storing lots of data, a "local disk" just isn't enough. Now you're spending your time solving problems Amazon already solved. Occasional downtime just isn't that big of an issue for many startups, who are still looking for product-market fit.
- nirvana 15y agoI see what you're saying, and I failed to illuminate that I'm coming from the perspective of someone who is working with a cluster of riak nodes. Riak is a ring topology much inspired by the Amazon S3 design. I'd never trust anything to just one spinning rust device, sure. I was taking a distributed storage architecture (using whatever open source platform you prefer) for granted. But I recognize that S3 predates many of them.
- vilda 15y agoPlease refrain from judging reliability of cloud providers unless you have a representative sample. By representative sample I mean several hundreds of instances. That your instance runs 99.9% does not say anything, you are lucky. That your instance dies in two weeks does not day anything, you are unlucky.
- deleted 15y ago[deleted]
- flog 15y agoWe've been down for 3 days now.
- generators 15y agogoogle app engine is rock solid since high replication database migration. and even rdbms is coming. and go language is also supported if you need some serious calculations. If you are not using gae, you are stupid.