9 ms·
Amazon Web Services are down
- yuvadam 16y agoCurrent status: bad things are happening in the North Virginia datacenter. EC2, EBS and RDS are all down on US-east-1. Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.
- samratjp 16y agohttp://status.heroku.com/ http://status.heroku.com/ for those on Heroku
- mike-cardwell 16y agoIs it just me, or is their status page down?
- guynamedloren 16y agoIt's been on-and-off for the past few hours.. It's up right now (for me, anyway).
- amorphid 16y agoThis morning from 5a - 6a Pacific time I was able to access my Heroku app just fine.
- JonnieCache 16y agoI can access all my heroku apps that have their own DNS. Anything with a .heroku.com subdomain is down for me. Frustrating, knowing that the apps are still running but aren't routable.
- cooldeal 16y agoYay, cloud.
- JonnieCache 16y agoPlease do not make content-free posts such as this. It adds no value to the conversation and is only noise. If you actually wish to make a useful point about the practicality or otherwise of massively virtualised systems for webapp deployment, please do. It's going to take more than two words though.
- mtogo 16y agoPlease do not make content-free posts such as this. It adds no value to the conversation and is only noise. If you actually wish to bitch about a post that we all got the point of, you're going to need more than two paragraphs.
- cooldeal 16y agoI guess you haven't seen the Microsoft ads about the cloud? http://www.youtube.com/watch?v=Lel3swo4RMc http://www.youtube.com/watch?v=Lel3swo4RMc Anyway, my bad, I was just trying to make a joke to lighten up the mood. Sorry.
- JonnieCache 16y agoYou're right, I hadn't. Fair enough.
- recoiledsnake 16y agoI laughed, it's relevant and puts things in perspective if you had seen the ads. So it's not really content free even if it's just two words.
- bskari 16y agoI know you probably have your answer by now, but this site always helps me out when I have that question: http://www.downforeveryoneorjustme.com http://www.downforeveryoneorjustme.com
- guynamedloren 16y agoIndeed they are. Right before the issues began, I pushed a bad update to one of my Heroku apps, causing it to crash. A minute later I fixed the bug, re-pushed the git repo to Heroku... and nothing. I've been stuck with an error message on my website for hours. Unfortunate timing!
- potomak 16y agoAlso Cuorizini (http://cuorizini.heroku.com http://cuorizini.heroku.com) is down!
- wmoxam 16y agoNot all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.
- jedberg 16y agoSame with us. About 10% of our 700+ volumes are having problems right now. It's hard to tell for sure since there isn't any load.
- nostromo 16y agoWe're fine on EC2 -- but everything on RDS seems to be giving us big problems. We started a few backups before we new it was systemic, and all of them are stuck at 0%. We also tried spinning up new instances -- and they're all stuck in booting.
- agotterer 16y agoWe had about 45 min of downtime around 4am EST. Our RDS instances, EBS backed and normal instances all returned without problems. We are in Virginia us-east-1a and us-east-1b.
- obfuscate 16y agoI think which physical data center "us-east-1a" etc. corresponds to differs from user to user, to load-balance given that people will probably be more likely to use 1a than the other zones.
- deleted 16y ago[deleted]
- antonioe 16y agoFoursquare is up now. Quora not showing the 503 anymore, Reddit still down.
- antonioe 16y agoI stand corrected. 4sq down again.
- jedberg 16y agoWe rely heavily on EBS still, so this is hurting us more than most others. Hopefully they'll have us back up soon.
- bvi 16y agoYou guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?
- jedberg 16y agoWe're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.
- deleted 16y ago[deleted]
- powdahound 16y agoI'm seeing 1 EBS server out of 9 having issues (5 in one availability zone, 4 in another). CPU wait time on the instance is stuck at 100% on all cores since the disk isn't responding. Sounds like others are having much more trouble.
- vnorby 16y agoFrom EngineYard: "It looks like EBS IO in the us-east-1 region is not working ideally at this point. That means all /data and /db Volumes which use EBS have bad IO performance, which can cause your sites to go down."
- kylec 16y agoI guess this is one Reddit outage that can't be blamed on poor scaling
- thezilch 16y agoLooks like troubles in only one availability zone.
- mechanical_fish 16y agoThat seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.
- justinsb 16y agoAWS randomize the zones per account, so "your" -1b is not necessarily the same as "my" -1b. I'm only seeing problems in my -1c. Are you seeing 3 zones failing all under the same account? If multiple AZs are down, AWS are going to have some serious explaining to do...
- jedberg 16y agoWe (reddit) are seeing failures in all zones.
- redthrowaway 16y agoIf memory serves, Amazon's reporting of what zones are experiencing problems has been...optimistic...in the past.
- nlo 16y ago"AWS randomize the zones per account" Interesting. This is news to me. I found some more info here (Google cache since alestic.com is presently unreachable): http://webcache.googleusercontent.com/search?q=cache:0jxzyFj0J7wJ:alestic.com/2009/07/ec2-availability-zones&hl=en&gl=us&strip=1 http://webcache.googleusercontent.com/search?q=cache:0jxzyFj...
- mcritz 16y agoThis feels the same way as hearing that the whole Internet just got shut down.
- jedberg 16y agoYes, they are. :(
- griosa 16y agono reddit at work today!
- jedberg 16y agoBelieve me, the last thing I want is to be up at 3am working on this. I'd much rather be sleeping and letting you not work.
- chrislomax 16y agoAre you part of the team resolving it?
- hendi_ 16y agoYay for relying on the cloud \o/
- jedberg 16y agoThat's like saying "yay for relying on a datacenter"
- hendi_ 16y agoNote that I've written "relying", not "hosting", meaning I'm well aware of the fact that using "the cloud" as an additional layer of capacity can be a good thing. But relying on the cloud means that you're giving up control. Power outage in your data center? Buy diesel generators next time! Faulty network card? Put in two redundant ones! AWS down? Wait and pray!
- jedberg 16y agoGo back to the building metaphor. Building datacenters is not our core business, so we outsource it. Much like building a building is not our core business, nor running a telephone system, so we outsource those things too. All business critical, but still outsourced.
- hendi_ 16y agoOf course it's okay to outsource things; and it's also okay to outsource critical stuff. I'm doing that too. But, outsourced or not, critical things should be as redundant as needed, and not relying on a single point of failure. If in a hospital a life-keeping machine depends on electricity, I hope they have diesel generators available even if their outsourced power supply fails.
- Peaker 16y agoI think people thought they were outsourcing the job of not having a single point of failure. Having a critical service supplied by someone does not necessarily mean there's a single point of failure, because that someone might be working to make sure there is no single point of failure.
- mathrawka 16y agoI think this is a good example of how the "cloud" is not a silver bullet to making your site always up. AWS provides a way to keep it up, but it is up to each developer to ensure that they are using AWS in a way to make sure their site can handle problems in one availability zone. I think we will see more of a focus from big users of AWS about focusing on how to create a redundant service using AWS. Or at least I hope we will!
- tybris 16y agoNetFlix should sell their chaos monkey as a commercial product.
- justinsb 16y agoDon't let AWS hear that, or they'll charge us for their failures by rebranding it as a feature.
- jedberg 16y agoThis outage is affecting all AZ's in the East. So even a multizone setup wouldn't help for this one. Only a multiregion setup. This outage is a lot like having your entire datacenter lose power.
- mathrawka 16y agoI thought AZs were supposed to be different physical data centers. If that is not the case, then having a multi-region setup would be a necessity for any major sites on AWS. Perhaps there will be a time where to truly be redundant, one would need to use multiple cloud providers. Which would be a _huge_ pain to do now I imagine, with all the provider lock-ins we have.
- jedberg 16y ago> I thought AZs were supposed to be different physical data centers. They are. Which means this is probably a software issue or some other systemic issue.
- potomak 16y agoQuora says: "We'd point fingers, but we wouldn't be where we are today without EC2."
- olegp 16y agoNice way to point fingers while saying you're not.
- stevepotter 16y agoYes, that was by far my favorite comment. Well played.
- smtm 16y agoAWS/S3 has become the new Windows - great SPOF to go for if you want to attack. This space needs more competition.
- deleted 16y ago[deleted]
- mathrawka 16y agoIt is more noticeable since so many large sites use it. Kind of like a MS BSOD was a common joke because so many people used MS Windows. If we were all having our own rental servers then.... well, many sites that we know wouldn't be around :)
- dangrossman 16y agoTwo years ago TechCrunch was publishing an article every time Rackspace went down listing all the hot startups down along with it. AWS is no more a SPOF than any other major hosting provider.
- helium 16y agoI just launched a site on Heroku yesterday and cranked up the dynos up in anticipation of some "launch" traffic. Now, I can't log in to switch them off. Thanks EC2, you owe me $$$s
- mike-cardwell 16y agoIsn't that a design flaw in Heroku? Shouldn't you be able to log into Heroku and change stuff like that even if the entire of Amazons cloud service is down?
- Maxious 16y ago> Nothing special-case here: we deploy with git push, just like any other Heroku user. Dogfooding is good for you. http://blog.heroku.com/archives/2009/4/1/fork_our_docs/ http://blog.heroku.com/archives/2009/4/1/fork_our_docs/
- mike-cardwell 16y agoThat's all well and good, but it's no use for their customers if/when Amazon goes down.
- SandB0x 16y agoAgree. You can delegate work but not responsibility.
- random42 16y agoActually, I'd expect Heroku to not charge for when the site was down, as they are clearly not available, it does not sound fair if they charge for it. Am I expecting too much from them?
- helium 16y agoMy app on heroku is running, it's just that I can't log in to their management console to de-allocate resources that I am paying for by the hour.
- dsl 16y ago4/21/2011 is "Judgement Day" when Skynet becomes self aware and tries to kill us all. http://terminator.wikia.com/wiki/2011/04/21 http://terminator.wikia.com/wiki/2011/04/21 I am just a little freaked out right now.
- jedberg 16y agoDon't worry. If skynet is in EC2, we'll be fine.
- HelloBeautiful 16y agoYou don't understand, Skynet is using all Amazon resources, hence the outages ;-) Amazon have stated many times that amazon.com itself runs mostly on AWS platform, but it works fine now ...
- cicloid 16y agoAWS platform on a private cloud, it is not the same as AWS platform for us commoners.
- ILIKECAKE 16y agoROFL well played
- crb 16y agoDo we all regret letting GLaDOS reboot yet?
- crb 16y agoSigh, Portal jokes were so much more popular on Reddit. :)
- mindcrime 16y agoHN really isn't the place for internet memes, jokes about pop culture, and things that are judged trivial / frivolous. Part of what makes the HN community what it is, is a focus on high-quality, reasoned, rational discourse. IOW: HN != Reddit
- oomkiller 16y agoThey better start writing their explanation now. Multiple AZ's affected?
- jedberg 16y agoThey better be cutting me a check too.
- ig1 16y agoAs an advertiser on Reddit, you guys better be cutting me a check too :) (although my app is also dead right now so not as if the advertising would do me any good...)
- jedberg 16y agoSend me the link to your ad (when we come back up) and I'll comp you a day.
- random42 16y agoAssuming the outage is only for a day, right?
- reustle 16y agoIsn't it paid on impressions?
- AgentConundrum 16y agoI don't know how the sidebar ads work, but the featured links at the top of the page are run on a sort of auction system. Everybody that wants a piece pays however much they want (minimum of $20) for the day, then all the ads are totaled up and each ad gets a percentage of pageviews corresponding to the their percentage paid on that days revenues. If that wasn't clear (and I'm not sure it was), assume you and I were both the only advertisers on reddit for a day. If I pay $20 and you pay $20, we would both have our ads displayed on 50% of pageviews for that day. If instead I paid $80 and you still only paid $20, then I would get 80% of pageviews to your 20%, regardless of the total pageviews for the day.
- deleted 16y ago[deleted]
- frekw 16y agoIt's a bit ironic that Amazon WS has become a SPoF for half the internet.
- ig1 16y agoA couple of hours into the failure, and no sign of coverage on Techcrunch (they're posting "business" stories though). It shows how detached Techcrunch has become from the startup world. Edit: I tweeted their European editor about it and he's posted a story up now.
- mathrawka 16y agoSo do we get some credit on our AWS accounts? I haven't really read their SLA for EC2.
- elliottcarlson 16y ago"In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below." http://aws.amazon.com/ec2-sla/ http://aws.amazon.com/ec2-sla/
- dpogg1 16y agoI did the math on it, with the downtime so far it's almost approaching the magic 99.95% barrier where everyone gets a 10% bill credit. Wouldn't that be something to light a fire under the collective arses of those in charge of keeping EBS stable.
- lylejohnson 16y ago> Wouldn't that be something to light a fire under the collective arses of those in charge of keeping EBS stable. Yeah, the only possible explanation is that Amazon's put a bunch of lazy guys in charge of EBS. How hard a job could it be?
- dpogg1 16y agoI'm certain they're plenty talented and hard-working, but there's got to be something in their way. Just like the guy said as he was leaving Reddit, management seems to be pointing fingers in different directions as to why this is going on and how it can be resolved. It's sad, though, because I don't want to think of Amazon as a company with a potentially toxic corporate culture. I enjoy my free Prime membership as a student :D
- mleonhard 16y agoAmazon's corporate culture works pretty well. The problem is that these large-scale multi-tenant services are new technology and very difficult to create and run.
- espeed 16y agoQuora is down, and evidently "They're not pointing fingers at EC2" -- http://news.ycombinator.com/item?id=2470119 http://news.ycombinator.com/item?id=2470119 -- I was going to post a screen shot, but evidently my Dropbox is down too.
- jwr 16y agoIt's really mostly EBS failures, so the title is overly dramatic. And EBS has been known to have issues.
- dogas 16y agoThis is incorrect. EBS is part of the issue, as well as intermittent connectivity failures for EC2 instances. We're also seeing connectivity issues with our elastic load balancers.
- bad_user 16y agoI've got 2 EC2 instances with EBS root devices -- both are unreachable. My RDS instance is responding, but it is very slow -- struggling to make a backup.
- espeed 16y agohttp://www.information-age.com/channels/the-cloud-and-virtualization/news/1619753/amazon-web-services-struck-by-performance-issues.thtml http://www.information-age.com/channels/the-cloud-and-virtua...
- olegp 16y agoAssuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.ua/video-2011-tech-podsechin http://iforum.com.ua/video-2011-tech-podsechin EDIT: here are the slides if you can't bother watching the video http://bit.ly/eqDNei http://bit.ly/eqDNei
- ghshephard 16y agoEvery time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology. While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime. I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.
- bad_user 16y agoLets hope they don't lose any of my data.
- olegp 16y agoThe problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great value for money. There is definitely value in platforms, but those platforms should be built on completely open standards with no vendor lock in or influence, as is the case with OpenStack for example: http://www.theregister.co.uk/2011/02/10/rackspace_buys_openstack_partner/ http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens... If you didn't watch my talk, I should point out that I'm working on Akshell (http://www.akshell.com http://www.akshell.com), which is itself a platform provider, so I am in fact agreeing with you.
- antonioe 16y agoHad our blog go down. Didn't realize it was AWS wide..did a reboot. Now I am in reboot limbo. Put an urgent ticket into Amazon. They just said they are working urgently to fix the issues. Let's see how long this goes.
- xlevus 16y agoRuh Roh. The service I'm using to acquire accommodation seems to be dependent on AWS. Guess I'm going to be homeless tomorrow if it doesn't get fixed. :X Note to self. Don't ever build a service reliant on AWS.
- crashpadder 16y agoIf its us (crashpadder.com) - sorry for the inconvenience. Email us hello [at] crashpadder.com and we'll do all we can when we're back up. We'd moved to AWS from an even worse host a few months ago, and until this morning had been pretty impressed...
- asymptotic 16y agoAmazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to commodity hard-drive failure rates of 4%. Hence, if you wanted a higher level of data availability you'd use more than one EBS volume in different regions. These outages are affecting North America, and not Europe and Asia Pacific. That's it. Why is this even news? Were you expecting 100% availability?
- stef25 16y agoReddit's been down for several hours today, I'm sure they are already way lower 99.95%.
- JonnieCache 16y ago0.05% of one year is 4 hours, 22 minutes and 48 seconds.
- bad_user 16y agoWhich means their entire quota for this year is all gone.
- yuvadam 16y agoand then some...
- stef25 16y agoDon't forget to take in to account ALL of Reddit's downtime; there is quite a bit of it.
- cmurdock 16y agoReddit experienced some issues with Amazon a month ago that resulted in the site being down for almost a day. I'm pretty sure they're way below that percentage.
- jws 16y agoSilver lining: Hopefully I can test my "aws is failing" fallback code. (my GAE based site keeps a state log on S3 for the day when GAE falls in a hole.)
- G_Morgan 16y agoThis code should be well tested by now. Amazon is doing you a failure by being rubbish.
- Jun8 16y agoThis is like witnessing your parents having sex while a kid: you sort of knew this is a possibility but it is a devastating blow to your belief system nevertheless. The amount of services I use that depends on Amazon is amazing. They have really become the utility company of the Web.
- smackfu 16y agoSo when big sites deal use Amazon Web Services for major traffic, do they get a serious customer relationship? Or is it just generic email/web support and a status page?
- mleonhard 16y agoThere are subscriptions for various levels of support, from $50/mo for 12 hour response time to $15k/mo for 15 minute response time. http://aws.amazon.com/premiumsupport/ http://aws.amazon.com/premiumsupport/
- smackfu 16y agoInteresting pricing. The Platinum seems priced to have no one use it, considering how much of a jump it is over Gold.
- iclelland 16y agoI would say, rather, that it is priced to have very specific sorts of customer using it. The relationship between the pricing tiers changes fairly drastically, depending on how much you are already spending on Amazon Web Services. Gold, for instance, starts out at 4x the price of Silver support, but by the time you're spending 80K/month on services, it's only a $900 premium (and stays there no matter how much bigger your bill is). At the $150K/month level, it's a 2x jump from Gold to Platinum, which may or may not be a huge jump, considering the extra level of service you get.
- paraschopra 16y agohttp://venuetastic.com/ http://venuetastic.com/ - feel bad for these guys. They launched yesterday and down today because of AWS. Murphy's law in practice.
- tedjdziuba 16y agoThey are scaling in the cloud, at least.
- hysterix 16y agoWow. I can only imagine the intense frustration the site owner must be feeling right about now. Makes you really stop and question the whole "cloud" based service. Or at least should make you realize you need fall-backs other than the cloud service itself.
- swedegeek 16y agoIn case anyone is late to the party and missed the non-green lights on the AWS status dashboard, here is the page as of about 9:30 EDT... http://screencast.com/t/p69xAoDJRSer http://screencast.com/t/p69xAoDJRSer
- ck2 16y agoSo what percentage of the top 1000 sites are now crippled by this?
- dmuth 16y agoBeing unable to get much done here, my co-workers have found other things to do in the office: http://www.youtube.com/watch?v=u1-oGxDHQbI http://www.youtube.com/watch?v=u1-oGxDHQbI :-P
- jedberg 16y agoWhat did I just watch and why?
- tybris 16y agoAll hosting services go down occasionally. If you want to stay up you need to build a fault-tolerant distributed system that spans multiple regions and potentially multiple providers. Also, Amazon should fix EBS.
- kennethologist 16y agoThankfully, my major clients are using the Asia EC2 Instances!
- timf 16y agoSome quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984 https://twitter.com/adrianco/status/61076362680745984
- campnic 16y ago"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
- jedberg 16y agoAt their level of income, this is true. For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.
- Goladus 16y agoI would also be shocked if Amazon isn't giving Netflix preferred pricing because it's such a high-profile customer.
- adrianco 16y agoNetflix pays standard rates for instances but uses reserved instances to pay less on bulk EC2 deployments
- adrianco 16y agoI think it's incredible that you guys can run a site at all with the few people you've got. Hope it all gets better again soon.
- ig1 16y agoGiven that Heroku's parent company (Salesforce) owns a cloud platform, it seems kinda inevitable now that Herkou will perhaps sooner-than-later switch back-ends (or at least use both)
- greaterscope 16y agoWish were able to download our ebs snapshots, which are supposedly hosted on S3. What does everyone else do?
- dangrossman 16y agoI take the snapshots. I also have servers send backups to each other each night. I also have a nightly cron job run and rotate backups of the most critical databases to an external drive on my home network. A Tonido Plug does that job (Ubuntu on a tiny ARM server in a plug that costs virtually nothing to run). Now, some of the databases are simply too large or under too much load to take a live backup while the sites are running. Those I run on Amazon RDS with the MultiAZ feature enabled. There should be two copies of the database running at all times, both servers keeping a 3 day binlog for point-in-time backups, and making a nightly snapshot to S3. I have to rely on Amazon for that. But I still take daily home backups of the most valuable individual tables off those servers, like user registrations and payment records. Even if I can't have off-site backups of the whole database, I'll have off-site copies of the part I'd need most in case of an Amazon-entirely-offline catastrophe.
- singlow 16y agoIt's definitely a limited outage. My three instances seem to have operated all night with no problem. Two of them are EBS instances.
- alexpopescu 16y agoInstead of enumerating who's down, I'd be more interested to hear about those that survived the AWS failure. We could learn something from them.
- dmuth 16y agoHoly crap. An Amazon rep actually just posted that SkyNet had nothing to do with the outage: https://forums.aws.amazon.com/message.jspa?messageID=238872#238872 https://forums.aws.amazon.com/message.jspa?messageID=238872#...
- piramida 16y agoToday, April 21st 2011, according to the "Terminator", Skynet was launched... No wonder AWS is down
- mtodd 16y agoWhy is ELB not mentioned at all on the Service Health Dashboard? We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress". A support issue with Amazon indicated that it was related to the ongoing issues and to monitor the Service Health Dashboard. But, as I mentioned before, ELB isn't mentioned at all.
- jen_h 16y agoWe've got a single non-responsive load balancer IP in one of our primary ELBs (others have been fine for several hours now), so while everything else for us is up & running, still have transient errors for folks that get shunted to through that one system. The interesting thing about the ELB in a situation like this is that I believe it may, in many instances, be better to hobble along and deal with an elevated error rate if at least some of your ELB hosts are working than to re-create the entire ELB somewhere else, especially if you're a high-traffic site where you may hit scaling issues going from 0 to 60 in milliseconds (OMMV, but we've been spooked enough in the past not to try anything hasty until things get back to normal).
- mtodd 16y agoWe have an identical load balancer to one that is causing problems so we're lucky enough to reroute traffic through that one instead to get to the same boxes. (The boxes serve two different APIs through two different DNS CNAMEs so we split the ELBs for future and sanity). In this case, it's helped us out. Alternatively, we would've just routed all traffic to our west coast ELBs.
- mtodd 16y agoQuote from the AWS support rep: "I can confirm that ELB has been affected by the EBS issue despite the lack of messaging on the AWS Dashboard".
- hcentelles 16y agoIt seems like availability zone us-east-1c it's working, i can launch a EBS backed instance right now.
- marjanpanic 16y agoAmazon down - just one more reason to try out BO.LT and their amazing CDN and page sharing services... Just launched: http://techcrunch.com/2011/04/21/page-sharing-service-bo-lt-lets-you-copy-edit-and-share-almost-any-webpage/ http://techcrunch.com/2011/04/21/page-sharing-service-bo-lt-...
- antonioe 16y ago1:23EST and Reddit is back up. Quora/4SQ still down. My site still down.
- jjm 16y agoEveryone talks about SLAs but I believe it doesn't consider the fact that the EBS vols are still up (not on fire, and available) and are phantom writing or that the network is queued up the wazoo so writes don't even happen in a timely manner as you'd expect.
- elliottcarlson 16y agoI'm not sure that just because they are up, yet unusable, would negate an SLA. You could have a dedicated server in a datacenter - if the network goes out, your machine is still up and happily waiting to serve requests - but it's still unusable and not actually in service.
- pextris 16y agoreddit.com is down, but luckily http://radioreddit.com http://radioreddit.com is not.