5 ms·
Ask HN: How has the Amazon AWS outage affected you?
Thanks to the Amazon AWS Outage today, I missed the chance to have a QR code generated by my app displayed in an ad in a major newspaper. My Heroku hosted app (I have several) is still down. My other app was experiencing a huge traffic (30X usual) right when the outage happened.
Both apps are just fun toys, but it hurts for them to be down at such a critical time.
How has the outage affected you?
- api 16y agoReddit is down, so I got more work done today.
- garyrichardson 16y agoI didn't. I was too busy reading blog posts about the downfall of cloud computing.
- venturebros 16y agoI didn't! I kept taking naps and coming back to see if it was up. Reddit actually helps me work more.
- deleted 16y ago[deleted]
- jk215 16y agoI woke up this morning in disbelief that it was still down. Its friday all I want to do is put off work and browse reddit.
- abraham 16y agoI wasn't able to view a Foursquare checking for a little while and I've had significant amounts of my morning news be related to AWS, the cloud, downtime, etc.
- zacharypinter 16y agoMy team's single sign on prototype was deployed to heroku so that we could develop our apps against it (Android, iPhone, WP7, jQuery Mobile). It would have stalled development, but with a bit of tweaking I was able to get the prototype deployed to cloudfoundry and everything continued as planned.
- ascendant 16y agoThe company I work for offers a small hosted document management SaaS solution that uses Amazon as the hosted environment, so we've been having issues related to (surprise surprise) EBS.
- haploid 16y agoStrangely enough, not at all. We have a few ec2 instances running in the us-east region, allegedly the region that was supposed to be affected. They have been up and available all day, without issues of any kind.
- colinplamondon 16y agoOur registration/login/download/stat servers for all our iOS apps (Free Books + Free Audiobooks + Classicly) are all hosted on Heroku, and have been completely down all day. Not good.
- bdclimber14 16y agoI run my main startup on Heroku, so OrangeSlyce.com has been completely down all day.
- dpcan 16y agoMight as well get some good out of it.... please describe what your service is.
- bdclimber14 16y agoSadly, I almost replied "just check out the website." I made a site for graphic design students to find freelance gigs that are posted by local small businesses who need cheap, small amounts of design work done. Thankfully this hasn't been very big, and mostly just a side project recently, so not too many users are being affected.
- dpcan 16y agoThat's a great idea, can't wait to check it out when it's back.
- mtogo 16y agoActually, i was surprised to find i wasn't affected by it at all. Apparently everything i use either knows what they're doing or wasn't in the us-east-1 region.
- sofuture 16y agoDitto, we're just in the process of moving to EC2, and were happy to find our staging environment (the only thing moved over so far) is happily healthy (of course it gets minimal load).
- webmonkeyuk 16y agoMy hastily thrown together article on how to work around EC2 outages got to #8 on HN and received ~2000 reads http://news.ycombinator.com/item?id=2471258 http://news.ycombinator.com/item?id=2471258
- jaredwill 16y agoI manage four EC2s all with EBS and located on the east coast; luckily haven't had any issues.
- mcotton 16y agoI am running my phone processing/tracking app on EC2 so it's been up and down all morning. The fall back also has some dependencies on S3 so it has been a bad day.
- adelmand 16y agoI provide media streaming services via EC2 to a few customers - basically all of the non-live streams are unavailable.
- nostromo 16y agoOur database (RDS) is completely inaccessible -- so our website has been down since 2am or so, showing our maintenance page. It's backed up -- but we have no way to download the most recent backup and move to another server. We tried rebooting our db about 11 hours ago -- and it's been stuck in reboot mode ever since. We tried creating a copy from a recent backup -- but it's stuck in boot mode. We can't move the snapshot to the West Coast farm, so we're really pretty blocked until Amazon gets things in order. A lot of people on the boards suggested Amazon make the automatic backups of RDS available for download for instances like these. Having a backup is great, but not if you can't access them in an emergency. On the message boards, someone said that they had selected to have RDS automatically keep a copy in multiple availability zones -- and they said that worked for them. I'm not sure however, since a top post on HN is saying that all of the Virginia zones were effected -- so your mileage may vary.
- st3fan 16y ago"Offsite Backups" - Where Offsite means, not on Amazon's infrastructure. Seriously.
- cperciva 16y agoNo effect at all that I've noticed. Tarsnap doesn't use EBS.
- hdragomir 16y agoNope. Our failovers are on self hosted servers, where the core of each app lives anyway.
- henryrose 16y agoOur test environments aren't available.
- mindcrime 16y agoHow has it affected me? My posting of the Wikipedia page on "Fallacies of Distributed Computing" made it to #2 on HN, and I got a big karma bump; and wasted more time than usual on HN today, discussing this stuff. Also, since Quora was down, I didn't get my usual quota of Quora surfing in. Otherwise, today has been "business as usual." :-)
- freerobby 16y agoWe're temporarily down with no permanent damage. But we have no way to relay that info to our customers because everything, including our users' email addresses, is inaccessible. Big oversight that I will be fixing as soon as we're back up. It feels absolutely terrible to only be able to assure customers that their data is safe _after_ they write a panicked email asking if their data is gone.
- adpowers 16y agoMy website and e-mail have been down for 12 hours and counting. Alas, my only instance that has survived this is running a website that is still under development.
- gcr 16y agoMy site still works! (Thanks, prgmr ;))
- triviatise 16y agotriviatise.com is down, but we just ended our last giveaway to close out our alpha on the 15th so the impact is minimal. We are working to figure out what a good backup plan to heroku is. Im sure a lot of other people (including heroku ) are doing the same.
- martingordon 16y agoI was hosting a client's app on Heroku and they had a big pitch today during which they weren't able to show the site. Really, really horrible timing.
- dholowiski 16y agoYikes. That's how I feel... Yesterday I explained how my site, although very ugly, could handle all the traffic you could throw at it and more. Today, it can't handle any.
- benologist 16y agoMy leaderboards, user-created levels and other miscellaneous bits of Playtomic run on AWS via MongoHQ and I've spent 1/2 the day answering emails explaining the situation. On the plus side MongoHQ are planning to introduce the ability to mirror databases onto our own servers which is going to be frigging awesome. It at least highlighted another issue - because users are trying to access databases that don't exist they're taking ages to time out which is bogging down the servers' other thousands of requests/second.
- ig1 16y agoStartup's site has been down all day, probably lost about ~$500-$1000 in sales, more concerned about the longer term loss of goodwill and the potential damage to google seo rankings though.
- kathryna 16y agoI run an app that helps farms manage their weekly or monthly CSA programs. One farm had their monthly distribution yesterday - thankfully they had already downloaded the data they needed to pack and label the orders, but they weren't able to tell customers their final total or send reminders, so they had higher-than-average missed pickups and are receiving money several days later than usual. Farmers are already distrustful of digital/automated solutions, so I'm sad that this has added to that distrust.
- callmeed 16y agoI've got an app in private beta on Heroku. Closing in on 2 days of downtime. Not a good way to instill confidence in the 50 or so potential customers I let into the beta. Plus my iOS app gets data from a Heroku app that is down.
- fomojola 16y agoSaw the same RDS behavior nostromo saw: was actually truly interesting. After 2 days of arguing with the RDS database I actually spun up a new local storage instance, logged in over the mysql console at a shell (which actually worked, surprisingly enough) and then found out that a single table (this is a Wordpress instance) was hung: I guess the RDS instance couldn't access the disk file for that table (it was MyISAM which I guess isn't in memory?) so any requests for that specific table hung. As it so happens that was the wp_options table, so there was no way to get past that. So I sat there with mysqldump and exported each table separately, then spun up a new MySQL instance in another AZ and then imported each table piecemeal. I then re-created the wp_options table from another dump (from an early prototype on Linode, actually) and then manually fudged the values till it all worked. That eventually worked for me.