29 ms·
Ask HN: Is S3 down?
I'm getting
{
"errorCode" : "InternalError"
}
When I attempt to use the AWS Console to view s3
- SubiculumCode 10y agonews.ycombinator.com seems really slow right now. s3 dependencies?
- myth_drannon 10y agoLooks like SoundCloud is hosting the tracks on S3 , can't program without my music...
- trakl 10y agohttp://venturebeat.com/2017/02/28/aws-is-investigating-s3-issues-affecting-quora-slack-trello/ http://venturebeat.com/2017/02/28/aws-is-investigating-s3-is... https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_goes_haywire/?mt=1488309539087 https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...
- murphy52 10y agoWe host with TierPoint and they are reporting a massive DDOS attack
- zedpm 10y agoCan anyone comment on mitigating issues like this with S3 Cross-region replication? I'm reading up on it now while one of my services is dead in the water.
- thraway2016 10y agoThe only appropriate comment is that this issue is affecting all of our buckets, both in us-west and us-east. Replicating to another region would yield no useful benefits in this specific failure scenario.
- nolite 10y agoCan't agree with this. Buckets in eu-west-1 are fine
- zedpm 10y agoI did some digging and experimentation, and so far it looks like you could keep a backup bucket in another region and use bidirectional replication [0] to keep the two buckets in sync. If something like this happened again, you could point your app(s) at the bucket in another region and keep accepting data. The objects would eventually get replicated back to the original bucket, and you could cut over again when service was restored. There does seem to be an appreciable replication lag, so you could run into problems during your cut where some objects had not yet been replicated, but your app ought to handle things like that gracefully anyway. [0] https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
- mtdewulf 10y agoYep, same here.
- josephlord 10y agoLooks like it. Brief panic caused here.
- baconomatic 10y agoSeeing it here as well.
- jacobevelyn 10y agoWe're getting errors indicative of an S3 outage too.
- TheVip 10y agoSame problem bro...
- mwambua 10y agoNot sure if it's related... but I'm having issues with Amazon Cloud-drive.
- iamdeedubs 10y agoI would assume, I couldn't even share a screen shot of my evidence to my team on slack!
- deleted 10y ago[deleted]
- kangman 10y agoany one get more info from AWS?
- rhelsing 10y agoHaving issues as well.. big issues..
- alexleclair 10y agoYup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved? [1] https://status.aws.amazon.com/ https://status.aws.amazon.com/
- tuna-piano 10y agoStill green now, 8 minutes in.
- bpicolo 10y agoI've had a few non-Amazon providers tell me AWS things are not working in the last 5 minutes, no note from Amazon though. Nice.
- leesalminen 10y agoJust sent out a notice to our customers via our status page. I really wanted to be able to add a link back to AWS detailing the issue but that's a pipe dream I suppose.
- fudged71 10y ago... still green
- cheeze 10y agoJust went yellow Edit: nevermind
- matthuggins 10y agoStill green for me
- deleted 10y ago[deleted]
- leesalminen 10y ago
- zitterbewegung 10y agoCan't access my website which is hosted on s3 (http://joshuajherman.com http://joshuajherman.com).
- leesalminen 10y agoFreshDesk makes extensive use of S3 and it's been unbearably slow to load for the past hour or so. All on S3 requests.
- deleted 10y ago[deleted]
- willcodeforfoo 10y agoUh-oh. Same here... and tried taking a screenshot of pinging s3.amazonaws.com and Slack upload hung.
- johngalt 10y agoSysadmin: I can forgive outages, but falsely reporting 'up' when you're obviously down is a heinous transgression. Somewhere a sysadmin is having to explain to a mildly technical manager that AWS services are down and affecting business critical services. That manager will be chewing out the tech because the status site shows everything is green. Dishonest metrics are worse than bad metrics for this exact reason. Any sysadmin who wasn't born yesterday knows that service metrics are gamed relentlessly by providers. Bluntly there aren't many of us, and we talk. Message to all providers: sysadmins losing confidence in your outage reporting has a larger impact than you think. Because we will be the ones called to the carpet to explain why <services> are down when <provider> is lying about being up.
- rdiddly 10y agoIt's not a lie, it's an "alternative fact" about how totally like awesome AWS is!
- dragonwriter 10y ago> Because we [sysadmins] will be the ones called to the carpet to explain why <services> are down when <provider> is lying about being up. But isn't that the whole point of lying: to the less technical manager (often the only person whose view matters at major customers), the status board saying "up" means the problem is the sysadmins, not the vendor.
- mjcl 10y agoThat works in the vendor's favor in the short term, but can screw them in the long term because you get staff who go the extra mile to avoid the vendor in the future, including structuring requirements to avoid them. For example, by experience and gossip I know Wind stream has awful reliability, but they handwave that away. By including a requirement I knew they couldn't meet (dynamic E911), they were knocked out of a 200 site VoIP RFP early.
- MaxfordAndSons 10y agoIt's unbelievable that the status page is still showing green checkmarks, almost what, 2 hours into the outage? edit: oh, it is actually because of the outage! So if they can't get a fresh read on the service status from s3, they just optimistically assume it's green... even though the service failing to provide said read... is one of the services they're optimistically showing as green XD
- cwe 10y agoDropbox using this? Can't seem to sync
- rckclmbr 10y agoThey used to, but i think they are off it now.
- GabeIsman 10y agoYes.
- meddlepal 10y agoTotally fucked.
- jgacook 10y agoYup - dead in the water
- c4urself 10y agoIt is, of course the checkmark will stay green throughout this as Amazon doesn't care about actually letting its customers know they have a problem.
- danielmorozoff 10y agoYea seeing the same thing
- scrollaway 10y agoDown in US-East-1 as of 17:40 GMT. Amazon SES also down in US-East-1 as of a few minutes later. Hearing reports of EBS down as well.
- _callcc 10y agoSES also down.
- pmalynin 10y agoDown from the outside; The internal access (from within EC2) APIs still work.
- 65827 10y agoDead as a doornail for me
- dang 10y agoAll: I hate to ask this, but HN's poor little single-core server process is getting hammered and steam is coming out its ears. If you don't plan to post anything, would you mind logging out? Then we can serve you from cache. Cached pages are updated frequently so you won't miss anything. And please do log back in later. (Yes it sucks and yes we're working on fixing it. We hate slow software too!)
- djb_hackernews 10y agoAnyone else seeing ELB/ALB issues?
- asolove 10y agoYes
- redthrowaway 10y agoYup. Some of our machines in us-east-1e dropped out of the load balancers.
- cryreduce 10y agoin the s3 web interface requests to S3 backend end with 503 Service Unavailable
- booleanbetrayal 10y agoS3 and Elastic Beanstalk (S3 dependencies) ... no issues with RDS at the moment
- 4wmturner 10y agoIs cli working for anyone else? I can't use the console UI, but aws s3 ls and get commands seem to be working fine.
- ARolek 10y agoSame here. US East (N. Virginia)
- rbirkby 10y agoCan anyone get Alexa to play music? Is this related?
- xvolter 10y agoSeeing the same here
- 4wmturner 10y agoIs cli working for anyone else? I can't use the console UI, but aws s3 ls and get commands seem to be working fine.
- twiss 10y agoS3 Ireland (eu-west-1) seems to be doing fine at first sight.
- amcrouch 10y agoIt appears to be down. My website runs on S3 and my monitors are going nuts!
- talawahdotnet 10y agoYup it looks so. My console says I have zero buckets, my Lambdas are timing out and https://aws.amazon.com/ https://aws.amazon.com/ returns a big: "500 The server encountered an error processing your request." message
- alanning 10y agoOur lambda functions are also unavailable. Lucky for us we didn't move any of our critical functionality to lambda yet although we are planning to once we have an EC2 backup in place...
- ec109685 10y agoDon't know if share911 is still your product (its about page is down), but you could run your critical services on top of something like PagerDuty to give you the reliability you need.
- andrewfong 10y agoYeah, we host on S3 (US-East-1 I think) with Cloudfront for caching / SSL. Some of our requests get through but it's been intermittent. Lots of 504 Gateway Time-Outs when retrieving CSS, JS.
- rawrmaan 10y agoIncredible how much stuff this affected for me. Opbeat is not loading and I can't even deploy because CircleCI seems to depend on S3 for something and my build is "Queued". This seems so dangerous...
- piquadrat 10y agoHi, Beni from Opbeat here. Our community site is indeed down unfortunately, but the rest should be unaffected. Where do/did you experience issues?
- rawrmaan 10y agoHi Beni! My dashboard wasn't loading (the JS assets from cloudfront.net seemed to be throwing an access-control-origin error) but it is now working again. Very slow, but it works. Fortunately nothing critical is happening right now so no worries. Love your service :)
- piquadrat 10y agoAh right, static assets are an issue, didn't notice it right away due to local browser caching. Sorry about the trouble!
- vidarh 10y agocircleci.com won't even load for me.
- tbeutel 10y agoI'm getting this using s3cmd: $ s3cmd ls WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 6 sec...
- greenhathacker 10y ago"I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened."
- ethanpil 10y agoWhat kills me is that their status page still shows nothing is wrong. https://status.aws.amazon.com/ https://status.aws.amazon.com/
- jeffijoe 10y agoTheir status page probably can't refresh because S3 is down.
- amazon_throw 10y agocorrect.
- jontro 10y agoI get this in my aws console. Increased API Error Rates 09:52 AM PST We are investigating increased error rates in the US-EAST-1 Region. Event data Event S3 operational issue Status Open Region/AZ us-east-1 Start time February 28, 2017 at 6:51:57 PM UTC+1 End time - Event category Issue
- jrs235 10y agoThey don't show it on the status dashboard at https://status.aws.amazon.com/ https://status.aws.amazon.com/ (at least at the time I originally posted this comment). But if you go to your personal health dashboard (https://phd.aws.amazon.com/phd/home#/dashboard/open-issues https://phd.aws.amazon.com/phd/home#/dashboard/open-issues) they report an S3 operational issue event there. Edit: Mine is reporting region us-east-1 Edit 2: And now the event disappeared from my personal health dashboard too. But we are still experiencing issues. WTH.
- deleted 10y ago[deleted]
- chickenfries 10y agoAlso seeing the intermittent event on the personal dashboard. Wonderful.
- jrs235 10y agoApparently they need a special/separate status page for everyone's personal health dashboard too. SMH.
- the_arun 10y agoSame here
- kyleblarson 10y agoAny specific regions? us-west-2 seems fine to me. [edit] now I can't see any of my buckets in the web interface.
- jrs235 10y agoAppears to be us-east-1
- magic_beans 10y agoDefinitely experiencing non-loading for dependencies hosted on S3 at the moment...
- jhaile 10y agoAll of our S3 assets are unavailable. Cloudfront is accessible but returning a 504 status with the message: "CloudFront is currently experiencing problems with requesting objects from Amazon S3."
- deleted 10y ago[deleted]
- samgranieri 10y agoI'm running into timeouts trying to download elixir packages, and I'm willing to bet this is the cause
- stevefram 10y agoYes, affecting elb in us-east-1 right now. web services are down and unable to bring up the elb screen in the aws console.
- chrisan 10y agoDown for us as well. We have cloudfront in front of some of our s3 buckets and it is responding with CloudFront is currently experiencing problems with requesting objects from Amazon S3. Can I also say I am constantly disappointed by AWS's status page: https://status.aws.amazon.com/ https://status.aws.amazon.com/ it seems whenever there is an issue this takes a while to update. Sometimes all you see is a green checkmark with a tiny icon saying a note about some issue. Why not make it orange or something. Surely they must have some kind of external monitor on these things that could be integrated here? edit: Since posting my comment they added a banner of "Increased Error Rates We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region." However S3 still shows green and "Service is operating normally"
- gtsteve 10y agoTo mitigate the effect of S3 going down I use cross-region replication to replicate objects to another S3 region. If S3 went down in my primary region I could update the app config to write back to the backup region and update the CDN configuration to use the backup region as an origin. I did that out of paranoia but it turns out this could happen to us. Does that sound like a sensible approach? Fortunately all my company's stuff is in eu-west-1 which still seems to be fine.
- kaishiro 10y agoI'm certainly not an expert here, but just to make you feel good (if nothing else), this is exactly what we did this morning (Melbourne time). Woke up to a bunch of flailing Lambda funcs on us-east-1. Luckily we're using Apex so cross deploying them to Singapore took all of 30 seconds. We were concerned about API Gateway since it was also sitting in us-east-1 but ended up not being an issue. Realized that redeploying S3 and Lambda across regions can be done in practically no time, but we would have been in trouble had we needed to replicated our APIs in another region. Going to start exploring spec'ing up our existing gateways in swagger to help with this.
- kangman 10y agowhat's the SLA for s3?
- nomadicactivist 10y agoHere it is, in typical SLA language... only AWS knows what it means https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- jpalomaki 10y agoThere's logic behind the complicated rules. The idea is that you can't cause the calculated availability to go down by generating lots of requests when the service is down.
- nomadicactivist 10y agoAnd my favourite part: "To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center." -- for a company that has built itself on automation, surely you could automate some bill credits based on the SLA.
- bdcravens 10y agoMany companies don't give out refunds and you have to proactively request. (alas my employer's value prop, in a completely different industry of course)
- ucaetano 10y agoDoes the AWS Support Center also run on AWS?
- kesor 10y agoYou can open tickets in support with API calls ... just sayin.
- BlackjackCF 10y agoYes. Have heard confirmation from Amazon that this outage is affecting us-east-1.
- agotterer 10y agoNot sure if its related or not (I'll just assume it is), but dockerhub is down as well. Haven't been able to push or pull for the last 15 minutes, some other folks complaining of the same thing.
- eggie5 10y agoconfirmed: dockerhub is down too
- jchmbrln 10y agoYeah, bad morning. I started by trying to pull a Docker image. When I couldn't, I tried to push some stuff to S3. Now of course I'm checking HN. :p
- serialpreneur 10y agoCan't pull images from either Dockerhub or Quay right now
- notheguyouthink 10y agoYea my self hosted concourse is down. Feels quite bizarre to have our internal CI so crippled by S3... though i'm still investigating, perhaps the workers are just hung after all of the failed docker pulls.
- 140am 10y agohttps://twitter.com/aws_shd/status/836635812020158464 https://twitter.com/aws_shd/status/836635812020158464 "Increased API Error Rates - 9:52 AM PST We are investigating increased error rates in the US-EAST-1" "S3 operational issue - us-east-1"
- Rapzid 10y agoUpload failing for me from Sacramento --> us-east-1
- notheguyouthink 10y agoSame here, i mistakingly went to the dashboard first too. Silly me.
- 0xCMP 10y agoSES is also down
- davidsawyer 10y agoSame
- huac 10y agoCanvas (the educational software platform) is down, and my friends/students are in bad shape now. 'sso.canvaslms.com' returns 504, assume from this S3 outage.
- pfela 10y agoTheir status page images are hosted on S3, so will be a while for the green checkmarks to update
- tommy1212 10y agofuck the police
- nanistheonlyist 10y agoEVERYBODY PANIC! US-EAST-1 is what we use, down for us.
- indytechcook 10y agoMy EC2 Servers are also not provisioning.
- deleted 10y ago[deleted]
- linsomniac 10y agoWe got timeouts to our bucket address from every location we tried starting at 10:37 Mountain time (GMT-7). Slack uploads started failing, imgur isn't working, and the landing page for the AWS console is showing a 500 error in the image flipper in the middle of the page. The Amazon status page has been all green, but there is a forum post about people having problems at https://forums.aws.amazon.com/thread.jspa?threadID=250319&tstart=0 https://forums.aws.amazon.com/thread.jspa?threadID=250319&ts... In the last couple of minutes that forum post has gone from not existing to 175 views and 9 posts.
- newsat13 10y agoYup, same here. For a moment, I was worried that the UI showed 0 buckets. Gave me a heart attack.
- AndyKelley 10y agoThis seems like an appropriate time as any... Anyone want to list some competitors to S3? Bonus if it also provides a way to host a static website.
- josephlord 10y agoRackspace's Cloudfiles. Does support static websites.
- leesalminen 10y agoI use both RS Cloud Files and Google's Cloud Storage. Google's is superior in nearly every way. The only con is that it is a Google product that could be deprecated at any point in time. But, with all the acquisition stuff happening over at RS, I'd be lying if I said I wasn't worried about them killing of their cloud offering.
- deesix 10y agoTwo things: 1) Google Cloud Storage can host static websites: https://cloud.google.com/storage/docs/hosting-static-website https://cloud.google.com/storage/docs/hosting-static-website 2) Google Cloud Platform has a 1 year deprecation policy, which would never happen with a product that so many companies and customer rely on (Google Reader had a small but passionate base) Disclaimer: I work on Google Cloud Platform
- leesalminen 10y agoTo clarify on #2, are you saying that Google Cloud Platform in its entirety has a 1 year deprecation policy? Or that individual products within the platform have a 1 year deprecation policy? I'm not worried about Google deprecating the Storage service but that they could kill off the entire platform. Also just wanted to say that I've been extremely happy with GCP thus far and all the services I've tried thus far have more features than RS. I really hope GCP is here for the long haul.
- methurston 10y agoDown for me.
- linsomniac 10y agoThe AWS status page is still showing all green but how has a header saying they are investigating increased error rates. https://status.aws.amazon.com/ https://status.aws.amazon.com/
- bkruse 10y agoThis is one of the times that I am glad to be running my own distributed object storage. I'm sure it's not as robust as Amazon, but......
- rhelsing 10y agoIs this only affecting US-EAST-1?
- learc83 10y agoOne of my heroku apps is down, and I cant' log into the heroku dashboard to check it out. I'm guessing this is related.
- robeastham 10y agoMe too. This couldn't have come at a worse time. Just launched a new site.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- jpmw 10y agoIt is, they are downloading slugs from S3.
- deleted 10y ago[deleted]
- prab97 10y agoQuora is down too.
- jahrichie 10y agosame here, east us seems non-responsive
- vinayan3 10y agoYes it's down for me. I can't access files stored on S3. Also, the service I run is hung trying to store files on S3.
- deleted 10y ago[deleted]
- sz4kerto 10y agoDropbox is down as well. This is going to be gud.
- sz4kerto 10y agoDropbox is down as well. This is going to be gud.
- gtrubetskoy 10y agothey're down to 3 nines edit: for the year, it only takes 52.57 minutes
- jflowers45 10y agotrello and giphy both seemed affected
- rrggrr 10y agoIts unreal watching key web services fall like dominoes. Its too bad the concept of "too big to fail" applies only to large banks and countries.
- elastic_church 10y ago"To big [to allow] to fail" is what that term means. This would extend to a service like amazon actually, where survival of the service would be an extraordinary effort in case this problem lasted for a long time. The way you imagined it, as 100% uptime, is incorrect.
- phil21 10y agoI didn't get that at all from the OP. His comment I'm fairly certain is of the "why the heck are we centralizing the web to 2 or 3 infrastructure companies" for what amounts to a minor amount of convenience. We've seen this story play out in other industries and it never works out well for average people. It's been astounding for me to watch the pace of this centralizing and who is helping it along. The tldr; point is that a single service provider should not have the amount of control Amazon does over the Internet. At least that's my take. I know my opinion on this wildly differs from the HN crowd and SV "decision makers" these days - what is so curious to me is that this is a complete 180 from that same demographic even 10 years ago.
- rrggrr 10y agoYes: "why the heck are we centralizing the web to 2 or 3 infrastructure companies". Its a systemic asset, unregulated, in a world where every systemic asset is regulated (eg. utilities, transportation infrastructure, etc.).
- witty_username 10y agoRegulations means you force your standards on other people who might not need or want them. If you want other infrastructure companies or decentralized internet, you are free to do that yourself via voluntary means.
- balls187 10y agoThanks for posting this. I've passed this information through my network. Slack image uploads are hanging.
- jsanroman 10y agoIt's down :(
- headcanon 10y agoStill down for us. S3 seems to be the only thing affected - our mobile apps work fine (EC2 and RDS backend)
- obeattie 10y agoBest thing about incidents like these: post-mortems for systems of this scale are absolutely fascinating. Hopefully they publish one.
- magic_beans 10y agoHave they even acknowledged the mortem..?
- malchow 10y agoEverything is green. There are no tanks in Baghdad center.
- obeattie 10y agoYes. "We've identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue."
- noir_lord 10y ago> We are actively working on remediating the issue. I do love corporate-speak. It's a rapidly oxidising waste receptacle (rather than a dumpster fire).
- officelineback 10y agoAgreed. AWS' postmortems are fascinating.
- exodos 10y agoYah getting the same error in multiple regions as of 1:12 EST
- vpeters25 10y agoApologizes for the "me too" post: It appears to be impacting gotomeeting, I get this error when trying to start a 12pm meeting here: CloudFront is currently experiencing problems with requesting objects from Amazon S3. Edit: ironically, my missed 12pm meeting was an Azure training session.
- tech4all 10y agoYes serious API problems started about 15 minutes ago. Around noon central.
- deleted 10y ago[deleted]
- gamache 10y agoA piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You're signing up for the oldest hardware and the most frequent outages. For legacy customers, it's hard to move regions, but in general, if you have the chance to choose a region other than us-east-1, do that. I had the chance to transition to us-west-2 about 18 months ago and in that time, there have been at least three us-east-1 outages that haven't affected me, counting today's S3 outage. EDIT: ha, joke's on me. I'm starting to see S3 failures as they affect our CDN. Lovely :/
- xbryanx 10y agoI'm getting the same outage in us-west-2 right now.
- ngtvspc 10y agoI can confirm this as well.
- Ph4nt0m 10y agoSame outage in ca-central-1
- STRML 10y agoSeeing it in eu-west-1 as well. Even the dashboard won't load. Shame on AWS for still reporting this as up; what use is a Personal Health Dashboard if it's to AWS's advantage not to report issues?
- STRML 10y agoNow it's in the PHD, backdated to 11:37:00 UTC-6. How could it take an hour to even admit that an issue exists? We have alerts set on this but they're useless when this late.
- gamache 10y agoHuh, I'm not seeing it on my us-west-2 services. Interesting.
- jefe_ 10y agoGetting Issues with Citrix Sharefile api (which I've suspected to run in S3). Seems to only be impacting writes in preliminary assessment.
- jefe_ 10y agoGetting Issues with Citrix Sharefile api (which I've suspected to run in S3). Seems to only be impacting writes in preliminary assessment.
- deleted 10y ago[deleted]
- eggie5 10y agoyes, confirmed.
- fletom 10y agowhat's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page. they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region." increased error rates? really? Amazon, everything is on fire. you are not fooling anyone edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- evtothedev 10y agoThe AWS Status page will lie to you: https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-to-you-4c24a68d8e0a#.1vr972hcx https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-...
- fletom 10y agoI like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.
- brianpgordon 10y agoI thought this was funny so I took a screenshot of the blog post and uploaded it to the company Slack. The upload failed because Slack uses S3. This is getting crazy.
- idlewords 10y ago@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."
- bseabra 10y agoSame here. We are seeing issues.
- afshinmeh 10y agoyeah, looks like Travis CI is down, too: https://www.traviscistatus.com https://www.traviscistatus.com
- freyr 10y agoAnd Trello
- Eyes 10y agoMy website is not down.
- thadjo 10y agosame
- ethanpil 10y agoCorporate language is entertaining while we all pull out our hair. "We are investigating increased error rates for Amazon S3" translates to "We are trying to figure out why our mission critical system for half the internet is completely down for most (including some of our biggest) customers."
- dyeje 10y agoExperiencing issues with Elastic Beanstalk and Cloudfront as well.
- seibelj 10y agoI cannot eb init or deploy to us-east-1
- dfischer 10y agoWow this is a fun one. I almost pooped my pants when I saw all of our elastic beanstalk architecture disappear. It's so relieving to see it's not our fault and the internet feels our pain. We're in this together boys! I'm curious how much $ this will lose today for the economy. :)
- deleted 10y ago[deleted]
- b01t 10y agoyup
- thadjo 10y agoheroku API is down for me
- smmnyc 10y agoThey placed their API into maintenance mode: https://status.heroku.com/incidents/1059 https://status.heroku.com/incidents/1059
- Raphmedia 10y agoSame here in US EAST
- gaia 10y agoSometimes refreshing the console gives this error instead of showing ZERO buckets https://pbs.twimg.com/media/C5xZVGKUYAAXYGj.jpg:large https://pbs.twimg.com/media/C5xZVGKUYAAXYGj.jpg:large
- oaktowner 10y agoApparently app updates on iOS are failing right now, too. Could be related?
- ondrae 10y agoWe're on AWS GovCloud and our S3 is all good. GovCloud is its own region.
- neom 10y agoShame we pay a bazillion dollars for it. Anyway, curious to know how you've thought about mitigating something like this? I worry if it happens to us govcloud users we obviously have much fewer options for redundancy.
- bandrami 10y agoThat sound you hear is every legacy hosting company firing up its marketing machine
- yichi 10y agoI've always wondered why people dismiss dedicated hosting without a second thought. It's actually cheaper than AWS if you factor in all of the performance you get.
- socialentp 10y agoSame here. I can log in to the new S3 console UI, but all of my buckets/resources are missing. Same error as you in the old UI. Also unable to connect through the AWS CLI (says, "An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied"). Fun.
- STRML 10y agoIt's not just us-east-1! They're being extremely dishonest with the green checkmarks. We can't even load the s3 console for other regions. I would post a screenshot, but Imgur is hosed by this too.
- ceejayoz 10y agoIIRC the Console is mostly hosted out of US-EAST-1. Direct API calls to other S3 regions are likely to work, but it's not surprising the Console's having trouble.
- Beacon11 10y agoWorks for me, in us-west-2.
- joshuahaglund 10y agoyou sure? it's not for me.
- austinkurpuis 10y agoSame here. Also having trouble publishing to S3 via CLI and API.
- dbg31415 10y agoYes, appears to be.
- grimmdude 10y agoMaking an already troublesome day worse. Yeehaw
- ianopolous 10y agoI'm seeing the same error on eu-west as well.
- eggie5 10y agois this affecting dockerhub for anyone?
- Globz 10y agoYes Trello is down and they are using S3 :(
- aarondf 10y agoYes
- chadscira 10y agoWell this took out Quay and CircleCI! Hopefully this gets resolved ASAP.
- tomharrisonjr 10y agoWe're seeing queries using Athena against S3 fail in us-east-1
- alfg 10y agoYeah, same here on US-WEST-2. Unable to use the S3 Console, but I can still upload/get content via the API it seems.
- mijustin 10y agoStarted a list of "things to do when S3 is down." https://justinjackson.ca/s3/ https://justinjackson.ca/s3/ What else should I add?
- beanland 10y agoI'm going to go do my laundry.
- mxuribe 10y agoHave a philosophical debate with yourself (just within your mind) as to whether all this interweb/webternet stuff is a worthwhile pursuit...knowing that it would not survive should a comet hit the earth again...or, maybe it will? ;-)
- vinayan3 10y agoYes. Go for a walk and interact with people in the community. You might meet someone interesting and learn something as well!
- contingencies 10y agoMy suggestion would be to rediscover the clarity and focus of thinking about systems and code on paper.
- remx 10y agoMigrate all your stuff to Cloudfront. S3 is not a CDN!
- hobofan 10y agoOr instead of Cloudfront, you could migrate to a service provider that provides meaningful status pages.
- coreywstone 10y agoIt's winter - hit the slopes!
- 10y ago
- ganesharul 10y agoSendgrid, Twilio, Quora is also down. Is this related to S3. Entire world depends on AWS
- leesalminen 10y agoTwilio's outgoing SMS seems to be working fine for me.
- yodon 10y agoSendgrid being down particularly hurts - we need it to notify our users of the problem
- jyriand 10y agoThat's quite ironic. http://isitdownrightnow.com http://isitdownrightnow.com is also down.
- ruchit47 10y agoI have in the middle of thoughts of moving out of AWS and having a dedicated provider as our billing has increased a lot with the scale. The only thing which was holding me was the uptime confidence. Now I feel it's not a bad idea.
- renzy 10y agogetting the same...
- oculusthrift 10y agoYes. and Now my HN profile page is down as well.
- grzm 10y agoI've been getting sporadic "Bad Gateway" errors on HN for the past half hour or so, not apparently associated with any one feature (such as profile pages). My suspicion is that it's unrelated to anything happening at AWS (HN doesn't have any AWS dependencies, does it?), though maybe everyone checking HN to see what the status is has increased load.
- adamveld12 10y agoWhere is that "Show HN" that will let me check if a site is affected by an S3 outtage?
- ryanmarr 10y agoMy ELBS and EB related instances are also down. I can't even get to Elastic Beanstalk or Load Balancers in the web console. Anyone else having this issue?
- JBerryMedX 10y agoYes, we're experiencing this issue as well. Are your ELBs also in us-east-1?
- Taek 10y agoMass outage like this is exactly one of the things we are looking to avoid by building a decentralized storage grid with Sia. Sia are immune to situations like this because data is stored redundantly across dozens of servers around the world that are all running on different, unique configurations. Furthermore, there's no single central point of control on the Sia network. Sia is still under heavy development, but it's future featureset and specifications should be able to fully replace the S3 service (including CDN capabilities). https://sia.tech https://sia.tech
- elbigbad 10y agoNot to push my own product...proceeds to push own product
- deleted 10y ago[deleted]
- thiht 10y ago>Not to push my own product Why would you say that? That's exactly what you're doing...
- deleted 10y ago[deleted]
- jamiesonbecker 10y agoAt first I thought, "meh.." and, seriously, just the globally distributed filesystem alone is incredibly hard to make very reliable, but that's the only critical job: give data back when asked. But then I looked at your site.. looks like bidding on surplus storage on different systems. Great idea, especially if you can ensure that people don't botnet it to death. I'm looking forward to hearing great things from you in the future.
- DevKoala 10y agoWorth a look tbh.
- 10y ago
- caravel 10y agoBut wait. Isn't S3 "the cloud". Everyone promised the cloud would never go down, ever. It has infinite uptime and reliability. Well good thing I have my backups on [some service that happens to also use S3 as a backend].
- ceejayoz 10y ago> Everyone promised the cloud would never go down, ever. No, they didn't. Large portions of AWS's documentation details how you, the developer, are responsible for using their tools to engineer a fault-tolerant, highly available system. Everything goes down. AWS promises varying amounts of nines everywhere, not 100%.
- djhworld 10y agoI know your comment is in jest, but Amazon do say their API SLA for S3 is 99.99% available [1] [1] https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- dragonwriter 10y ago> Isn't S3 "the cloud". Everyone promised the cloud would never go down, ever. S3 is not the cloud, it's one system running in the cloud. The cloud is not down, S3 and services dependent on (and possibly related to) it are. One of the selling points of the cloud is that dynamically provisioned services from multiple providers enable engineering fault tolerant systems that are relatively secure against the failure of any single backend. But, yeah, if you are dependent on one infrastructure vendor's service -- particularly running in one particular region/zone -- you are probably better off than running on a single server for reliability against failures, but you aren't anywhere close to immune to failures. I don't think even cloud vendors have been particularly reluctant to make that point.
- davidcollantes 10y agoAzure is also down. Related?
- Theodores 10y agoWhat, to Yahoo Mail being down? Can't see the connection myself. Even with HN I had the twitter status down page before I reloaded five seconds later, in disbelief... Maybe it is not S3.
- framebit 10y agoWow, amazing to watch stuff go down as this problem ripples out!
- vegasje 10y agoWe're in US-West-2 and our ELBs are dropping 5XXs like there's no tomorrow. This is definitely cascading.
- soccerdave 10y agoWe're in US-West-2 and not seeing any issues. Are the instances behind your ELB trying to access S3 in their application logic?
- vegasje 10y agoNope. No S3 logic behind the scenes. A few of the ELBs are fine, and a few are not. Seems random. The EC2 instances themselves are fine, but the affected ELBs are spitting out 500s.
- twistedpair 10y agoAll our systems are running just fine in us-west-2 right now.
- dageshi 10y agoHuh, I wonder if that's why Origin (EA's Steam competitor) cloud sync just stopped working
- travelton 10y agoI can't get to my Amazon Orders page. "There's a problem displaying some of your orders right now."
- palad1n 10y agoYeah, it looks like that was part of the failure.
- deleted 10y ago[deleted]
- ryanmarr 10y agoMy EB instances and Load Balancers are also down. I can't even get to load balancers in ec2 web console or to elastic beanstalk in web console. It's been almost an hour now.
- rnhmjoj 10y agoWow, S3 is a much bigger single point of failure than I have imagined. Travis CI, Trello, Docker Hub, ... I can't even install packages because the binary cache of NixOS is down. Love living in the cloud.
- booleandilemma 10y agoThank you, HN, for giving me the answer the AWS Service Health Dashboard could not.
- mabramo 10y agoThanks for sharing. I overheard someone on my team say that a production user is having problems with our service. The team checked AWS status, but only took notice of the green checkmarks. Through some dumb luck (and desire to procrastinate a bit), I opened HN and, subsequently, the AWS status page and actually read the US-EAST-1 notification. HN saves the day.
- davewritescode 10y agoSame here, was eating lunch and browsing HN when hipchat started lighting up with customer complaints.
- foxylion 10y agoOn thing I learned here. When something seems horribly wrong, check HN first, it may be "global" problem.
- nodefortytwo 10y agoNot seeing any errors from eu-west-1
- Exuma 10y agoYep, currently have over 20,000 people on site seeing no images. Wonderful
- verelo 10y agoYears ago when we launched our product i decided to use the US-WEST-2 region as our primary region and to build fail over to US-EAST-1 (Anyone here remember the outage of 2011? Yeah, that was why). There is something to be said about not being located in the region where everything gets launched first, and where most the customers are not [imo all the benefits of the product, processes and people, but less risk]. Good luck to everyone impacted by this...crappy day.
- kopy 10y agoLooks like they store the statuses on S3
- Rockastansky 10y agoIs anyone else also seeing 500 errors for cognito on us-east-1?
- kevindong 10y agoIt really is amazing how many web services are dependent on S3. For instance, the Heroku dashboard is currently down for me. Along with all of my services that are on Heroku.
- TravelTechGuy 10y agoSame here, but worse. Some of the apps I have hosted on Heroku (including APIs) are showing "Application Error". Like you, tried logging into dashboard and got a Heroku error page.
- kevindong 10y agoTurns out that not all Heroku dynos are hosted on US-East. One of my friend's dyno is still up and running great.
- mcjiggerlog 10y agoSame here :(. Not sure why serving a connection to my dyno depends on S3 being up...
- kevindong 10y agoI'll bet that what Heroku does is precompile all the code when you push to Heroku (or doesn't), saves it to a S3 bucket, and kills the running process after a certain amount of inactivity. Once it detects a pending network request, the code gets loaded from the S3 bucket into the EC2 instance, and then your code spins up.
- olegkikin 10y agoQuora is down.
- deleted 10y ago[deleted]
- natashabaker 10y agoFor those using Heroku - any workarounds to at least put the app in maintenance mode? It seems their entire platform, including API, is down.
- kolemcrae 10y agoYup. Every single image on my site is hosted there.... eek! :|
- paulddraper 10y agoNo, it's not https://status.aws.amazon.com/rss/s3-us-standard.rss https://status.aws.amazon.com/rss/s3-us-standard.rss
- lancefisher 10y agoLooks like the RSS feed hasn't been updated for an hour and a half.
- deleted 10y ago[deleted]
- Rockastansky 10y agoAnybody else seeing 500 errors with AWS Cognito for us-east-1? They are consistent for me.
- bkanber 10y agoI'm having issues with CloudWatch and related monitoring services; eg auto-scaling groups are unable to scale up or down.
- etse 10y agoAnyone want to share their real experience with their reliability of Google Cloud Storage.
- bandrami 10y agoAnd they've just broken four-9's uptime (53 minutes). They must be pretty busy, since they still haven't bothered to acknowledge a problem publicly...
- simook 10y agoyes it is.
- deleted 10y ago[deleted]
- qaq 10y agoCmon but the cloud is magic and very reliable let's move everything to the cloud
- Exuma 10y agoGreat, all my billing services on Heroku are turned off. Why do they need S3 access for me to access my web dynos? I'd rather my app load but appear broken so I can show my own status rather than just shutting down every single app...
- buildbuildbuild 10y agoSame here. S3 as a point of failure makes zero sense for dyno uptime, I'm very frustrated with Heroku. (the dyno was already running, no need to download a new slug in my opinion)
- janlukacs 10y agoWe're down too with www.paymoapp.com - pretty frustrated that the status page shows everything is up and running.
- k__ 10y ago1. Announce security vulnerability 2. People push updates as fast as possible to fix security 3. No tests, so everything blows up
- chx 10y agohex.pm and docker hub are both failing, a lot of projects can't CI because of these. The house of cards we built.
- benwilber0 10y agoNotice how Amazon.com itself is unaffected. They're a lot smarter than us.
- officelineback 10y agoBrowsing works but some functionality is broken; for example you can't view your order history.
- yichi 10y agoI do recall reading somewhere that Amazon.com isn't actually hosted or fully leveraging on the AWS platform, mostly due to the political struggle between the AWS and the merchant department.
- Raphmedia 10y agoSales Department: "We require 100% uptime! Can you do that?" AWS Department: "Wellll, if we don't change the status to red, it's as if we were up all the time!"
- p0rkbelly 10y agoThere are public talks on Youtube from Amazon.com titled "Drinking our own Champagne" where they say the opposite.
- krakensden 10y agoYeah, that was the original pitch for AWS. Engineering presentations since the initial launch have included "yeah... not quite" admissions though.
- bpicolo 10y agoThe ability is there for any company to take advantage of multiple regions. Takes time and money, but it's doable.
- deleted 10y ago[deleted]
- orn 10y agoI'm trying to reach S3 hosted website, no luck
- j_shi 10y agoIs there a list of all apps/services that rely on S3?
- scottlinux 10y ago- launching AMIs (they are stored on s3) - cloudfront - ses - ebs - rds (snapshots/backups on s3) - lambda functions (appear to be stored in s3) perhaps others
- mierenga 10y agoWould be easier to compile the inverse.
- Trisell 10y agoUS-West(Oregon) just went down as well.
- deboflo 10y agous-west-2 (Oregon) is still up for me from the CLI.
- ahmetcetin 10y agoThe same here
- happyrock 10y agoAnyone doing a region failover? Any issues so far? We are making plans to flip to us-west-1
- whorleater 10y agoWe did a failover to us-west-2 and it seemed to work.
- deleted 10y ago[deleted]
- ahmetcetin 10y agoThe same here still
- garindra 10y agoDockerHub is down as well. DockerHub was down in Oct 2015 because S3 was down in US-EAST. They should have known to cache images in multiple S3 regions since then.
- homakov 10y agoWas just pentesting it, and have some minor result. If you are using S3 browser uploads, make sure parameters you supply to Presign do not contain \n or it can lead to format injection https://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html https://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthenti... Many aws SDK libs don't remove \n for you. (I hope it wasn't me who broke it lol)
- buildbuildbuild 10y ago"Was just pentesting it" ... hopefully with their permission. Be careful.
- homakov 10y agoIt wasnt heavy pentesting, just some params jungling. No way it could cause anything :) still funny coincidence
- malchow 10y ago<% if(service.isUp || true) { renderGreenButton() } %>
- dhairya 10y agoregion-west2 is also down
- zerotolerance 10y agoDo the engineering thing and build fault tolerant systems. Maybe adopt features that have been around since 2015: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/ https://aws.amazon.com/blogs/aws/new-cross-region-replicatio...
- krlkv 10y agoS3 is down? Official Twitter feed is also "unaware" https://twitter.com/awscloud https://twitter.com/awscloud
- philliphaydon 10y agoI'm confused, just logged into work account, and site, and some contract stuff I do. All use S3 / Cloudfront... no errors...
- awsoutage 10y agohttps://twitter.com/sadserver/status/818937064552951809 https://twitter.com/sadserver/status/818937064552951809 Interesting tweet from last month.
- JBerryMedX 10y agoMy company's ELBs in us-east-1 are experiencing massive amounts of latency causing the instances to be marked unhealthy.
- atombender 10y agoIt's interesting to note the cascading effects. For example, I was immediately hit by three problems: * Slack file sharing no longer works, hangs forever (no way to hide the permanently rolling progress bar except quitting) * Github.com file uploads (e.g. dropping files into a Github issue) don't work. * Imgur.com is completely down. * Docker Hub seems to be unavailable. Can't pull/push images.
- Ph4nt0m 10y ago* Hipchat file sharing no longer works, hangs forever * CircleCI cannot access artifacts
- mcphilip 10y agoA doctor's office that's unable to process patients due to the outage: https://mobile.twitter.com/drjincali/status/836657863887998976 https://mobile.twitter.com/drjincali/status/8366578638879989...
- cookiecaper 10y agoI mean, that's not really AWS's problem, is it? Outages happen. If you have a mission-critical service like health care, you really shouldn't write systems with single points of failure like this, especially not systems that depend on something consumer-grade like S3. This appears to be a normal doctor's office where there are routine appointments. Emergencies would be referred to the ER anyway. And while I obviously don't know the details of how his office is run, you'd think that you could get by on a pen-and-paper fallback to manage the office. Maybe that's an advantage to keeping experienced office staff on board.
- mcphilip 10y agoI work in the healthcare industry and there's a big push from AWS into offering HIPAA compliant services for things like patient records. It's becoming much more common to tie in third party services into electronic healthcare software. Obviously no mission critical system should have a single point of failure and doctor's offices should have fallback plans for handling service outages, but most care providers don't have staff onsite with the technical expertise to understand the extent of the coupling. I'm just closely watching this space and found that tweet interesting in relation to the parent comment's remark about realizing the scope of this S3 outage. There's no blame unique to AWS here, but it is becoming an increasingly important piece of plumbing in the industry.
- dgelks 10y agoGetting the same error on the GUI but the aws cli and sdk seem to be working fine (our site is still up too)
- hyperanthony 10y agoExperiencing issues with S3 and ELB for over an hour now.
- tzaman 10y agoIt appears Docker Hub is hosted on S3 as well, none of the official images can be pulled.
- nickstefan12 10y agobets as to the cause? internal DDoS against their dynamo clusters backing s3? DNS issues between amazon's services?
- spacecadets 10y agoThere goes my Trello to do list. Now I'm lost. Oh well.
- knaik94 10y ago"We’re continuing to work to remediate the availability issues for Amazon S3 in US-EAST-1. AWS services and customer applications depending on S3 will continue to experience high error rates as we are actively working to remediate the errors in Amazon S3." Last Update 1:54pmEST It shows up in the event log now too.
- soheil 10y agoI try not to put all my eggs in one basket, that's why for images I use imgur. They have a great API and it's 100% free. There is a handy ruby gem [1] which takes a user uploaded image and sticks it on imgur and returns its URL with dimensions etc. On top of that you don't have to pay for traffic to those assets. [1] https://github.com/soheil/imgur https://github.com/soheil/imgur
- arrty88 10y agoimgur doesn't use S3 behind the scenes?
- BoorishBears 10y agoIt uses it, and it's completely down right now.
- deleted 10y ago[deleted]
- bastawhiz 10y agoUsing imgur over a service that you pay for (with an SLA) is, as my college CS professor used to call it, "skating on thin ice".
- Narretz 10y agoWell, imgur.com is currently down, too. I assume they are still using AWS: http://blog.imgur.com/2013/06/04/tech-tuesday-our-technology-stack/ http://blog.imgur.com/2013/06/04/tech-tuesday-our-technology...
- K2L8M11N2 10y agoImgur's down right now as well (except direct links to images) and it's not lossless (they resave images).
- deleted 10y ago[deleted]
- SubiculumCode 10y ago
- aytekin 10y agoNever depend your business on a single provider.
- maxerickson 10y agoCoincidence? https://twitter.com/homakov/status/836649802842591232 https://twitter.com/homakov/status/836649802842591232 I've been fuzzing S3 parameters last couple hours... And now it's down.
- deleted 10y ago[deleted]
- remx 10y agoPost about S3 not being a CDN hosted on an S3-powered blog: https://jdorfman.posthaven.com/medium-bitcoin-660x493-dot-jpg-cdn-vs-s3 https://jdorfman.posthaven.com/medium-bitcoin-660x493-dot-jp... The irony
- maccard 10y agoMy fire tv stick is totally unusable too. Seems I can't access any applications (even Lodi or Netflix)
- mystcb 10y agoBeen unreliability informed about 1 hour ETA for a fix. fingers crossed
- khamoud 10y agoI think this explains why the docker registry is down as well. http://status.docker.com/ http://status.docker.com/
- sweddle 10y agoyeah still all green in AWS status.... maybe their red and yellow icons are kept on S3. :-)))
- camperman 10y agoMany a true word is spoken in jest - that's exactly what the problem is :)
- BrandonM 10y agoAdditionally, Zendesk is apparently failing to process new tickets, so our users can't report the errors they're encountering.
- jliptzin 10y agoThank god I checked HN. I was driving myself crazy last half hour debugging a change to S3 uploads that I JUST pushed to production. Reminds me of the time my dad had an electrician come to work on something minor in his house. Suddenly power went out to the whole house, electrician couldn't figure out why for hours. Finally they realized this was the big east coast blackout!
- stevehawk 10y agoirc.freenode.net / ##aws (must be registered with nickserv to join) outage first reported around 11:35CST.
- Havoc 10y agoDisadvantage of being in the detail I guess. My thinking was Imgur seems broken today >>> Something major on the intertubes must be fk'd.
- TeMPOraL 10y agoPrecisely how I discovered it. Imgur down. Imgur is almost like a piece of critical Internet infrastructure. That + some other site misbehaving tipped me off that something very wrong is happening...
- deleted 10y ago[deleted]
- skiril 10y agothey already admitted it: https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_goes_haywire/ https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...
- draw_down 10y ago"bit-barn bods" is some of the worst alliteration I've ever seen.
- edcoffin 10y agoIf you've ever felt the AWS health dashboard was dubious before now...
- dorianm 10y agoHeroku API/Dashboard is down, Bugsnag is down, etc.
- deleted 10y ago[deleted]
- mrep 10y agoquite ironic that 'isitdown.com' is also down
- dorianm 10y agoHeroku API/Dashboard is down, Bugsnag is down, etc.
- soheil 10y agowow even services like Intercom are affected, I can't see who is on my website right now.
- machinarium 10y agoOmg I wish I googled this earlier. Wasted hours debugging :(
- SnowingXIV 10y agoYep. I was wondering why my heroku deploys were hanging so I was looking into every possible issue on my end. And then I see the news.
- manmal 10y agoOur static site hosted on eu-central-1 is still up: http://www.creativepragmatics.com.s3-website.eu-central-1.amazonaws.com http://www.creativepragmatics.com.s3-website.eu-central-1.am...
- thepumpkin1979 10y agois it just us-east-1? could it be prevented by using a different region?
- JustinAiken 10y agoFor our app, both S3 and SES have been completely down in us-east-1 for hours now.
- SubiculumCode 10y agoMr. Robot live shoot? :) slack file services down too
- shifted316 10y agoThe status page is stored in s3. It can't be updated. The page you see is cached in cloudfront. They are working on updating the status page.
- devenrl 10y agoSorry, my simplistic mind is only thinking this right now: http://alessandrobender.com.br/wp-content/uploads/2015/07/fix.jpg http://alessandrobender.com.br/wp-content/uploads/2015/07/fi...
- deleted 10y ago[deleted]
- devenrl 10y agoSorry, my simplistic mind is only thinking this right now: http://alessandrobender.com.br/wp-content/uploads/2015/07/fix.jpg http://alessandrobender.com.br/wp-content/uploads/2015/07/fi...
- ignaces 10y agoHeroku apps are also down because of this!
- phildougherty 10y agoI wrote a quick post discussing this outage. I figured I should share here https://blog.containership.io/aws-got-you-down https://blog.containership.io/aws-got-you-down
- valine 10y agoApple's iCloud is having issues too, probably stemming from AWS. Ironically Apple's status page has been updated to reflect the issue while Amazon's page still shows all green. https://www.apple.com/support/systemstatus/ https://www.apple.com/support/systemstatus/
- brandon272 10y agoI can't stream music from my iCloud library.
- deleted 10y ago[deleted]
- chiph 10y agoI'm seeing problems with Kindle downloads.
- geerlingguy 10y agoFrom Amazon: https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656 The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates. So it's not just a joke... S3 being down actually breaks its own status page!
- etler 10y agoFor this kind of page it might be best for them to use a data URI image to remove as many external resources as possible.
- Symbiote 10y agoUnicode characters would work fine, and be even smaller. Warning sign, octagonal sign, no Entry (all filtered by HN). There are plenty of possibilities.
- JangoSteve 10y agoI was thinking they should host the little green check mark icons on s3.
- DocK 10y agoEven Kindle books aren't able to be served; download attempts hang.
- ayemeng 10y agoFunny, status page is incorrect because of S3 https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656
- edgartaor 10y agoQUESTION. There could be data lost from this failure?
- LeonM 10y agoJust posted on their Twitter: "The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."
- dangle 10y agoAWS is updating twitter here. No red icons on the status page IS an AWS issue: https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656
- rebornix 10y agoAlexa smart home component stopped working, if you try to reinstall the Alexa app on your phone, you'll find that you can't even login anymore.
- eggie5 10y agoall of your jokes about the dashboard not turning red b/c the icon is hosted on US EAST are true: Amazon Web ServicesVerified account @awscloud 8m8 minutes ago More The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.
- koolba 10y agoI bet the outage is related to the new color coded CloudWatch metrics: https://twitter.com/awscloud/status/836630468778864640 https://twitter.com/awscloud/status/836630468778864640 As part of the release they wanted to make sure everybody gets a chance to see "red" metrics.
- jsperson 10y agoJust finished reading The Everything Store... I bet a "?" email went out.
- ajmarsh 10y agovia AWS twitter account "The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."
- axg 10y agoAmazon: The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates. https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656
- DenisM 10y agoNow might be a good time to ponder a lasting solution. Clearly, we cannot trust AWS, or any other single provider, to stay up. What is the shortest, quickest to implement, path to actual high availability? You would have to host your own software which can also fail, but then at least you could do something about it. For example, you could avoid changing things during critical times of your own business (e.g. a tradeshow), which is something no standard provider could do. You could also dial down consistency for the sake of availability, e.g. keep a lot of copies around even if some of them are often stale - more often than not this would work well enough for images.
- alexbilbie 10y agoHigh availability is improved by hosting in multiple AWS regions. S3 offers alternative region replication functionality and you can use Cloudfront of another CDN to load balance between buckets
- Cshelton 10y agoYup, all of our fail overs worked flawlessly. You wouldn't even be able to tell if you didn't know.
- DenisM 10y agoBut do you always serve files from S3? Wasn't the entire S3 down today? Or was it just some regions? I couldn't even connect to s3.amazonaws.com ...
- bendbro 10y agoOnly one region, us-east.
- deleted 10y ago[deleted]
- temuze 10y agoHow can you use Cloudfront to point to multiple buckets? I'm pretty sure each origin needs a path and Cloudfront serves the first path that matches.
- deleted 10y ago[deleted]
- myth_drannon 10y agoLooks like SoundCloud is hosting the tracks on S3 , can't program without my music...
- yuxt 10y agothis still works for me https://musicforprogramming.net https://musicforprogramming.net
- bas 10y ago"Amazon CloudFront: Service is operating normally" This is bullshit if you're using an S3 origin in your distribution.
- jotaen 10y agoDoes anyone have trouble with the Cloud Console? The JS assets for the CloudFront dashboards seem broken, so unfortunately it’s not possible to change the behaviours of the Distributions (e.g. to point them to another bucket)
- murphy52 10y agoTierPoint, a large hosting service, is reporting a massive DDOS attack on their infrastructure.
- ignaces 10y agosource?
- xtus 10y agoAfter few requests timed out, started to dig a bit. The CNAME for a bucket endpoint was pointing to s3-1-w.amazonaws.com with a TTL of at least an other 5600 secods. Doing a full trace was giving back a new s3-3-w.amazonaws.com The IP related to s3-1-w was/is timing out, all cool instead for the s3-3-w.
- skryshtafovych 10y agoIts also affecting http://status.fabric.io/ http://status.fabric.io/ cant build Android apps or update beta builds
- Animats 10y agoAmazon outage just reported on NBC News.[1] AMZN stock down $3.45 (0.41%). [1] http://www.nbcchicago.com/news/national-international/Amazons-Web-Services-Down-Causes-Massive-Outages-Online-415003823.html http://www.nbcchicago.com/news/national-international/Amazon...
- Animats 10y agoJust reported on USA Today.[1] [1] http://www.usatoday.com/story/tech/news/2017/02/28/amazons-cloud-service-goes-down-sites-scramble/98530914/ http://www.usatoday.com/story/tech/news/2017/02/28/amazons-c...
- Animats 10y agoFox News now has the story.[1] [1] http://www.fox5ny.com/news/238689310-story http://www.fox5ny.com/news/238689310-story
- Animats 10y agoLondon Daily Express [1], CBS[2] now reporting the outage. [1] http://www.express.co.uk/life-style/science-technology/773320/amazon-down-web-hosting-services-not-working http://www.express.co.uk/life-style/science-technology/77332... [2] http://losangeles.cbslocal.com/2017/02/28/amazon-web-services-disrupted-outages-reported-across-multiple-sites/ http://losangeles.cbslocal.com/2017/02/28/amazon-web-service...
- Animats 10y agoReuters, Associated Press, and The Hill now reporting the outage. "http://www.isitdownrightnow.com/" http://www.isitdownrightnow.com/" and DownDetector are down.
- BWStearns 10y agoIt's been down that level from before this started happening. Surprised this outage hasn't moved it yet.
- fjabre 10y agoI never understood why so many devs flocked to AWS. I actually find their abstraction of services gets in the way and slows down my dev instead of making it easier like so many devs claim it does. I prefer Linode.
- Sanddancer 10y agoIt lets them pretend that ops isn't a skillset you need people to specialize in. Just throw more devs and servers at the problem instead of building a good infrastructure.
- robxu9 10y agoNew update: "Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue."
- fletom 10y agoare you openly admitting that the AWS service status page runs on AWS? because that is far more embarrassing than this downtime ever could be
- flavor8 10y ago> Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue. "Believe" is not inspiring.
- nlightcho 10y agohttps://twitter.com/awscloud/status/836630468778864640 https://twitter.com/awscloud/status/836630468778864640 At least now we can see all the network failures in full RGB.
- AtheistOfFail 10y agoWe have a red error, finally! Source: https://status.aws.amazon.com/ https://status.aws.amazon.com/ After two hours, they have finally updated their dashboard.
- c4urself 10y ago> We have now repaired the ability to update the service health dashboard -- AWS Status Well that explains all the green checkmarks /s
- oneeyedpigeon 10y agoFinally! The status page admits something's up.
- contingencies 10y agoWhat a shame they took down MegaUpload! Clearly we need greater competition in the wholly-owned-infrastructure, file-hosting-as-a-service space.
- rabidonrails 10y agoUpdated: Amazon Elastic Compute Cloud (N. Virginia) Increased Error Rates less 11:38 AM PST We can confirm increased error rates for the EC2 and EBS APIs and failures for launches of new EC2 instances in the US-EAST-1 Region. We are also experiencing degraded performance of some EBS Volumes in the Region. Amazon Elastic Load Balancing (N. Virginia) Increased Error Rates more Amazon Relational Database Service (N. Virginia) Increased Error Rates more Amazon Simple Storage Service (US Standard) Increased Error Rates more Auto Scaling (N. Virginia) Increased Error Rates more AWS Lambda (N. Virginia) Increased Error Rates more
- Svenskunganka 10y agoThis can't be only the US-EAST-1 region. I'm a european resident and most things are down for me too.
- samat 10y agoI've checked eu-west-1 and it works fine for both reads and writes.
- Svenskunganka 10y agoI'm not an AWS customer, but if it is comparable to Google Cloud's Multi-regional Storage it should be geo-redundant. Doesn't S3 replicate the data across regions, so in case a region goes offline it won't affect the service?
- nicpottier 10y agoSES seems to be downf or us as well.
- nicpottier 10y agoSES seems to be down for us as well.
- gmisra 10y agoFYI to S3 customers, per the SLA, most of us are eligible for a 10% credit for this billing period. But the burden is on the customer to provide incident logs and file a support ticket requesting said credit (it must be really challenging to programmatically identify outage coverage across customers /s) https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- machbio 10y agothats for below 99.9% - they are at 99.997% .. you are never getting that 10% credit..
- christop 10y ago0.1% of 28 days is 40 minutes, so it seems likely to happen.
- machbio 10y agoI was calculating it for a year - maybe the availability applies to per billing cycle - you may be correct..
- joatmon-snoo 10y agoYou got your orders of magnitude wrong ;) 99.9964583 = 100 - 153/(30*24*60) 99.6458333 = 100 * (1 - 153/(30*24*60))
- machbio 10y agoI was calculating it for a year - maybe the availability applies to per billing cycle.. I am still not able to understand your math - mind explaining
- joatmon-snoo 10y agoYour numbers are still off for a year. 99.9997089 = 100 - 153/(365*24*60) 99.9708904 = 100 - 100*153/(365*24*60) The formula is 100% minus 100% times downtime/time in month/year. 153 is the number of minutes they were down going off the reported updates at https://status.aws.amazon.com/ https://status.aws.amazon.com/ - 11:35AM PST was when they fixed the status page, 2:08PM PST was when S3 was fully back online. (And 153 is underestimating it, because there were errors going on for long before they fixed the status page, but I don't have timestamps on that.)
- nicpottier 10y agoSES seems to be down for us as well in Virginia. Of course nothing on the status page.
- rrecuero 10y agoIs anybody else having trouble loading http://platform.twitter.com/widgets.js http://platform.twitter.com/widgets.js? It is probably hosted on S3 I assume
- fernandopj 10y agoUpdate[1]: AWS Status dashboard now showing icons other than green. https://status.aws.amazon.com/ https://status.aws.amazon.com/ [1] https://twitter.com/awscloud/status/836662601090134017 https://twitter.com/awscloud/status/836662601090134017
- joatmon-snoo 10y agoAccording to the personal health dashboard, they've root-caused the S3 outage and are working to restore. In the meantime, EC2, ELB, RDS, Lambda, and autoscaling have all been confirmed to be experiencing issues.
- joatmon-snoo 10y agoAccording to the personal health dashboard, they've root-caused the S3 outage and are working to restore. In the meantime, EC2, ELB, RDS, Lambda, and autoscaling have all been confirmed to be experiencing issues.
- joatmon-snoo 10y agoAccording to the personal health dashboard, they've root-caused the S3 outage and are working to restore. In the meantime, EC2, ELB, RDS, Lambda, and autoscaling have all been confirmed to be experiencing issues.
- KurtMueller 10y agoYou can always check by going to www.isitdownrightnow.com/ Oh wait. The site sits on S3. Never mind.
- magic_beans 10y agoDashboard has been updated, finally!
- mpetrovich 10y agoUpdate: AWS dashboard has been fixed and is now showing outages https://status.aws.amazon.com/ https://status.aws.amazon.com/
- l0c0b0x 10y agoGoogle DNS 8.8.8.8 was (for the first time that I've noticed) spotty about 30 minutes ago. Something big is happening: http://map.norsecorp.com/#/ http://map.norsecorp.com/#/
- gcoguiec 10y ago> We have now repaired the ability to update the service health dashboard. It seems their status page is hosted ... as a S3 static website.
- tudorconstantin 10y ago"We have now repaired the ability to update the service health dashboard. " - full of yellow red icons now indeed https://status.aws.amazon.com/ https://status.aws.amazon.com/
- ttttytjj 10y agoIt's fixed... I mean the status page https://status.aws.amazon.com/ https://status.aws.amazon.com/
- robineyre 10y agoHi all. I came across this forum on Google. I have the same error - and it's all a bit beyond me. I'm not a techie or coder but set up Amazon S3 several months ago to backup my websites and it generally works fine - and has saved my bacon on a couple of occasions. (Also back up in Google Drive.) As someone who's really only a yellow belt (assuming you're all black belts!), just so I understand ('cos I'm cacking myself!) ... I'm seeing the same issue. Does this mean there's a problem with Amazon? I can't access either of my S3 accounts even if I change the region, and I'm concerned it may be something I've done wrong, and deleted the whole lot. It was working yesterday!!! Would be massively grateful for a heads up. Thanks in advance.
- z4chj 10y agoYes this is an issue with Amazon and there is little you can do besides wait until it has been resolved
- leesalminen 10y agoNow at the top of Drudge http://drudgereport.com/ http://drudgereport.com/
- artur_makly 10y agohttps://twitter.com/xiaodown/status/836656364965371904 https://twitter.com/xiaodown/status/836656364965371904 https://twitter.com/ArturMakly/status/836665379233628161 https://twitter.com/ArturMakly/status/836665379233628161
- samat 10y agoOne of a really rare times when it's good to be in Europe (s3 works here).
- deleted 10y ago[deleted]
- manshoor 10y agofinally status are updated https://goo.gl/wCINaC https://goo.gl/wCINaC
- redm 10y agoIt looks like the S3 outage is spreading to other systems or the root cause of the S3 problem is affecting different services. There are at least 20 services listed now. [1] [1]: http://status.aws.amazon.com/ http://status.aws.amazon.com/
- simplehuman 10y agoIt's still down. All morning! So much business lost.
- mixedbit 10y ago'Increased Error Rates' is a bit harsh, couldn't they call it 'Sub-prime Success Rates'?
- deleted 10y ago[deleted]
- FussBudget86 10y agoYou think this is bad? Just look at what's happening in Sweden...
- kardashev 10y agoYou'll remember me when the west wind moves Upon the fields of barley You'll forget the sun in his jealous sky As we walk in fields of green
- oshoma 10y agoThe status page shows a lot of yellow and red now. From http://status.aws.amazon.com/ http://status.aws.amazon.com/ Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.
- mmansoor78 10y agoPer AWS : For S3, we believe we understand root cause and are working hard at repairing. Future updates across all services will be on dashboard. https://twitter.com/awscloud/status/836666548311859200 https://twitter.com/awscloud/status/836666548311859200
- frik 10y agoIncreased Error Rates Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue. Amazon hosted their status page on their failing service, ouch. Now they fixed the status page, after more than one hour. The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates. https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656
- jpwgarrison 10y agoI am having trouble sending attachments in the Signal app - seems unlikely, but could this be related? [edit- looks like they do have a pretty heavy reliance on S3, per https://github.com/WhisperSystems/Signal-Server/blob/master/config/sample.yml https://github.com/WhisperSystems/Signal-Server/blob/master/... and various other sources.]
- mayneack 10y agohttps://twitter.com/whispersystems/status/836651250842124288 https://twitter.com/whispersystems/status/836651250842124288
- Globz 10y agoMy Atom keep crashing and the log says it can't resolve : https://atom-installer.github.com/ https://atom-installer.github.com/ is there a part of this hosted on S3? I cannot open Atom anymore, it keep crashing on the check for updates screen...
- beeftime 10y agohttp://downforeveryoneorjustme.com/isitdownrightnow.com http://downforeveryoneorjustme.com/isitdownrightnow.com RIP
- cdnsteve 10y agoLook like the dashboard has been updated to no longer use S3: AWS is having a major meltdown right now http://status.aws.amazon.com/#ecr-us-east-1_1488312155 http://status.aws.amazon.com/#ecr-us-east-1_1488312155
- soheil 10y agoIt doesn't look that bad, think about it S3 is such a critical part of almost any web application, it is treated like a realtime micro-service. So looks like most of the Internet in the U.S. is affected but nevertheless no one is dead yet and the world has not ended. So even if hypothetically let's say China attacked us using cyber-warfare it wouldn't be so bad after all... This was kind of like a test.
- thomassharoon 10y agoIs S3 down outside of Us-East too? I can't seem to create a bucket in US-West or EU
- officelineback 10y agoOther regions still work, but the web console relies on us-east-1 so you should use the API to create new buckets until the issue is resolved.
- rajangdavis 10y agoHate to ask, but does anybody now of an alternative storage solution? Also, anyone have any alternative to Heroku for now?
- moreisee 10y agohttps://cloud.google.com/storage/ https://cloud.google.com/storage/
- wsh91 10y agoGoogle Cloud Storage (as mentioned) for storage, Google App Engine for PaaS. :) (I work on Cloud, specifically Datastore.)
- rajangdavis 10y agoThanks for the recommendation; is there a way to do back ups? For example, I have images and various assets stored on S3; would there be a way to change the storage provider on the fly on a website? The other case is could I have apps hosted on Heroku and set up a service to duplicate the app code and database over to Google for redundancy? This isn't super critical as the apps are not customer focused, but they generate content that is customer focused.
- mmaunder 10y agoSo much has broken thanks to this. Web apps, slack uploads, parts of Freshdesk etc. I don't love you right now AWS. https://status.aws.amazon.com/ https://status.aws.amazon.com/
- aabajian 10y agoLeap day bug?
- kyled 10y agoAnd yet people think I'm crazy for wanting to wrap get time functions so code can be tested...
- deleted 10y ago[deleted]
- vanpupi 10y agoAny opinions you can post on http://wp.me/p7HKNy-5h http://wp.me/p7HKNy-5h as well
- Exuma 10y agoNo
- vanpupi 10y agoPost your opinion on http://wp.me/p7HKNy-5h http://wp.me/p7HKNy-5h
- the_arun 10y agoQuora is down too. Getting 504. Gateway Timeout. Is it related to S3??
- bdcravens 10y agoI was listening to sessions from AWS Re:invent last night. What jumped out at me was the claim of 11 9's for S3. How many of those 9's have they blown through with this outage?
- ta_wh 10y agoThat's a durability target, not an availability SLA. Durability != Availability.
- bdcravens 10y agoThat makes more sense (since I was listening to it in the context of a conference session, I'm not sure if I heard a distinctive term being used, though I'm sure they're careful about the language they use)
- wintermute-_- 10y agoThat's for data retention, S3 only guarantees three 9's availability.
- Fej 10y agoOkay, it's been a few hours and this is starting to get ridiculous. When was the last time that we had a core infrastructure outage this major, that lasted for this long?
- deleted 10y ago[deleted]
- samaysharma 10y agoFrom https://status.aws.amazon.com/ https://status.aws.amazon.com/: "Update at 12:52 AM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour." (I think the AM means PM)
- boulos 10y agoAnd now: > Update at 1:12 PM PST: S3 object retrieval, listing and deletion are fully recovered now. We are still working to recover normal operations for adding new objects to S3.
- uranian 10y agoHeroku seems to suffer from this too
- AzzieElbab 10y agoNetflix is up. Enjoy
- Animats 10y agoAWS is claiming that Simple Storage (US Standard) is starting to come back up as of 12:54 PM PST.
- cyberferret 10y agoWell, at least our decision to split services has paid off. All of our web app infrastructure is on AWS, which is currently down, but our status page [0] is on Digital Ocean, so at least our customers can go see that we are down! A pyrrhic victory... ;) [0] - http://status.hrpartner.io http://status.hrpartner.io EDIT UPDATE: Well, I spoke too soon - even our status page is down now, but not sure if that is linked to the AWS issues, or simply the HN "hug of death" from this post! :) EDIT UPDATE 2: Aaaaand, back up again. I think it just got a little hammered from HN traffic.
- insomniacity 10y agoHTTP 500 :(
- crack-the-code 10y agoPlot twist: Digital Ocean is secretly hosted on AWS
- neom 10y agoshhhhhhhhhhh!!!
- cyberferret 10y agoLOL...I just got notice that our status server is down now too! Maybe DO is just a rebranded offshoot of AWS after all... :D
- AtheistOfFail 10y agoCould be worse, your entire infrastructure could be hosted on Heroku. You don't use S3 but because they do, your entire infrastructure crumbles.
- lambdasquirrel 10y agoI don't see why this is being downvoted. It's a pretty legitimate concern.
- la6470 10y agoback up
- afshinmeh 10y agoHeroku as well: https://status.heroku.com/ https://status.heroku.com/
- linsomniac 10y agoWe are starting to see recoveries, our SES emails have mostly gone out and our data synchronization has updated 2 of our 3 feeds. Amazon has posted a message that they expect "improved error rates" in the next 45 minutes.
- outericky 10y agoStatusPage.io survived. Thanks gents.
- sk2code 10y agoNow what kind of business choose to remain down for 2 hours plus during the peak business hours? Seems cloud computing still has a lot to learn.
- shiven 10y agoStatus page is lit up like a Christmas tree! Looks like AWS finally found the myriad non-green icons.
- devy 10y agoSo S3's been down for at least 3 hours. Does AWS break this year's S3 durability & reliability promise of eleven 9s by now? [1][2] [1]: https://aws.amazon.com/s3/details/ https://aws.amazon.com/s3/details/ [2]: https://en.wikipedia.org/wiki/High_availability#Percentage_calculation https://en.wikipedia.org/wiki/High_availability#Percentage_c...
- noir_lord 10y ago11 9's is durability not availability.
- dcosson 10y agoThat's durability (data loss) not availability. Here's [1] their official SLA. This outage so far brings them to less than 3 nines of uptime this month (43.8 minutes) but still more than 2 nines (7.2 hours) so it sounds like everyone gets 10% off their S3 bill. Very curious if Amazon will apply this automatically or only if you complain. Edit: from further down the same page, it looks like only if you write in to support do you get these broken SLA credits. Kind of lame since everything else about their billing is so precise and automatic. [1] https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- kseifried 10y agoI have gotten several credits from AWS for under a dollar due to incorrect billing calculations on their end that they caught, notified me about and sent a credit for. So I will assume yes, they do this automatically (if they don't I'd be quite surprised).
- kccqzy 10y agoI don't think this has anything to do with durability & reliability (they probably haven't lost data). It's about availability.
- melor 10y agoOnly limited impact to Aiven services due to service migration capability http://help.aiven.io/announcements/aiven-customer-notice-aws-us-east-1-outage-on-2017-02-28 http://help.aiven.io/announcements/aiven-customer-notice-aws...
- vacri 10y agoSo this is particularly weird - one of my instances was showing 0% CPU in CloudWatch (dropped from 60% at the start of the event), but the logs were saying 'load 500'. I ssh'd in... and the problem resolved itself. The only thing I did was run htop to look at the load, and it dropped from 500 (reported in htop) to it's normal level. Just ssh'ing in fixed that issue.
- benevol 10y agoHere we go again: Technology leads to technology (and wealth) monopolies, in other words: more centralization. Which has always been bad. Just like with Cloudflare leaking highly sensitive data all over the Internet, a couple of days ago.
- jasonl99 10y agoI can't download purchased MP3's from amazon's own site, I get "We’re experiencing a problem with your music download. Please try downloading from Your Orders or contact us." When I go to my orders I get "There's a problem displaying some of your orders right now. If you don't see the order you're looking for, try refreshing this page, or click "View order details" for that order." It seems that Amazon is eating its own dog food.
- francesco1975 10y agoYes it is down https://status.aws.amazon.com/ https://status.aws.amazon.com/
- francesco1975 10y agoYes it is down https://status.aws.amazon.com/ https://status.aws.amazon.com/ Half internet is down the data center in Virginia the one with the cloud is totally dead apparently. Enjoy the cloud bullshit :)
- splatcollision 10y agoI just spent the last hour trying to figure out why in the hell I can't update the function code on a lambda instance. Next time I will remember to check HN first!
- boulos 10y agoDisclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to storage.googleapis.com), and it works like a charm. Resumable uploads work differently between us, but honestly since we let you do up to 5TB per object, I haven't needed to bother yet. Again, Disclosure: I work on Google Cloud (and we've had our own outages!).
- masterleep 10y agoCompetition is great for consumers!
- JPKab 10y agoI've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support) It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.
- espeed 10y agoMe too. We switched to Google Cloud years ago at its inception and have never looked back -- always viewed it as a competitive advantage due to its solid, more advanced infrastructure -- faster network, reliable disks, cleaner UI that's easier to manage. Just a cleaner operation all the way around.
- eknkc 10y agoWe were on GCP for around a year, it was my decision I really wanted to love GCP and I initially did. But we recently switched to AWS. I think there is little GCP does better than AWS. Pricing is better on paper, but performance per buck seems to be on par. Stability is a lot worse on GCP, and I don't just mean service outages like this one (which they had their fair share) but also individual issues like instances slowing down or network acting up randomly. Also lack of service offerings like no PostgreSQL, functions never leaving alpha, no hosted redis clusters etc... Support is also too expensive compared to AWS. Management interfaces are better on GCP and sustained use discount is a big step up against AWS reservations. Otherwise, I think AWS works better.
- carimura 10y agoWe're seeing recovery across our services now.
- jedicoder107 10y agoStatus Pages (Services & Products affected by S3 outage) - https://status.heroku.com/ https://status.heroku.com/ - https://status.aws.amazon.com/ https://status.aws.amazon.com/ - https://medium.statuspage.io/ https://medium.statuspage.io/ - https://status.slack.com/ https://status.slack.com/ - http://status.filestack.com/ http://status.filestack.com/ - http://www.trellostatus.com/ http://www.trellostatus.com/ - https://health.autodesk.com/ https://health.autodesk.com/ - http://status.ifttt.com/ http://status.ifttt.com/ - http://status.imgur.com/ http://status.imgur.com/ - http://status.docker.com/ http://status.docker.com/
- tyingq 10y agoPretty good list of other affected sites/services in this article: http://venturebeat.com/2017/02/28/aws-is-investigating-s3-issues-affecting-quora-slack-trello/ http://venturebeat.com/2017/02/28/aws-is-investigating-s3-is... Some big names and services popular with HN mentioned there. Quora, AirBnb, SendGrid, Downdetector(heh).
- ocdtrekkie 10y agoI like how you know this comment is in poor taste, and posted it anyways.
- treehau5 10y ago"you might find the comment poor in taste" != "they knew it was poor in taste"
- markcerqueira 10y ago"I'm sorry if you're offended..." :)
- TeMPOraL 10y agoWell, getting offended by something that isn't a direct, personal attack is a sign of emotional immaturity.
- boulos 10y agoRight, it's more of a grey area. It's clearly a bad day for all involved, but there are enough sub-threads here about "What can I move to" that I felt a top-level comment about how GCS has worked hard to continue S3 interop support was warranted (it's clearly not widely known).
- ocdtrekkie 10y agoI would argue the very notion that he thought the disclaimer might be necessary meant he full well knew he was posting in poor taste. And as he states, he knows they have outages too. So it's like if there were two towns in tornado alley, and one town got hit by a tornado, and the other was like "well, it might be in poor taste, but there isn't currently a tornado in our town". If Google had some higher ground to stand on when they made their marketing posts, maybe they'd have merit. But when they're just pushing product that's no better than the product they're commenting about, it's just spam. If he said "we have 30% less outages than AWS" or something, at least there'd be merit to posting it.
- 10y ago
- tjpaudio 10y agoInterestingly, I placed an order on amazon.com and while the order appears when I look at my account, none of the usual automated emails have come. I wonder how deeply this is effecting their retail customers.
- netvisao 10y agoLooks like our dashboard is still sustaining it https://acedashboard.cbp.dhs.gov/ https://acedashboard.cbp.dhs.gov/
- poofyleek 10y agoThis is truly serverless computing at work.
- newman314 10y agoThere is no cloud, there is only someone else's computer. :(
- Animats 10y agoNegative comment on all this in Forbes.[1] Too much centralization. CEOs read that. [1] https://www.forbes.com/sites/ryanwhitwam/2017/02/28/amazon-s3-outage-has-broken-a-large-chunk-of-the-internet/#4ec46414c467 https://www.forbes.com/sites/ryanwhitwam/2017/02/28/amazon-s...
- Animats 10y ago"Inc." is quoting comments from here.[1] [1] http://www.inc.com/sonya-mann/amazon-web-services-outage.html http://www.inc.com/sonya-mann/amazon-web-services-outage.htm...
- nvarsj 10y agoeu-west-1 is doing great. Obviously European ops are superior to their US counterparts.
- cperciva 10y agoS3 is currently (22:00 UTC) back up. The timeline, as observed by Tarsnap: First InternalError response from S3: 17:37:29 Last successful request: 17:37:32 S3 switches from 100% InternalError responses to 503 responses: 17:37:56 S3 switches from 503 responses back to InternalError responses: 20:34:36 First successful request: 20:35:50 Most GET requests succeeding: ~21:03 Most PUT requests succeeding: ~21:52
- thenewregiment2 10y agono. soundcloud uses aws s3. it is still down. this is false information.
- ta_wh 10y ago"[RESOLVED] Increased Error Rates Update at 2:08 PM PST: As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally." https://status.aws.amazon.com/ https://status.aws.amazon.com/
- thenewregiment2 10y agooh the famous downvote for the smear campaign. just admit it.
- ta_wh 10y agoThink you're mistaken, I don't have downvote privileges!
- joatmon-snoo 10y agoYou're getting downvotes because you don't understand that B being down is not an effective indicator of the status of A, even if B depends on A.
- espeed 10y agoClaiming a statement is false when it's demonstrably true is something that will likely get downvoted every time. It's misleading to others and fills the board with noise.
- reiichiroh 10y agoAny truth to this being a DOS by some kiddies named Phantom Squad?
- thenewregiment2 10y agosoundcloud uses aws s3. it is still down.
- gopalakrishnans 10y agoJust in case if anybody needs :) https://www.microsoft.com/developerblog/real-life-code/2016/05/23/S3-Proxy.html https://www.microsoft.com/developerblog/real-life-code/2016/...
- mcheshier 10y agoIf I listen closely, I think I can hear the pagers going off in South Lake Union from Downtown Seattle.
- cdevs 10y agoI think there was some fontawesome loading issues related to this, I also noticed a site trying to load twitter messages but couldn't Get the JavaScript loaded during that time today.
- metafunctor 10y agoBased on reports from the field, it looks like S3 was down for about three hours for most of their customers. S3 promises four nines of availability (11 nines of durability), so today we got about 3-4 years worth of downtime in one fell swoop. Oops.
- machbio 10y agodurability is different from availability
- IAmGraydon 10y agoWhich is why he made the distinction between the two.
- mrep 10y agoIt is 3 nines [1] for availability and .999*365/12/24=1.266 hours per month so it is actually 3-4 times their monthly SLA depending on your get/put requests (gets came up sooner than puts but that is not factored into the SLA AFAIK). [1]: https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/
- cowkingdeluxe 10y agoWhere do you see four nines for S3 SLA? https://aws.amazon.com/s3/sla/ https://aws.amazon.com/s3/sla/ shows 99.9%
- metafunctor 10y agoIt was from memory. I distinctly remember it being advertised as "four nines". Perhaps they've adjusted their marketing. The S3 FAQ still says [1]: "S3 Standard is designed for 99.99% availability". [1]: https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/
- sonnyhe2002 10y agoIs it down again?
- sonnyhe2002 10y agoIs it down again?
- nodesocket 10y agoMeanwhile engineers across the globe scramble to fix outages due to AWS s3, $AMZN is unaffected on the stock market. Just shows the disconnect between emotions and reality. https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&chvs=Linear&chdeh=0&chfdeh=0&chdet=1488324845083&chddm=391&chls=IntervalBasedLine&q=NASDAQ:AMZN&ntsp=0&ei=6Ai2WOnrMsax2AaQ1rvwDg https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&...
- IAmGraydon 10y agoWhy would this affect Amazon's stock? Amazon generates $136B per year. AWS comprises $8B of that. Even if 5% of customers left because of this (and that's never going to happen), it would barely create as much as a tiny ripple in their ocean of revenue. Investors care not about Amazon's cloud services.
- ec109685 10y agoAWS's profit helps fund the rest of their business.
- ianamartin 10y agoThis is why it's important to write code that doesn't depend on only a single service provider. S3 is great. But it's better to set up a Riak cluster on AWS than to actually use S3, if you can. The only services my team uses directly are EC2 and RDS, and I'm thinking of moving RDS over to EC2 instances. We are entirely portable. We can move my entire team's infrastructure to a different cloud host really quickly. Our only dependency is a Debian box. I flipped the switch today and cloned our prod environment, including VPN and security rules, over to a commodity hosting provider. Change the DNS entry for the services, and we were good to go. We didn't need to do anything because everyone was freaking out about everything else being down. But our internal services were close to unaffected. At least for my team. Obviously, we aren't Trello or some of the other big people affected. And we don't have the same needs they do. But setting up the DevOps stuff for my team in the way that I think was correct to begin with (no dependencies other than a Debian box) really shined today. Having a clear and correct deployment strategy on any available hardware platform really worked for us. Or at least it would have if people weren't so upset about all our other external services being down that they paid no attention to internal services. Lock-in is bad, mmkay? If your company is the right size, and it makes sense, do the extra work. It's not that hard to write agnostic scripts that deploy your software, create your database, and build your data from a backup. This can be a big deal when some providers are flipping out. All-your-junk-in-one-place is really overrated, in my opinion. Be able to rebuild your code and your data at any given point in time. If you don't have that, I don't really know what you have.
- ec109685 10y agoNot knowing your situation exactly, but there could be a cost of running your own infrastructure and not taking advantage of their services? For example, are the chances of losing data higher in riak (take into account disaster and operational bugs that could result in data loss or availability issues) than in one of amazon's supported data stores. I don't necessarily disagree with what you are saying but there is cost of doing everything yourself. You would have been equally protected if you had been in more than one region.
- ianamartin 10y ago
- all_usernames 10y agoAs of 4:30PM Pacific, we're still having trouble with EC2 autoscaling API operations in US-East-1. Basically very long delays in launching new instances or terminating old ones.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- sc30317 10y agoStill no RCA? I'd love to hear what the issue was for this. A couple of coworkers and I are betting that it was a networking issue of some sort.
- ibmcloud 10y agoOur IBM cloud- Softlayer provides secure and stable cloud environment with private network, for baremetal,dedicated, private and public cloud. Leave a comment if you want to learn more. also HIPAA ready.
- mvindahl 10y agoWe all laughed at the notion om moon people dropping rocks at the earth. Then they started dropped rocks on S3 and who is laughing now?
- julenx 10y agoOutage as a Service
- nomadic_09 10y agoI'm curious to see the postmortem for this.
- kfkhalili 10y agoSo why did the outage occur?