7 ms·
AWS S3 Outage
Seeing huge numbers of 503s from the S3 API in us-east-1. Anyone else having problems? I only found one other on Twitter: https://twitter.com/cperciva/status/630641484677558273
- gschier 11y agoLots more complaints on Twitter...
- nmjohn 11y agoI'm seeing it as well - majority of connections are being dropped for us atm The Amazon S3 team recently completed some maintenance changes to Amazon S3’s DNS configuration for the US STANDARD region on July 30th, 2015. You are receiving this email because we noticed that your bucket is still receiving requests on the IP addresses which were removed from DNS rotation. These IP addresses will be disabled on August 10th at 11:00 am PDT, at which time any requests still using those addresses will receive an HTTP 503 response status code. Applications should use the published Amazon S3 DNS names for US STANDARD: either s3.amazonaws.com or s3-external-2.amazonaws.com with their associated time to live (TTL) values. Please refer to our documentation at: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for more information on Amazon S3 DNS names. Something to do with that perhaps? AWS sent us that last thursday
- cperciva 11y agoIf it is related, someone screwed up badly. I didn't receive such an email; and I'm seeing the same error rate with IPs I resolved 5 minutes ago.
- bbrazil 11y agoJava by default can cache DNS forever, which may be why many people are seeing problems. Set networkaddress.cache.ttl to adjust this. http://javaeesupportpatterns.blogspot.ie/2011/03/java-dns-cache-reference-guide.html http://javaeesupportpatterns.blogspot.ie/2011/03/java-dns-ca... has more detail.
- ork 11y agoReading this article, it seems that starting at 1.6 the default TTL became 30s.
- Xorlev 11y agoIt's not true as far as I can tell. You need this flag set or it keeps DNS entries forever, at least on the JVM we run (Oracle, 1.8u51).
- gnethercutt 11y agoThe default behavior in Java 7 & 8 is to cache forever when a security manager is installed, and to cache for an implementation-specific period of time when a security manager is not installed. Sans security manager, I see 30s in Java 7 -- reference the DEFAULT_POSITIVE constant in sun.net. InetAddressCachePolicy
- jtwaleson 11y agoloading fonts on aws pages is slow as hell because of this https://aws.amazon.com/ses/ https://aws.amazon.com/ses/
- jsonperl 11y agoYep, seeing the same.
- adamtulinius 11y agoCan't pull docker images from the hub either, and their statuspage currently shows S3-problems: https://status.docker.com/ https://status.docker.com/
- dhruvbhatia 11y agoCan't docker login or pull images from the hub. Both operations hang.
- endymi0n 11y agohttp://status.docker.com/ http://status.docker.com/ - they are "Investigating issue with high load", now yellow, but was red before. Can't pull any images either.
- mrsuprawsm 11y agoFrom http://status.aws.amazon.com/ http://status.aws.amazon.com/: 12:36 AM PDT We are investigating elevated errors for requests made to Amazon S3 in the US-STANDARD Region.
- deleted 11y ago[deleted]
- rwitoff 11y agosame here. our s3 services are reporting similar 503's and network timeouts. a few of our partners are already down as well with their own 500s. another stormy night in the cloud.
- thinkindie 11y agoI'm also getting problems with Cloudfront attached to an S3 bucket
- nebulon 11y agoSame here, first it only affected some dev deployment (s3+cdn), now it spilled over to other buckets :-/
- simonpantzare 11y agoSeeing the same thing. Got back from vacation an hour ago, probably related. :)
- sschueller 11y agoAt least it didn't happen while you where on vacation. :)
- dutchbrit 11y agoWelcome back!
- shaper60 11y agooh... seeing the same
- fizx 11y agoI wonder what it would take for amazon to show one of the yellow icons on their status page? Has it ever happened? Would a datacenter have to fall in the ocean?
- gschier 11y agoYa, it bothers me that their status messages for major outages are simply "elevated error rates".
- cperciva 11y agoI'm sure "elevated error rates" is the first alarm which goes off. And once they've put that description onto the status page, they're probably more worried about getting it fixed than going back and changing the wording.
- mrsirduke 11y agoThey should worry about that a lot. Amazon are notoriously bad at communicating during outages. They've gotten better, but they're big enough that it should have priority.
- cperciva 11y agoI agree. I was speculating, not trying to defend. :-)
- Perdition 11y agoAmazon isn't a couple of guys in a garage. They have hordes of administrative personnel who could be tasked to update a status page. Companies 1/1000th the size of Amazon can manage it.
- cperciva 11y agoYes and no. Sure, Amazon has administrative personnel. Sure, some of those administrative personnel would probably be happy to get paid extra to carry a pager and be summoned to work at 3AM to update a status page. But the last thing you want to do is put inaccurate information onto a status page; so mere administrative personnel isn't enough -- you'd need people who understand enough about the system to be able to write about it without introducing errors. I'm guessing that the intersection of "administrative personnel", "willing to carry pagers" and "understand the internals of AWS services" is a very small set.
- shaper60 11y agoまだ直らないねえ
- kernel_sanders 11y agoCan't launch instances in EC2 in US-East-1 at the moment.
- jschorr 11y agoIt appears EC2 is affected as well now: 12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
- crodjer 11y agoCould this be a reason why Heroku is misbehaving? https://status.heroku.com/incidents/792K https://status.heroku.com/incidents/792K
- crodjer 11y agoHeroku build logs spitted this out: Unable to fetch source from: https://s3-external-1.amazonaws.com/heroku-sources-production/heroku.com/<some-uuid>?AWSAccessKeyId=<some-access-key>&Signature=<some-signature>&Expires=1439198046
- robotfelix 11y agoI can't help but be a little surprised that Heroku's entire build system is disabled by an S3 failure in one region. Now I'm unable to add a notice about the issues to my site's HTML...
- crodjer 11y agoTrue. Given the business Heroku is in, they should have redundancies in multiple regions if not multiple service providers.
- Xorlev 11y agoSince S3 is the defacto artifact delivery system for most people that run on AWS, it's not much of a surprise. For the most part, very isolated incidents aside, S3 is rock solid. Even EC2 relies on S3 for launching non-EBS instances.
- deleted 11y ago[deleted]
- blowski 11y agoI'm also having issues connecting to buckets based in Ireland (eu-west-1). Just hangs at authentication stage. Tried from 3 different internet connections, all having the same problem.
- mrsuprawsm 11y agoFor what it's worth, eu-west-1 buckets are working fine for me here, via s3cmd and aws cli.
- onre 11y agoDitto. API does not answer at all, latency is high and customers are not amused. Outage has lasted for hours already. edit; does not seem to affect all the buckets though. Only one of ours is experiencing this, others are fine.
- JoshGlazebrook 11y agoI'm assuming this is also why I can't start any instances. > 12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
- rpmartz 11y agoSame here
- geomark 11y agoYep. Can't even get a response to a s3cmd command.
- jackyjjc 11y agoeven ec2 is down: Amazon Elastic Compute Cloud (N. Virginia) Increased API Error Rates 12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
- ranrub 11y ago"1:08 AM PDT We believe we have identified the root cause of the elevated error rates and latencies for requests to the US-STANDARD Region and are working to resolve the issue." looks like the cavalry are coming
- deleted 11y ago[deleted]
- jakozaur 11y agoOpen-source library request: A library that lets you use S3 and Google Storage Cloud simultaneously and fail-back to another if one have problems. There are many use-case when paying 2x for storage is a reasonable tradeoff for higher availability and also be provider independent.
- ketralnis 11y agoWhat's stopping you from writing it?
- jakozaur 11y agoTime and priorities. There is a difference between "I wish this thing existed, so I can use it/contribute to it" and "I need it so badly that I'm willing to spend a lot of time to make it production ready". So far S3 seems to be reliable enough...
- toomuchtodo 11y agoMight be something to integrate into Libcloud [1] instead of rolling your own. [1] https://libcloud.apache.org/ https://libcloud.apache.org/
- mnutt 11y agoDepending on your use case, it may be slightly easier to accomplish this with s3 event notifications + AWS Lambda to write to a different region or service. Importantly, make sure you CNAME your bucket under your own domain so that you can switch services. edit: Much easier than AWS Lambda, actually: http://aws.amazon.com/about-aws/whats-new/2015/03/amazon-s3-introduces-cross-region-replication/ http://aws.amazon.com/about-aws/whats-new/2015/03/amazon-s3-...
- rplnt 11y agoThere used to be this https://deltacloud.apache.org/ https://deltacloud.apache.org/ (not exactly what you want + it seems to be dead).
- 11y ago
- rgbrgb 11y agoyep, we're seeing timeouts and 404s for images stored on s3 :( good luck to the on-call engineers at amazon!
- greenleafjacob 11y agoWe are seeing lots of 503s, empty response bodies, and peer reset / dropped connections.
- chncdcksn 11y agoGitHub is having release download issues, possibly due to this. https://status.github.com/ https://status.github.com/
- Maxious 11y agoMapbox and Hipchat too https://www.mapbox.com/status/ https://www.mapbox.com/status/ https://status.hipchat.com/ https://status.hipchat.com/
- DanKlinton 11y agoFeels like internet is not anymore distributed thing that if one website/node goes down others keep working... Feels like in future... If cloud provider goes down... All internet will stop working :)
- rodgerd 11y agoThis is pretty much the case. Years of evangelising the idea that (a) everybody should be on Amazon and (b) everybody should be on the cheapest regions of Amazon mean that while the underlying datacentres are probably much better managed, individually speaking, than the tapestry of colos that made up the world a decade ago, an outage has much more wide-ranging effects than you'd get at that point.
- deleted 11y ago[deleted]
- jontro 11y agoLooks like this brought down typekit too. "Font Network is experiencing issues caused by an outage at our storage provider." http://status.typekit.com/ http://status.typekit.com/
- zubairov 11y ago+1 for us it's CDN (CloudFront) - only HTTP 503 responses
- jschorr 11y agoLatest Update from http://status.aws.amazon.com/ http://status.aws.amazon.com/: 1:52 AM PDT We are actively working on the recovery process, focusing on multiple steps in parallel. While we are in recovery, customers will continue to see elevated error rate and latencies.
- kore_sar 11y agoThe icon got yellow. I repeat: YELLOW
- eloisant 11y agomy service looks pretty red now
- hughstephens 11y agoIt only goes red when a nuclear event occurs, obliterating most of humanity and only the machines remain.
- gotroot 11y agoThat would actually turn it green again, I think.
- hughstephens 11y agomaybe true – but if a server gets no requests, does it really exist? or would the inter-machine chatter continue ad infinitum? would they run out of IPs or successfully transition to IPv6? So many questions.
- mryan 11y agoCory Doctorow's When Sysadmins Ruled the Earth touches on this - what happens to the internet activity during a global crisis? http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...
- deleted 11y ago[deleted]
- kureikain 11y agoIt's two hours. Luckily it happenned around 12:00AM otherwise we are having a bad day. Whomever being a DevOps or SysAdmin probably cannot sleep tonight :(. In our case, we put Fastly on top of our assets/images so that only a partil of request get errors. The cached object on Fastly is still fine.
- jbbarth 11y agoActually on some parts of the world it's not 12:00AM you know, and there are sysadmins/devops there too :-)
- addandsubtract 11y ago11:40am: Currently can't push to Heroku :(
- kureikain 11y agoyeah, forgot that :(. Hope it comes back soon.
- hughstephens 11y agohaha if two hours is nothing to you, that's pretty lucky. We work 24x7 (global customers, yes US is a bit busier but not by a lot) and we're an Australian company, so not a whole lot of fun.
- kureikain 11y agoYeah, sorry to hear that :(. I mean it's already two hours, a long time. However, if it doesn't come back before 5AM I think we are screw :(
- eloisant 11y agoOur European customers are not amused.
- vaibhavrajput 11y agoWhy it is behaving like this each other day?
- eva1984 11y agoWhat is the last time that AWS gets two consecutive major outages within, maybe, 30 days?
- lubos 11y agoLast one was 10 days ago. https://news.ycombinator.com/item?id=9980222 https://news.ycombinator.com/item?id=9980222
- vaibhavrajput 11y ago+1, Also it was down on leap second. http://mashable.com/2015/06/30/aws-disruption http://mashable.com/2015/06/30/aws-disruption
- cheeseprocedure 11y agoTo be fair, that was not really AWS's fault, nor (apparently) a leap second issue: http://www.bgpexpert.com/article.php?article=167 http://www.bgpexpert.com/article.php?article=167 https://twitter.com/Axcelx/status/616058414746202113 https://twitter.com/Axcelx/status/616058414746202113
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- mentat 11y agoNow updates about ELB scaling and Lambda failures.
- kevindeasis 11y agoThat would explain why my console was not performing well even if http://status.aws.amazon.com/ http://status.aws.amazon.com/ says "Service is operating normally", good thing their api seems to be functioning during that outage, for me at least
- hughstephens 11y agoNext update is live 2:38 AM PDT We continue to execute on our recovery plan and have taken multiple steps to reduce latencies and error rates for Amazon S3 in US-STANDARD. Customers may continue to experience elevated latencies and error rates as we proceed through our recovery plan.
- zyzyis 11y agowtf, aws again
- gedrap 11y agoI see this thread as a list of services depending on S3 being healthy.
- jschorr 11y agoAlso a good list of developers and devops people not getting any sleep tonight :-/
- bardworx 11y agoYup! 3am "Dry Run" Staging on Heroku...fail 7am Deployment to Production....fail Now: "We have confirmed elevated latencies affecting our SendEmail, SendRawEmail and SendSMTPEmail APIs in the US-EAST-1 Region and are working to address the problem." Which is perfect since most of our PO orders are being placed between 6am-10am. Tonight, adult beverages will be needed after everything is resolved.
- mentat 11y agoCurrent list of additional services affected: CloudSearch Elastic Compute Cloud Elastic Load Balancing Elastic MapReduce Relational Database Service CloudTrail Config Lambda OpsWorks
- clebio 11y agoSeems Hasicorp is maybe affected by this as well. $ vagrant up Bringing machine '...' up with 'virtualbox' provider... ==> ...: Box 'debian/jessie64' could not be found. ... ...: Downloading: https://atlas.hashicorp.com/debian/boxes/jessie64/versions/8.1.0/providers/virtualbox.box An error occurred while downloading the remote file. The error message, if any, is reproduced below. Please fix this error and try again. The requested URL returned error: 500 Internal Server Error EDIT: not Markdown.
- uxp 11y agoVagrantCloud, now known as Atlas, is merely a redirector service for Vagrant box management. I host all my own boxes (on S3), but still use them for an easy means of sharing without having to remember a long-ass URL.
- colinbartlett 11y agoI started receiving lots of alerts from my side project https://StatusGator.io https://StatusGator.io which monitors status pages. It's astonishing to me how many services depend on AWS directly or indirectly.
- padelt 11y agoThe most ironic thing I've seen in a while: Their homepage features icons of services they monitor. Many of them 503-fail - they are hosted on cloudfront.
- xenoclast 11y ago[3:25] AM PDT We are still working through our recovery plan. Man, I'd love to see that plan.
- cddotdotslash 11y agoI'd also love to see a post mortem on this, but I highly doubt they'll release anything about it.
- pemp 11y agoAs it has happened before, Amazon AWS status page is lying to us. S3 is in yellow, which means "performance issues". But not being able to download files from many buckets it's clearly a "service disruption" (red).
- lostdd 11y agoWhat a wonderful beginning of a week! Thanks AWS.
- ranrub 11y agoLooks like it's getting better on our side now
- cubicfur 11y agoGood thing I built myself a local game streaming server instead of putting that in a remote GPU instance.
- cperciva 11y agoAs of 10:29:33 UTC, everything is back to normal as far as I can measure.
- theyeti 11y agoIt seems to have come back now for me. Could someone else confirm the same ?
- whyleyc 11y ago3:36 AM PDT Customers should start to see declines in elevated errors and latencies in the Amazon S3 service. Fixed ?
- dangravell 11y agoLooks like I picked a bad week to stop sniffing glue.
- mryan 11y agoShould be back to normal now. The latest update is: 3:46 AM PDT Between 12:08 AM and 3:40 AM PDT, Amazon S3 experienced elevated error rates and latencies. We identified the root cause and pursued multiple paths to recovery. The error has been corrected and the service is operating normally.
- ramon 11y agoThis happens a couple of times, specially when replacing files frequently! I submit things to S3 everyday, if you're uploading a chunck of files you'll get errors every now and then when replacing files.
- pydevops 11y agoAWS API still works while AWS web console is not.