10 ms·
S3 was down
- ak217 9y agoWhat do people use for mocking S3 behavior with high fidelity, to test exponential backoff and other resilience strategies in this situation? I've seen a handful of test libraries but none of them seem to make realistic up-to-date error injection a priority.
- silasb 9y agoMaybe minio.
- YawningAngel 9y agoWe use S3-proxy internally and are pretty happy with it.
- rob-olmos 9y agohttps://github.com/andrewgaul/s3proxy https://github.com/andrewgaul/s3proxy ? (no hyphen)
- stuffaandthings 9y agoyup EB interface is affected too
- parthdesai 9y agoYup, it is down for our site. Some of the images are not being loaded. (US-east-1) Console says: Failed to load resource: the server responded with a status of 503 (Slow Down)
- sotojuan 9y agoHappening here too (NYC, us-east-1).
- deleted 9y ago[deleted]
- toddwprice 9y agoDown for us as well. `An error occurred with the message 'Please reduce your request rate.' when writing an object`
- zedpm 9y agoKudos for Amazon sorting out their status page. The last time this happened, the status page didn't show anything until hours after the outage began. I just noticed this one maybe 15 minutes ago, and 5-10 minutes later they acknowledged the problem. On the other hand, I'm feeling a strong case of "Not this shit again". Wondering if US-East-1 is more trouble than it's worth, as these outages seem to happen mostly there.
- sitharus 9y agoUs-east-1 is where all the shiny new features are released and where most customers are. From experience the other regions are much more reliable.
- lightbyte 9y agoI thought us-east-1 had the most issues because it is the oldest region and thus uses the oldest hardware.
- BillinghamJ 9y agoIt’s primarily just due to the incredible scale of the region. It’s easily twice as large as any other, so any issues relating to scale appear there first and are fixed before they affect any of the other regions.
- lozenge 9y agoIt might also be related to IAM and some other stuff being "based" there.
- yeukhon 9y agoWhat do you mean "based" there?
- BillinghamJ 9y ago
- imrehg 9y agoDefinitely seems like that... And we should back off and reduce our request rates, HN, otherwise we'll make it worse!
- rficcaglia 9y agoI tried CTRL+ALT+DEL but it didn't help!! :)
- drusepth 9y agoTurning it off and back on again as we speak.
- klinskyc 9y agoThought I was going crazy. We're seeing 'Please reduce your request rate.' for a lot of our requests as well
- rficcaglia 9y agoDefinitely affecting S3 bucket ops, static site hosting on s3, and even things like CloudTrail. Anything that uses S3 East VA my guess. Rather severe impact. Maybe less focus on avocados is what's called for...
- crgwbr 9y agoWe're see 'Please reduce your request rate' frequently on read and write operations.
- maccam912 9y ago"Slow Down"s for me. We started seeing some weird S3 behavior since about 11:45 PST.
- CityLims 9y agoRussia? China? Jealous of our cat pics much?!
- barillax 9y agoWe're seeing the same thing. Been tracking it via Twitter for the last 15 minutes or so: https://twitter.com/search?q=s3%20slow%20down https://twitter.com/search?q=s3%20slow%20down
- alexandros 9y agoLots of people confirming this on Twitter[1], and we're also seeing it at resin.io [1]: https://twitter.com/AlecSanger/status/908402829349572608 https://twitter.com/AlecSanger/status/908402829349572608
- fernandopj 9y agoFrom status page: (https://status.aws.amazon.com https://status.aws.amazon.com) 11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
- fernandopj 9y ago12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.
- fernandopj 9y agoNow it is CodeCommit too: 12:28 PM PDT We are investigating increased error rates for Git Push, Git Pull and API calls in the US-EAST-1 Region.
- rhelsing 9y agoI'm getting issues as well. Here's to hoping this issue is just a hiccup and not like the last outage.. otherwise, it may be time to seriously consider alternatives. Anyone know of a good solution for a self-hosted s3-esque service?
- benwilber0 9y ago> it may be time to seriously consider alternatives they have multi-az support if you use it
- xenophonf 9y agoHow does one set that up?
- rficcaglia 9y agotrue, but some internal AWS features themselves rely on S3 and apparently us-east-1. Those are hosed right now, too. You can't even modify bucket metadata via AWS Console or cli tools. So like checking permissions or logging right now is not possible, or CloudTrail.
- fernandopj 9y agoCodeCommit, CodePipeline and EBS on my environments are all affected. I have a domain down and can't push a new deploy. They are all multi-az.
- zedpm 9y agoNot exactly. S3 doesn't have AZs at all, it's only split on regions. Further, a bucket can exist in only one region. You can set up cross-region replication, but you of course need to flip the bucket coordinates in all your applications to fail over. It's not nearly as easy as Multi-AZ support in things like RDS.
- yeukhon 9y ago> split on regions I think you really meant s3 objects are redundant in each region, which actually is spanned across multiple DCs.
- xfax 9y agoDoes anyone else have some of their buckets missing from the console list?
- ceejayoz 9y agoI'd fully expect an S3 outage to affect that display.
- quickConclusion 9y agoOur buckets are there, just we cannot GET content, with error messages about reducing the rate.
- deleted 9y ago[deleted]
- klinskyc 9y agoSeeing the "reduce your requests" error message on our end
- xfax 9y agoDoes anyone else have some of their buckets missing from the console list?
- pierrebeaucamp 9y agoSeems like Github might be affected as well? Can't comment or merge PRs.
- pierrebeaucamp 9y agonah, works now. Probably unrelated
- pm90 9y agoGithub isn't hosted on AWS btw.
- jwilk 9y agoAt least downloads are hosted on S3.
- coreyw 9y agoWe are seeing increased error rates for GET and HEAD requests. So far we have not been having any issues with PUT requests. We are in us-east-1.
- andrewaylett 9y agoIn us-east-1. Don't worry me like that! It's definitely not down _globally_ or else I'd have been paged by now...
- scrollaway 9y agoQuick tip for those using static (jekyll, hugo...) sites on s3: If you have cloudflare in front of it, you can turn on aggressive html caching by creating a page rule: example.com/* => Cache level => Cache everything https://support.cloudflare.com/hc/en-us/articles/200172256-How-do-I-cache-static-HTML- https://support.cloudflare.com/hc/en-us/articles/200172256-H... This downtime made me realize we weren't caching any html on cloudflare. I just turned it on and all our static sites are doing fine now (and our bills are smaller!). If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.
- kccqzy 9y agoProgrammatically purging caches is slow and costs extra. Just use normal cache-busting techniques like a different URL after a deployment. EDIT: Oops. Misread cloudflare/CloudFront.
- hamandcheese 9y agoThats fine for assets, but not at all ideal for pages.
- sgl75 9y agoChanging URLs is not really an option for html pages of a static site.
- jamiesonbecker 9y agoOnly for the root of a site, but there are better cache expiration options as noted elsewhere anyway. For inner pages, browsers do consider /your-page-url?v=1505418887 to be a completely different page from /your-page-url?v=1505418888. However, just expiring cache and/or properly setting HTTP cache control headers at most CDN's is a cleaner and more correct option.
- sgl75 9y agoI'm not following. How does that help my visitor see the latest version of the page when coming from a search engine or external site?
- temuze 9y agoNow let's see who learned from the last major outage 200 days ago :)
- pilom 9y agoThis was us! Added CloudFlare in front of every bucket with our own "cdn.example.com" DNS name and changed every reference to s3 in code. Didn't even know S3 had problems today until I saw this on HN. (note: This outage taught us to set up monitoring of the S3 assets separate from the "cdn")
- bm1362 9y agoThey've fixed the status indicator, it's now a static asset on the status page: * http://status.aws.amazon.com/images/status1.gif http://status.aws.amazon.com/images/status1.gif
- cordite 9y agoCan't receive orders :D
- ksenzee 9y agoNice to see this getting reported on the AWS status page during the actual event, for once.
- deleted 9y ago[deleted]
- jaysunn 9y agoWe are seeing errors in US-EAST: error: S3ServiceException:Please reduce your request rate.,Status 503,Error SlowDown,Rid
- scrollaway 9y agohttps://news.ycombinator.com/item?id=15251374 https://news.ycombinator.com/item?id=15251374
- jaredstenquist 9y agoUnable to deploy with Elastic Beanstalk currently. Internal 500 error. They put up the Blue Diamond icon, so this must be serious.
- rabidonrails 9y agoNew update from AWS: 12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.
- veb 9y agoIs Slack using S3? I can't seem to upload anything.
- lasermike026 9y agoOh boy.
- 1001101 9y agoI can see our us-east-1 buckets, and just pulled a file out. WFM, FWIW. IAM got me a few weeks ago though. :(
- Achshar 9y agoI am having trouble on my vultr VPN as well. Any chance it's related?
- namidark 9y agoNo
- stormcode 9y agoIt's effecting heroku deploys that rely on build-packs hosted on S3 (like ruby build-pack).
- insomniacity 9y agoThat's really disappointing - why isn't something so critical to heroku deployments distributed across multiple AZs and regions?
- chosenken 9y agoYup, its taking all our stuff down now too. We are working on moving our static site to S3 behind cloudfront, testing it now. Its down. Our Airflow tasks that use S3, they are all down. Basically, we are down :)
- scrollaway 9y agoUnrelated to the downtime but what do you use Airflow for? We've been considering using it for our ETL, but I'm even wondering if we could replace our Jenkins instance with it (Jenkins currently only runs cronjobs and one-off tasks; for CI we use travis).
- chosenken 9y agoWe use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOps team, running our infrastructure and working on back end systems, so my knowledge of airflow is more high level. I just keep the system up so it can run ;P FYI I work for a startup out of Cincinnati, OH called Astronomer, you can find us at https://www.astronomer.io https://www.astronomer.io Also, we weren't like all down, we just saw lots of time out issues when reading/writing to S3.
- scrollaway 9y agoYour site's ssl cert is not setup correctly :) Very interesting re redshift => google analytics though, I've never heard of it done in that direction. Do you think airflow is suitable for one-off/cron task management as well?
- chosenken 9y agoActually yes. Airflow has cron support baked in, when you run a task you can give it a cron schedule. It then takes care of running your tasks when scheduled. As for the ssl, I messed up my link. We are in the process of moving the site, so some of the redirects are not working perfect at the moment >.>
- rdtsc 9y agoStill 99.99% availability just the averaging time was increased by another 100 years :-)
- ceejayoz 9y agoS3 commits to being 99.9% available (~9 hours a year). You may be thinking of the object durability numbers.
- hyperanthony 9y agoConfirmed this is also impacting CodeDeploy in us-east-1, makes sense since it has S3 dependency on revision location.
- tuna 9y agohttps://www.whoownsmyavailability.com/ https://www.whoownsmyavailability.com/
- ajoy 9y agoMost of the recent issues seems to be in this particular colo (us-east-1), which is also their oldest facility. Maybe its prudent to move resources to other newer colos or pay for multi-zone availability.
- twistedpair 9y agoAmen. We moved everything to US-West-2 years ago and never looked back. When we see VA on fire, we just sigh.
- yeukhon 9y agoWell, if your users are in the east cost / London, you want to build something closer without spinning up one in Europe yet, us-east-1 is your choice. So.. YMV.
- redthrowaway 9y agoWe're getting 503s on s3 assets. Definitely having issues. Cloudfront is still good.
- philhartmanonic 9y agoI've gotten a lot of those errors while using the AWS CLI for S3, but it was intermittent. I was using the sync command, so I just kept kicking it back off again as soon as it finished, and eventually got everything up there.
- mrb 9y agoEvery time there is an outage at some cloud provider, I enjoy knowing that my site has maintained 100% availability since its launch. I run 3 redundant {name servers, web servers} pairs on 3 VPS hosted at 3 different providers on 3 different continents. Even if the individual providers are available only 98% of the time—7 days of downtime per year—my setup is expected to still provide five nines availability (details: http://blog.zorinaq.com/release-of-hablog-and-new-design/#conclusion http://blog.zorinaq.com/release-of-hablog-and-new-design/#co...) Edit: It's not about bragging. It's not about the ROI. I want to (1) experiment & learn, and most importantly (2) show what is possible with very simple technical architectures. HN is the ideal place to "show and tell" these sorts of projects.
- devhead 9y agoto be fair, it's a static html blog; while most of us who are now experiencing issues are running a bit more complicated set of services. edit: punctuation
- d3ad1ysp0rk 9y agoThe ROI on keeping a personal blog up at 5 9s is awful... I understand the desire from a geeky perspective, but it's only really useful for personal enjoyment of the challenge, or bragging.
- jsmthrowaway 9y agoDon't forget completely irrelevant HN comments, the only kind of bragging that matters. "S3 is down, but check out my .01RPS multihomed Wordpress which doesn't use S3 in the slightest, and the meta posts on it about how I did it!" Never fails. I should start talking more about my 100% availability on a blank white page I use for parked domains. That's right: I see your nines and raise you zeroes, baby!
- mattrjacobs 9y agoIf you want to understand the impact of an S3 outage (or many other kinds) ahead of time, Gremlin (http://gremlininc.com http://gremlininc.com) has built a tool to run these scenarios on your infrastructure. Happy to answer any questions about the tool, either here or over email. (Disclaimer: I work for Gremlin)
- runesoerensen 9y ago"12:49 PM PDT We are now seeing recovery in the throttle error rates accessing Amazon S3. We have identified the root cause and have taken actions to prevent recurrence."
- enkay 9y agoNot sure if related but had unusual SES timeouts and "too many requests" errors around the same time this was posted.
- vim_wannabe 9y agoThe great thing about AWS is that you shift the burden of responsibility to Amazon.
- notyourday 9y agohttps://www.whoownsmyavailability.com/ https://www.whoownsmyavailability.com/
- discordianfish 9y agoI've just launched https://pub.latency.at/ https://pub.latency.at/ which give you free Prometheus metrics for various cloud endpoints like S3. Doesn't look too bad right now though. It's also hosted on S3 but still up and running. (The main service is independent of S3 anyway after setup)
- discordianfish 9y agoI've added a play-with-docker button to give you a prometheus+grafana instance scraping this. Spun up this few hours ago, will be gone soon but if anyone wants to check it out without waiting for the metrics to trickel in: http://pwd10-0-33-3-3000.host3.labs.play-with-docker.com/dashboard/db/http-duration-site-per-region?refresh=1m&orgId=1&var-site=s3.ap-northeast-1.amazonaws.com&var-site=s3.ap-northeast-2.amazonaws.com&var-site=s3.ap-south-1.amazonaws.com&var-site=s3.ap-southeast-1.amazonaws.com&var-site=s3.ap-southeast-2.amazonaws.com&var-site=s3.ca-central-1.amazonaws.com&var-site=s3.eu-central-1.amazonaws.com&var-site=s3.eu-west-1.amazonaws.com&var-site=s3.eu-west-2.amazonaws.com&var-site=s3.sa-east-1.amazonaws.com&var-site=s3.us-east-1.amazonaws.com&var-site=s3.us-east-2.amazonaws.com&var-site=s3.us-west-1.amazonaws.com&var-site=s3.us-west-2.amazonaws.com&var-region=All&var-ssl_expires=All http://pwd10-0-33-3-3000.host3.labs.play-with-docker.com/das...
- cddotdotslash 9y agoGood time to setup cross region replication with multiple CouldFront origins. I made a tutorial site to show you how: https://spwa.io https://spwa.io
- Slippery_John 9y agoAs always I recommend people practice Chaos Engineering [0] to minimize the impact of such events. Even if the complexity of cross-region failover is to expensive, having some sort of graceful failure is preferable to simply dying. Netflix's Simian Army [1] toolset is particularly useful here, though I don't see an OSS version of Chaos Kong (which simulates the failure of an entire region) sadly. [0]: http://principlesofchaos.org/ http://principlesofchaos.org/ [1]: https://github.com/Netflix/SimianArmy/wiki https://github.com/Netflix/SimianArmy/wiki
- sgl75 9y agoSeems to be resolved.
- daxorid 9y agoStatus page claims resolution, but errors are still ongoing for us.
- devhead 9y agolooks like s3 apis are starting to respond correctly now. hopefully this raises awareness on how important planning for failure is before you make a design choice to introduce a dependency.
- bdibs 9y agoEverything seems to be running normally again, at least for me.
- vacri 9y agoI wish that AWS would settle on a standard timezone (preferably UTC). Troubleshooting the fallout had me mentally converting their PDT status pages with console graphs in both UTC and 'local browser' time[1]. All for a region located in EDT. [1] I think they even have a graph somewhere whose axes are in UTC, but whose tooltips are in local browser time, but I can't recall for sure.
- RomanPushkin 9y agoIt happens pretty often, so I started awesome list: https://github.com/ro31337/awesome-aws-alternatives https://github.com/ro31337/awesome-aws-alternatives Please contribute if you know alternatives.