6 ms·
A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You're signing up for the oldest hardware and the most frequent outages. F
by gamache 10y ago
A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You're signing up for the oldest hardware and the most frequent outages.
For legacy customers, it's hard to move regions, but in general, if you have the chance to choose a region other than us-east-1, do that. I had the chance to transition to us-west-2 about 18 months ago and in that time, there have been at least three us-east-1 outages that haven't affected me, counting today's S3 outage.
EDIT: ha, joke's on me. I'm starting to see S3 failures as they affect our CDN. Lovely :/
- xbryanx 10y agoI'm getting the same outage in us-west-2 right now.
- ngtvspc 10y agoI can confirm this as well.
- Ph4nt0m 10y agoSame outage in ca-central-1
- STRML 10y agoSeeing it in eu-west-1 as well. Even the dashboard won't load. Shame on AWS for still reporting this as up; what use is a Personal Health Dashboard if it's to AWS's advantage not to report issues?
- STRML 10y agoNow it's in the PHD, backdated to 11:37:00 UTC-6. How could it take an hour to even admit that an issue exists? We have alerts set on this but they're useless when this late.
- gamache 10y agoHuh, I'm not seeing it on my us-west-2 services. Interesting.
- firloop 10y agoThe dashboard doesn't load, nor does content using the generic S3 url [1], but we're in us-west-2 and it works fine if you use the region specific URL [2]. In practice this means our site on S3/Cloudfront is unaffected. [1]: https://s3.amazonaws.com/restocks.io/robots.txt https://s3.amazonaws.com/restocks.io/robots.txt [2]: https://s3-us-west-2.amazonaws.com/restocks.io/robots.txt https://s3-us-west-2.amazonaws.com/restocks.io/robots.txt
- madmod 10y agoGood catch. My bet is that because s3.amazonaws.com originally referred to the only region (us-east-1) the service that resolves the bucket region automatically is really hosted in us-east-1. I think AWS recommends using the region in the URL for that reason, however that is easier said than done I think. I would bet that a few of Amazon's services use the short version internally and are having issues because of it.
- codelitt 10y agoIIRC the console for S3 is global and not region specific even though buckets are.
- seanp2k2 10y agoAlso, cross-region replication is a new-ish thing: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/ https://aws.amazon.com/blogs/aws/new-cross-region-replicatio...
- WaxProlix 10y agoSame here, and it's 100% consistent, not 'increased error rates' but actually just fully down. I'd just stop working but I have a demo this afternoon... the downsides of serverless/cloud architectures, I guess.
- pm90 10y agoWell what if you'd hosted it on your hard drive and it crashed? It seems like the probability of either is similar nowadays.
- JupiterMoon 10y agoGrab different machine, git clone your repo, good to go. What's the odds of the server with your repo and your own hard drive crashing at the same time?
- _ao789 10y agoStrangely, your comment made me read this entire post about working out probabilities.. http://www.statisticshowto.com/how-to-find-the-probability-of-two-events-occurring-together/ http://www.statisticshowto.com/how-to-find-the-probability-o... Quite interesting really!
- JupiterMoon 10y agoIf we assume that the events are largely uncorrelated+ then we are multiplying the probabilities and our chance of wipe out are far lower. +I would suggest that for situations where the probability of my machine and github's/bitbucket's servers being down due to the same event would be events of such magnitude that I would not be worried about my project anymore being more focused on basic survival...
- jacobwg 10y agoThe difference there is you can potentially do something about it, vs having to wait on an upstream provider to fix an issue for everybody.
- all_usernames 10y agoOur services in us-west-2 have been up the whole time. I think the problem is globally accessible APIs are impacted. As others have noted, if you can use region/AZ-specific hostnames to connect, you can get though to S3. CloudFront is faithfully serving up our existing files even from buckets in US-East.
- illumin8 10y agoS3 bucket creation was down in us-west-2, because it relied on us-east-1 (I expect that dependency will get fixed after this), but all S3 operations should have continued to function in us-west-2, other than cross-region replication from us-east-1.
- notheguyouthink 10y agoThat's a really good point!
- traskjd 10y agoReminds me of an old joke: Why do we host on AWS? Because if it goes down then our customers are so busy worried about themselves being down that they don't even notice that we're down!
- nabla9 10y agoReminds me of an even older joke (from 80's or 90's): Q: Why computers don't crash at the same time? A: Because network connections are not fast enough. (I think we are starting to get there)
- contingencies 10y agoThese are both pretty good. Added to color fortune clone https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
- jchmbrln 10y agoProbably valid, though in this case while us-west-1 is still serving my static websites, I can't push at all.
- compuguy 10y agoIs us-east-2 (Ohio) any better (minus this aws-wide S3 issue)?
- mullen 10y agous-east-2 is brand new and us-east-1 is the oldest region. Any time there is an issue, it is almost always us-east-1. If possible, I would migrate out of us-east-1.
- twistedpair 10y agoAmen. We setup our company cloud 2 years ago in US-West-2 and have never looked back. No outage to date.
- jacquesm 10y agoIf you have a piece of unvarnished wood handy...
- nola-radar 10y agoThe s3 outage covered all regions.
- movedx 10y agoReally? Even Australia? Can you provide evidence of this so I know for any clients that call me today? :) EDIT: Found my answer. "Just to stress: this is one S3 region that has become inaccessible, yet web apps are tripping up and vanishing as their backend evaporates away." -- https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_goes_haywire/ https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...
- nola-radar 10y agoThe s3 outage covered all regions.
- bischofs 10y agoIt shouldnt be technically possible to lose S3 on every region, how did amazon screw this up so bad?
- boulos 10y agoI believe the reports here are misleading: if you try to access your other regions through the default s3.amazonaws.com it apparently routes through us-east first (and fails), but you're "supposed to" always point directly at your chosen region. Disclosure: I work on Google Cloud (and didn't test this, but some other comment makes that clear).
- shirleman 10y agoI used to track DynamoDB issues and holy crap, AWS East had a 1-2 hour outage at least every 2 weeks. Never in any of the other regions. AWS East is THA WURST
- movedx 10y agoMy advice is: don't keep your eggs in one basket. AZs a localised redundancy, but as Cloud is cheap and plentiful, you should be using two or more regions, at least, to house your solution (if it's important to you.) EDIT: less arrogant. I need a coffee.
- jacquesm 10y agoTwo different vendors if you can afford it. It's a bit of a hassle though.
- movedx 10y agoI like to stick to one, but I have seen some success stories with an AWS/GCE mix :-) HashiCorp's Terraform makes it a lot easier to go multi Cloud, and abstracting away configuration of the OS and applications/state with Ansible makes the whole process a lot easier too.
- cfieber 10y agoHere is one success story: https://cloudplatform.googleblog.com/2017/02/guest-post-multi-cloud-continuous-delivery-using-Spinnaker-at-Waze.html https://cloudplatform.googleblog.com/2017/02/guest-post-mult...
- gamache 10y agoBut now you're talking about added effort. Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much. It's easy to say things like this, but people who can do ops are not cheap and plentiful.
- movedx 10y agoThe only difficult aspect of multi-region use is data replication, which I can confirm is a (somewhat) difficult problem. This issue was with S3 which has an option to automatically replicate data from the bucket's region to another one. It's a check box. A simple bit of logic in the application and you can move between regions with ease. Even data replication has options for this, too. And I work in Ops.