4 ms·
(disclaimer: I work for another cloud provider) I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a
by Twirrim 8y ago
(disclaimer: I work for another cloud provider)
I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:
1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.
or:
2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
- toomuchtodo 8y agoThat shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4]. Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know. Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage. Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region. [1] https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ [2] https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-prime-day-downtime.html https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-p... [3] https://www.cnet.com/news/google-cloud-issues-causes-outages-for-snapchat-spotify-and-others/ https://www.cnet.com/news/google-cloud-issues-causes-outages... [4] https://www.datacenterknowledge.com/uptime/microsoft-blames-severe-weather-azure-cloud-outage https://www.datacenterknowledge.com/uptime/microsoft-blames-...
- ignoramous 8y agoCan't speak for Google, but Facebook and Salesforce chose Cells for HA. http://highscalability.com/blog/2012/5/9/cell-architectures.html http://highscalability.com/blog/2012/5/9/cell-architectures....
- toomuchtodo 8y agoDoesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently. > Facebook Platform Appears to be down > A check of https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps. https://news.ycombinator.com/item?id=18434262 https://news.ycombinator.com/item?id=18434262
- doodliego 8y agoLook how frequent and detailed Amazon's update logs are in that first Register article. Multiple updates throughout the day going into some detail.
- deleted 8y ago[deleted]
- Latteland 8y agoWhen I was at google, the big outages were almost always bad routing to the service - it was never that the service couldn't handle the load, and bad service instances were kind of hidden. my service did have some problems on new releases, but because we had multiple instances we could just redirect traffic to the instances we hadn't updated, so they stayed up.