5 ms·
So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will
by ThePhysicist 3y ago
So they run everything in AWS USE1? That doesn't seem very redundant, but then I guess if the whole of USE1 goes down Slack won't be the only service that will be affected.
- radicality 3y agoIsn’t the point of the article that they don’t? And it describes how they implemented region drains to traffic shift between the different regions. edit: Hmm or maybe not? I still sometimes confuse aws terminology. Perhaps it is all in us-east—1, just in different availability zones (buildings?)
- ThePhysicist 3y agoIf I understand it correctly they have an edge network for ingress traffic but host all of their core services in a single AWS region (USE1) in multiple availability zones there.
- jldugger 3y ago>edit: Hmm or maybe not? I still sometimes confuse aws terminology. Perhaps it is all in us-east—1, just in different availability zones (buildings?) Correct, us-east-1 has several AZs, names like us-east-1a, us-east-1b etc. IIRC us-east-1 has six of them now.
- messe 3y agoAWS also uses Slack internally, so add that to the list of shit that can hit the fan if us-east-1/IAD goes down.
- mynameisvlad 3y agoDon’t they also use Chime? It wouldn’t be a single point of failure.
- skullone 3y agoLots of teams use Slack as well. Oddly enough, I didn't mind Chime as an end-user, but 6 years ago their API features were somewhat lacking.
- nostrebored 3y agoTo contribute to the tangled ball of messaging, slack also uses chime sdk to handle huddles
- notyourwork 3y agoWonder why huddles sound better compared to chime?
- fotta 3y agoHuh, I’m surprised they’re not all in on Chime.
- shepherdjerred 3y agoIt was all on Chime until the Pandemic. Then they moved to Slack.
- deleted 3y ago[deleted]
- deanCommie 3y agothe "whole" of USE1 very rarely goes down [0], because unlike other cloud providers, Amazon's availability zones are actually independent and decoupled, and if you're running on EC2 in a zonal way it's highly unlikely an outage will affect multiple zones. [0] There are of course exceptions that come once every few years, but most instances people can think of in terms of widespread outages is one specific service going down in a region, creating a cascade of other dependencies. e.g. Lambda or Kinesis going down and impacting some other higher-level service, say, Translate.
- asah 3y agoAm I missing something about us-east-1 reliability ? https://www.google.com/search?q=us-east-1+reliability https://www.google.com/search?q=us-east-1+reliability https://www.google.com/search?q=us-east-1+outage https://www.google.com/search?q=us-east-1+outage
- temp_praneshp 3y agoYes. to put it a bit bluntly, you are using a very generic google search and being blind to nuance. us-east-1 does have more problems than other zones due to a variety of reasons, but it rarely (ie, once a few years) goes down as a whole. As long as you're in several AZs within us-east-1, the impact of most outages should not take you down completely. In the context of the comment you are replying to, your google search links are lazy and fail to see the big picture.
- asah 3y agothanks - jfyi this was from my personal experience spanning a decade with USE1, but again maybe my experience is out of date, so thanks for the update. (p.s. the use of a google search vs direct results wasn't "lazy" - it's to allow readers to do their own research vs pasting one result and then getting accused of bias)
- oceanplexian 3y agoAZs are buildings often times right next to each other on the same street. People who think this is a great failure domain for your entire business are deeply misguided. All it takes is a hurricane, a truck hitting a pole, a fire, or any number of extremely common situations and infra will be wiped off the map. Build stuff to be properly multi-region.
- johannes1234321 3y agoBut then everybody trying to recover from USE1 outage can't use Slack to coordinate the recovery ...