4 ms·
>In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebo
by SMAAART 5y ago
>In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded.
SinglePointOfFailure.NoRedundancies.FB
- seaman1921 5y agoNice try, but they have separate communication channels for SREs so don't worry.
- jrodthree24 5y agoYou mean they don't receive all their alerts through facebook messenger?
- regnull 5y agoHaving worked for a similar company, I remember there were some good old IRC servers up and running to communicate in an emergency just like that.
- makeworld 5y agoFacebook has this too, but they require facebook.com DNS to work, so they are also down.
- coldcode 5y agoThanos snapped his fingers and Zuckerberg vanished with the keys. My (very large) employer had a worldwide outage a few years ago where a single bad DNS update stopped everything in its tracks (at the time many things were still in our own data centers, now more is in Amazon/etc). It took most of the day to restart everything. But it's not something most people would have noticed like FB. Thankfully I worked in mobile so not involved.
- manquer 5y agoIt is hard to balance dogfooding (good) with SPOF (bad), many big companies do get it wrong (AWS with S3, Slack in the recent past) all the time. It is easy to get it wrong if your company provides internet services that every developer typical depends in their workflows and to keep educating your own developers on how to use them and when not to use your own services.