7 ms·
Seem like a massive overreaction. Keep the sys admins on-site to manage your on prem gear but they should have been doing that anyway?! Surely when you're runni
by jmfldn 5y ago
Seem like a massive overreaction. Keep the sys admins on-site to manage your on prem gear but they should have been doing that anyway?! Surely when you're running your own servers for anything of any scale you need actual people nearby 24/7. Am I missing something?
- kiernanmcgowan 5y agoRemember that a huge part of this fall out was that FB staff couldn't talk to each other in a moment of crisis. Not just the people trying to manage the crisis, but the entire company faced a work stoppage in one way or another. If you do the cost benefit of allowing people to work from home vs the cost of this one outage, I bet the outage is more expensive.
- jmfldn 5y agoSurely they need easy comms fallback options that are independent of their own tech / infra. Even if they were on site this would still be pretty terrible to have everything so coupled right?
- CydeWeys 5y ago> Surely they need easy comms fallback options that are independent of their own tech / infra. And you know what's excellent for that ... literal word of mouth. I'm back in the office with 5 of my coworkers right now, and when we have issues we can just talk about them, zero technology needed.
- jmfldn 5y agoTrut, that helps. However, even medium sized companies are often spread across multiple cities and timezones so it only helps to an extent. Having a single point of failure for literally everything seems to be a pretty large elephant in the room here.
- fortuna86 5y agoSurely each co-worker had the phone number of their colleagues ?
- leesalminen 5y agoI don’t make a habit of giving my personal phone number to everyone I work with. I mean if it happens naturally (say, meeting up after work for a drink) then sure but I don’t do it as a matter of course.
- deleted 5y ago[deleted]
- fortuna86 5y agoRight, but I have a few people's numbers that can then spread out widely if you ask "hey do you know x's number?" etc.
- thakoppno 5y agoI monitored the Facebook amateur radio club’s repeater during the outage to see if they utilized it in any capacity like that. I didn’t hear it used that way.
- the-dude 5y agoIsn't commercial use of amateur radio prohibited?
- throwanem 5y agoYes, and so is encryption. Either would rule it out for this kind of crisis. That said, I don't know what's so wrong with telephones.
- fknorangesite 5y ago> That said, I don't know what's so wrong with telephones. Do you have all your coworkers' personal phone numbers?
- zrail 5y agoSure they are ...on.... Facebook oh I see the problem now.
- bityard 5y agoI don't... but why would I? My manager has my phone number, as does HR, and I imagine this is a standard situation at nearly every company.
- mulmen 5y agoI have carried a company provided on-call cell phone.
- htrp 5y agoDon't forget that all of those numbers are only available via the internal directory which just went down....
- smallerfish 5y agoSomehow Facebook have a SOC2 (wonder who their auditor is and how much they paid them?) Part of a SOC2 audit is demonstrating that you have a good BCP, and part of a good BCP is demonstrating that you have fallback communication mechanisms available. So, either their BCP was out of date, or possibly they just ignored it during the moment of crisis.
- kristjansson 5y agoYup, it's not really about remote hands in datacenters (which of course have been staffed continuously), it's about losing 'run over to their desk' and 'shout across the office' as a viable communication channels in a crisis when literally all others have failed.
- not2b 5y agoIt means that for certain functions you need people on site, so someone can physically get to the server or router that is causing the problem. It doesn't mean you suddenly need everyone on site. It may also mean that you need more decentralization. They put all their eggs in one basket, so they took down Instagram, Messenger, and WhatsApp as well as Facebook, as well as their internal network, their building security, everything. Seems that they need to look at redesign to create more resiliency even if some efficiency is lost, so they don't lose everything again at some point.
- treesknees 5y agoIs this what happens when product engineering runs the company? In all of my previous jobs, the production customer-facing systems have been separated from the internal day-to-day IT systems. Beyond potentially sharing a physical building/dc, IT and engineering could in no way interfere with each other.
- Jare 5y agoIf your company dogfoods their own products then this can happen. I would expect the value of dogfooding to be massively larger than the cost of internal losses in cases like this. But it's good to remember to have a plan B.
- treesknees 5y agoI'm not sure I agree. It's one thing to treat IT as a customer for a product or service your company offers. I don't doubt that the business also uses FB messenger to communicate for example. But FB as a consumer product doesn't offer DNS or SAML or any of these other critical services, so why would IT share infrastructure for those? That's not dogfooding.
- htrp 5y agoIts a very efficient single point of failure
- tomp 5y agoHow would people in the office talking to each other help resolve an issue in the data center?
- alecbz 5y agoPeople need to talk to each other to quickly debug and understand what the problem even is. To understand that the fix is a change to equipment in a data center, and on top of that what that change needs to be, how to make that change safely (though in this situation it's hard to imagine making things worse).
- deleted 5y ago[deleted]
- toomuchtodo 5y agoYou're not. The process that allowed this to occur was broken, and critical changes should only occur when you've confirmed someone is on site and available in the event you need a crash cart hooked up to a piece of gear (or perhaps automated rollbacks in the event of an unexpected state I suppose is the new hotness with software defined everything). If you're making a change where you're unsure if someone can perform a physical recovery, you're flying without a chute. It works most of the time until it doesn't. (infra/physical colocation/hosting/sysadmin in a previous life for ~20 years) EDIT: @mprovost (HN is throttling me, so I have to reply to your comment here): Absolutely. It's a combination of experience, context, and wisdom. I myself have made changes that I believed were trivial but, because I lacked context, data loss or an outage occurred. These are painful lessons to learn. A culture of psychological safety and commitment to continual process improvement are key, with shared responsibility between contributors and the org. You are going to fail, but you shouldn't get cold sweats when you do as long as you did your best to derisk.
- mprovost 5y agoThe hard part is determining with 100% accuracy what is a "critical change". Many of these incidents start with someone making a change that "should be" safe. So you either have to make all your changes with a crash cart nearby, or trust in your ability to recover from a bad change remotely.
- ruairidhwm 5y agoFor my own learning, what do you mean by a 'crash cart' here?
- renewiltord 5y agoA computer on a movable cart that is correctly provisioned to get direct access to a server in a rack in a datacenter. Nowadays just an appropriate laptop on a desk with casters.
- trashcan 5y agoA crash cart used to be a literal cart in the datacenter with a monitor and keyboard to plug into a rack-mounted server to recover access. Not sure if there is something more modern now.
- deleted 5y ago[deleted]
- lanza 5y agoThis isn't new. This is clickbait. This announcement was made months ago but they reposted it today to get people to react the way you reacted.
- PragmaticPulp 5y ago> Seem like a massive overreaction. Everything else I’m reading says that January 2022 was always the planned return-to-office date ( https://www.cnbc.com/2021/08/12/facebook-delays-return-to-office-until-january-2022-for-us-some-international-employees.html https://www.cnbc.com/2021/08/12/facebook-delays-return-to-of... ) The claim that this was related to the recent outage doesn’t match all of the pre-outage articles that also point to the January 2022 date. Unless someone has evidence that this is a recent change, I think this is a case of heavy editorialization by a journalist eager to capitalize on the combined anger toward Facebook and return-to-office plans.