8 ms·

Employees given three months to return to Facebook office

133 points by jmfldn 5y ago

sidhanthp 5y ago
NGL, I have no clue what this article is saying. Does this mean that people who were approved for remote work will be forced to come back into the office? Or that people who are supposed to come into the office Jan 2022 will continue to have to come into the office in Jan 2022 (i.e., no change).
- codemac 5y ago
  The article is saying the former, the actual corporate policy is the latter.
ambar123 5y ago
Facebook.com is bunch of douchebag
ambar123 5y ago
Facebook.com is just bunch of douchebag
newfonewhodis 5y ago
Move fast and break things and blame remote work so managers can feel useful again.
- ashtonkem 5y ago
  Yes, although to be fair a lot of managers don’t want to return to an office either, myself included.
LurkingPenguin 5y ago
Seems like a convenient excuse to walk back WFH. Surely the vast majority of Facebook employees would have been as helpless as Facebook's users to fix this outage. What's preventing Facebook from continuing to offer WFH to many/most employees, with a minority of critical staff required to be on-site and compensated for it if necessary?
iso1210 5y ago
Take it with a pinch of salt as it's from the daily mail, but "Some reports claimed company keycards were also knocked offline, meaning employees were unable to gain access to the offices or the server rooms - with some claiming employees were forced to, quite literally, break in." Well that would work well for people working from an office.
- htrp 5y ago
  I assume at that point once you're inside, someone could just call and ask you to prop the door open?
  - brian_cloutier 5y ago
    I assume another component of the problem is: why would you have the phone number of any of your co-workers? Facebook runs on messenger!
    - touisteur 5y ago
      Eeeh outlook directory synced locally for at least all your contacts, with professional mobile number? Too 1990s?
  - bluGill 5y ago
    Most places I've been have been badge in - badge out. You can still prop the door open, but first you have to find that hidden fire emergency override button.
jmfldn 5y ago
Seem like a massive overreaction. Keep the sys admins on-site to manage your on prem gear but they should have been doing that anyway?! Surely when you're running your own servers for anything of any scale you need actual people nearby 24/7. Am I missing something?
- kiernanmcgowan 5y ago
  Remember that a huge part of this fall out was that FB staff couldn't talk to each other in a moment of crisis. Not just the people trying to manage the crisis, but the entire company faced a work stoppage in one way or another. If you do the cost benefit of allowing people to work from home vs the cost of this one outage, I bet the outage is more expensive.
  - jmfldn 5y ago
    Surely they need easy comms fallback options that are independent of their own tech / infra. Even if they were on site this would still be pretty terrible to have everything so coupled right?
    - CydeWeys 5y ago
      > Surely they need easy comms fallback options that are independent of their own tech / infra. And you know what's excellent for that ... literal word of mouth. I'm back in the office with 5 of my coworkers right now, and when we have issues we can just talk about them, zero technology needed.
      - jmfldn 5y ago
        Trut, that helps. However, even medium sized companies are often spread across multiple cities and timezones so it only helps to an extent. Having a single point of failure for literally everything seems to be a pretty large elephant in the room here.
    - fortuna86 5y ago
      Surely each co-worker had the phone number of their colleagues ?
rvz 5y ago
Nothing lasts 'forever'.
- JohnClark1337 5y ago
  HN disagrees
- dyingkneepad 5y ago
  Except for "companies exploiting their employees as much as they legally can and often even more".
m0sa 5y ago
At this point, would anybody be surprised if the outage was plann(t)ed by FB leadership?
- chartpath 5y ago
  Because FB doesn't have a habit of leaving money on the table?
AndrewGaspar 5y ago
There is no policy change - RTO in January 2022 was already the policy. https://www.cnbc.com/2021/08/12/facebook-delays-return-to-office-until-january-2022-for-us-some-international-employees.html https://www.cnbc.com/2021/08/12/facebook-delays-return-to-of...
- tylerhou 5y ago
  Nothing in your article contradicts the OP -- a RTO for non-remote workers being delayed doesn't mean that all workers had to RTO. (But now it seems like FB may have changed that policy.) https://www.cnbc.com/2021/06/09/facebook-will-let-all-employees-who-can-work-remotely-to-request-full-time-remote-work.html https://www.cnbc.com/2021/06/09/facebook-will-let-all-employ...
  - AndrewGaspar 5y ago
    The OP seems to only source its claims to the Daily Mail[0], which itself does not contain any new claims except for this: > The need to request formal permission also applies to employees offered pay cuts in return for staying at home in cases where the local cost of living is lower than around the office where they are usually based. To start, this claim is unsourced. I'm having a hard time interpreting this claim, but it seems like _maybe_ it's saying that FB is requesting individuals who were already approved for remote work to RTO. But it also seems like they're just describing the existing process, which is that if you want to remain remote beyond January 2022, you must be explicitly approved for remote work. Anyway, I can't speak for everyone, but as a remote approved FB employee, I've received no such request to RTO. [0] https://www.dailymail.co.uk/news/article-10064865/Facebook-gives-staff-three-month-deadline-desks.html https://www.dailymail.co.uk/news/article-10064865/Facebook-g...
    - midasuni 5y ago
      Daily Mail and most of the U.K. press and government has spent a lot of effort demonising remote working. The same people who own interest in commercial real estate and companies like Starbucks. No idea why they would be so concerned with pushing their agenda.
0x500x79 5y ago
This seems like a very large change as a response to the outage. What about employees that have already migrated to different locations? They are expected to move back within 3 months with the housing market like it is right now? The subset of people that could fix the issue that caused the outage was probably .1% of the engineers at FB. Was FB looking for an excuse to end those rules?
- tharkun__ 5y ago
  Why would you migrate to a different location unless you have a contract that state that you have the right to work from anywhere (or laying out the conditions, such as maximum time zone differences and such)? Anything that is simply a current policy at a company basically has to be ignored. It's a nice to have but if you "can't live without that", go look for another job or adjust otherwise. And even stuff that _is_ in your contract is not a full on guarantee either as contracts can be amended. Which you still have to sign but they will make it very hard for you not to sign those amendments. But at least it gives you some warning when that happens. I.e. you can refuse to sign and then you have time to look for another job while they figure out how to get you fired.
  - lowbloodsugar 5y ago
    Or you could just say "I'm looking at my future, and I'm thinking I could grow more by joining a different team." And then just drop "working from home" as something that impacts your quality of life, and you're good. This is a sellers market.
  - kenward 5y ago
    "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem." Technically you're correct. If only a handful of employees migrated, they could probably be let go pretty easily. No skin off Facebook's back. Now if hundreds or thousands of employees migrated, Facebook needs to make a decision whether enforcing the RTO or not will hurt them more. Regardless of technicalities, demanding a non-significant percentage of your workforce to uproot their lives is not an easy ask (or great look).
democracy 5y ago
A CTO of a startup I joined said "we are moving fast and break things". Sounded great! But after the first outage they were looking for someone to blame and quickly revoking production access)))
- christopherwxyz 5y ago
  Zuck changed the motto in 2014 to be "Move fast with stable infrastructure." It's obvious that the first mentality was a bit reckless and only optimized growth at all costs. At some point you have to pay for that debt. https://www.wired.com/2014/04/zuckerberg-f8-interview/ https://www.wired.com/2014/04/zuckerberg-f8-interview/
- tombert 5y ago
  Yeah, I think a lot of managers and executives want the social credit of "we're totally open to experimentation and trying new things!" because that makes them sound cool. I don't think they're "lying" since that requires intent, but I think a lot of the time that's also not a 100% honest sentiment either. They are nominally open to experimentation...so long as you know the experiment is going to work. Not going to say the company name (though you might be able to figure it out based on my post history, I ask that you don't paste it here if you figure it out), but I was working for a bigass megacorporation where management told me upon being hired "we don't play the blame game here, things are going to break". That sounded cool, and about three months later, I committed some "risky" code (which passed the PR review process evidently), and it broke production for about two hours. I then got VoIP call with that same manager who yelled at me telling me that I need to be more careful because this is unacceptable. Presumably this manager was frustrated and just needed someone to blame, so fair enough, but it's not something I ever really forgave if I'm being honest.
  - jostmey 5y ago
    Haha! An experiment you know will work is not an experiment!
    - tombert 5y ago
kristjansson 5y ago
The causal relation to WFH is ... tenuous ... but now can see how losing all internal comms to an incident could make managers more nervous about WFH. Of course announcing a policy change now is closing barn doors behind horses.
- lamontcg 5y ago
  When I was at Amazon in 2001 outages were done via everyone sitting at their desks on a POTS conference call. The engineers working the issue would be spread out around building at least, and the people who were actually on-call would generally be at home. Remote hands were often in a datacenter that was on the other side of continent. The whole idea of piling everyone into a "war room" doesn't really scale anyway. The bigger problem here was just no out of band comms.
  - kristjansson 5y ago
    Yeah, knocking out their backup IRC (or maybe it was the authentication to the backup IRC?) along with everything else was the key.
    - duskwuff 5y ago
      Word-of-mouth I've heard is that the backup IRC server was off-site working fine. The problem was that, because it was only used in emergencies and wasn't hooked into FB's central authentication, a lot of employees simply didn't remember how to log in to it.
maccard 5y ago
This headline is misleading at best, and outright false at worst. It's technically correct, but the article doesn't actually say that the the end of the WFH forever is _because_ of the outage, just that they are going back.
- modeless 5y ago
  It's not even technically correct. Any employee who doesn't want to return has the option to request full time remote work.
lbriner 5y ago
Isn't this another example of "how does a company with 1000s of the most able employees not notice a massive risk to business continuity sitting there waiting to happen"? DNS hosted on your own registrar, within your own data center for a whole host of domains. Somehow integrating your office security so that it also depends on that same DNS and running BGP updates without an emergency "undo" plan? I hate to over-simplify but Single point of failure is the first thing you ever learn about Business Continuity. We shouldn't need to learn these lessons from a big mistake, that's what college is for.
- fksadfji12 5y ago
  Spoken like a typical MBA freshman who thinks he has all the answers lol
- marcosdumay 5y ago
  Nobody expects a BGP issue. It's main weapon is surprise. But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.
  - midasuni 5y ago
    I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.
- dijit 5y ago
  Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight. And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
  - tharkun__ 5y ago
    Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back. The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time. Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like: External power goes out for whatever reason (don't remember) Batteries take over Diesel generators start up Batteries keep draining Diesels are spinning All Servers shut down What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.
stuaxo 5y ago
Covid still isn't over.
- jaywalk 5y ago
  When will you consider Covid to be over? What's the trigger?
  - micromacrofoot 5y ago
    it will be endemic, but we need kids to get vaccines at least
  - spookthesunset 5y ago
    You’ll never get an answer to this. Covid is over when you move on from it.
    - ImPostingOnHN 5y ago
      looks like they got an answer to it
  - smileysteve 5y ago
    Hospital ICUs being below 85% occupancy. A test positivity rate below 2.5%, where contact tracing definitively works. According to the April 16, 2020 3 Phase Reopening Plan for the United States, each state should follow, "14 days of declining cases[/positivity rate] per phase and hospital capacity that exists in case you have a rebound" were the criteria for each phase; The 8 worst states for Delta variant have declining positivity rates starting about 3 weeks ago.
    - lamontcg 5y ago
      Really its all about ICUs. The virus is never going away and will spread epidemically again in the future. At some point the population has enough T-cells though that the infection fatality rate and infection hospitalization rate is that of influenza or colds, and even though it spreads people don't wind up in the hospital that much more than usual for cold and flu season. And we're still probably going to have vaccine stragglers for years winding up finally getting their Herman Cain Awards at some level so it'll never be zero.
NotAnOtter 5y ago
Before I even read this thread it was pretty obvious this meant they never planned on offering WFH forever. If they wanted some on site employees to handle these kind of outages, they could just pay their top 1% to come to office or move within 15 minutes of the office or something.
- tombert 5y ago
  That's my thought as well. These corporations get a ton of free publicity for being "the cool companies that are going to make remote permanent." Then all they have to do to more or less save face is wait for something bad to happen, show some vague graphs about how this could have been prevented if we weren't remote, and that they are forced to implement a RTO order.
vkou 5y ago
The correct way to read this title is 'Facebook gives itself three months to push RTO back another few months.'
yepthatsreality 5y ago
Everyone I talk to in tech has a RTO date of January 2022. However they also made fully and partially remote an official offering (I say official because of all the tech companies I’ve worked for it was never out of the question, just arranged and not offered). That’s fine, but most of my coworkers are now either partial or fully remote (to the point where we have almost no existing NYC team that would come into the office). So what does this January 2022 RTO really mean for any company?
- michaelcampbell 5y ago
  The firm I currently work for (in the 4 digits employee size) has pushed theirs back to Mar-2022. Many recruiter emails I have received are pushing "100% remote" - for how long I can't say as I don't answer them, but it feels like RTO is becoming more in the minority.
- throwawayboise 5y ago
  I had a RTO of August 2021. Mandated vaccine in an exchange for no masks (among other things, such as getting to remain employed). Well they reneged on the no mask rule. I can't undo my vaccine, so I'm working from home until they can provide a safe work environment.
Sebb767 5y ago
Assuming they can get into the building ;-) (context: https://mobile.twitter.com/sheeraf/status/1445099150316503057 https://mobile.twitter.com/sheeraf/status/144509915031650305... )
softwaredoug 5y ago
> Now, Facebook is pointing the finger at remote working. I’m no Facebook fan, but is there actual evidence of “Facebook pointing the finger at remote working”?
sylens 5y ago
3 months to collect more documents to hand over to federal regulators before allowing yourself to be poached by other tech firms
throw8932894 5y ago
Good! Usual digital nomad spots are getting too crowded. I welcomed covid initially, it normalised remote work. But it drives cost of nice places too high.
noobermin 5y ago
This is probably somewhat insensitive but given everything recently, if this leads some workers from quitting facebook, it probably will be good for them (the workers that is), or at the very least, a silver lining is easy to see. I don't think or know if fb will fall or decay (or ever will) but after watching their company get so many beatings and if those people are lucky enough to have options, it's not hard to see people wanting to get out. And good for them.
rickspencer3 5y ago
I've been wondering how these huge companies with town-sized campuses, and all of that capital investment would respond. I guess it would take a really special kind of company to recognize that these campuses are a sunk cost and pivot to true remote.
lamontcg 5y ago
"never let a crisis go to waste"
m-p-3 5y ago
Non-AMP link: https://www.techradar.com/uk/news/facebook-ends-wfh-forever-rule-after-mega-outage https://www.techradar.com/uk/news/facebook-ends-wfh-forever-...
- cheshire137 5y ago
  I hadn’t even realize the HN link was to an AMP page. Your comment made me finally get Amplosion for my iPad. I’d been on the fence before because I was thinking, “I never load AMP pages, I don’t see amp. in my URL bar much.” I didn’t consider that a site could just have /amp in the URL path and not a specific subdomain.
roflchoppa 5y ago
God damn it this is gonna ruin traffic on highway 84 again, it was so nice not to have traffic on the Dumbarton.

133 points by jmfldn 5y ago

NGL, I have no clue what this article is saying. Does this mean that people who were approved for remote work will be forced to come back into the office? Or that people who are supposed to come into the office Jan 2022 will continue to have to come into the office in Jan 2022 (i.e., no change).

codemac 5y ago
The article is saying the former, the actual corporate policy is the latter.

ambar123 5y ago

Facebook.com is bunch of douchebag

ambar123 5y ago

Facebook.com is just bunch of douchebag

newfonewhodis 5y ago

Move fast and break things and blame remote work so managers can feel useful again.

ashtonkem 5y ago
Yes, although to be fair a lot of managers don’t want to return to an office either, myself included.

LurkingPenguin 5y ago

Seems like a convenient excuse to walk back WFH. Surely the vast majority of Facebook employees would have been as helpless as Facebook's users to fix this outage. What's preventing Facebook from continuing to offer WFH to many/most employees, with a minority of critical staff required to be on-site and compensated for it if necessary?

iso1210 5y ago

Take it with a pinch of salt as it's from the daily mail, but "Some reports claimed company keycards were also knocked offline, meaning employees were unable to gain access to the offices or the server rooms - with some claiming employees were forced to, quite literally, break in." Well that would work well for people working from an office.

htrp 5y ago
I assume at that point once you're inside, someone could just call and ask you to prop the door open?
- brian_cloutier 5y ago
  I assume another component of the problem is: why would you have the phone number of any of your co-workers? Facebook runs on messenger!
  - touisteur 5y ago
    Eeeh outlook directory synced locally for at least all your contacts, with professional mobile number? Too 1990s?
- bluGill 5y ago
  Most places I've been have been badge in - badge out. You can still prop the door open, but first you have to find that hidden fire emergency override button.

jmfldn 5y ago

Seem like a massive overreaction. Keep the sys admins on-site to manage your on prem gear but they should have been doing that anyway?! Surely when you're running your own servers for anything of any scale you need actual people nearby 24/7. Am I missing something?

kiernanmcgowan 5y ago
Remember that a huge part of this fall out was that FB staff couldn't talk to each other in a moment of crisis. Not just the people trying to manage the crisis, but the entire company faced a work stoppage in one way or another. If you do the cost benefit of allowing people to work from home vs the cost of this one outage, I bet the outage is more expensive.
- jmfldn 5y ago
  Surely they need easy comms fallback options that are independent of their own tech / infra. Even if they were on site this would still be pretty terrible to have everything so coupled right?
  - CydeWeys 5y ago
    > Surely they need easy comms fallback options that are independent of their own tech / infra. And you know what's excellent for that ... literal word of mouth. I'm back in the office with 5 of my coworkers right now, and when we have issues we can just talk about them, zero technology needed.
    - jmfldn 5y ago
      Trut, that helps. However, even medium sized companies are often spread across multiple cities and timezones so it only helps to an extent. Having a single point of failure for literally everything seems to be a pretty large elephant in the room here.
  - fortuna86 5y ago
    Surely each co-worker had the phone number of their colleagues ?

rvz 5y ago

Nothing lasts 'forever'.

JohnClark1337 5y ago
HN disagrees
dyingkneepad 5y ago
Except for "companies exploiting their employees as much as they legally can and often even more".

m0sa 5y ago

At this point, would anybody be surprised if the outage was plann(t)ed by FB leadership?

chartpath 5y ago
Because FB doesn't have a habit of leaving money on the table?

AndrewGaspar 5y ago

There is no policy change - RTO in January 2022 was already the policy. https://www.cnbc.com/2021/08/12/facebook-delays-return-to-office-until-january-2022-for-us-some-international-employees.html https://www.cnbc.com/2021/08/12/facebook-delays-return-to-of...

tylerhou 5y ago
Nothing in your article contradicts the OP -- a RTO for non-remote workers being delayed doesn't mean that all workers had to RTO. (But now it seems like FB may have changed that policy.) https://www.cnbc.com/2021/06/09/facebook-will-let-all-employees-who-can-work-remotely-to-request-full-time-remote-work.html https://www.cnbc.com/2021/06/09/facebook-will-let-all-employ...
- AndrewGaspar 5y ago
  The OP seems to only source its claims to the Daily Mail[0], which itself does not contain any new claims except for this: > The need to request formal permission also applies to employees offered pay cuts in return for staying at home in cases where the local cost of living is lower than around the office where they are usually based. To start, this claim is unsourced. I'm having a hard time interpreting this claim, but it seems like _maybe_ it's saying that FB is requesting individuals who were already approved for remote work to RTO. But it also seems like they're just describing the existing process, which is that if you want to remain remote beyond January 2022, you must be explicitly approved for remote work. Anyway, I can't speak for everyone, but as a remote approved FB employee, I've received no such request to RTO. [0] https://www.dailymail.co.uk/news/article-10064865/Facebook-gives-staff-three-month-deadline-desks.html https://www.dailymail.co.uk/news/article-10064865/Facebook-g...
  - midasuni 5y ago
    Daily Mail and most of the U.K. press and government has spent a lot of effort demonising remote working. The same people who own interest in commercial real estate and companies like Starbucks. No idea why they would be so concerned with pushing their agenda.

0x500x79 5y ago

This seems like a very large change as a response to the outage. What about employees that have already migrated to different locations? They are expected to move back within 3 months with the housing market like it is right now? The subset of people that could fix the issue that caused the outage was probably .1% of the engineers at FB. Was FB looking for an excuse to end those rules?

tharkun__ 5y ago
Why would you migrate to a different location unless you have a contract that state that you have the right to work from anywhere (or laying out the conditions, such as maximum time zone differences and such)? Anything that is simply a current policy at a company basically has to be ignored. It's a nice to have but if you "can't live without that", go look for another job or adjust otherwise. And even stuff that _is_ in your contract is not a full on guarantee either as contracts can be amended. Which you still have to sign but they will make it very hard for you not to sign those amendments. But at least it gives you some warning when that happens. I.e. you can refuse to sign and then you have time to look for another job while they figure out how to get you fired.
- lowbloodsugar 5y ago
  Or you could just say "I'm looking at my future, and I'm thinking I could grow more by joining a different team." And then just drop "working from home" as something that impacts your quality of life, and you're good. This is a sellers market.
- kenward 5y ago
  "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem." Technically you're correct. If only a handful of employees migrated, they could probably be let go pretty easily. No skin off Facebook's back. Now if hundreds or thousands of employees migrated, Facebook needs to make a decision whether enforcing the RTO or not will hurt them more. Regardless of technicalities, demanding a non-significant percentage of your workforce to uproot their lives is not an easy ask (or great look).

democracy 5y ago

A CTO of a startup I joined said "we are moving fast and break things". Sounded great! But after the first outage they were looking for someone to blame and quickly revoking production access)))

christopherwxyz 5y ago
Zuck changed the motto in 2014 to be "Move fast with stable infrastructure." It's obvious that the first mentality was a bit reckless and only optimized growth at all costs. At some point you have to pay for that debt. https://www.wired.com/2014/04/zuckerberg-f8-interview/ https://www.wired.com/2014/04/zuckerberg-f8-interview/
tombert 5y ago
Yeah, I think a lot of managers and executives want the social credit of "we're totally open to experimentation and trying new things!" because that makes them sound cool. I don't think they're "lying" since that requires intent, but I think a lot of the time that's also not a 100% honest sentiment either. They are nominally open to experimentation...so long as you know the experiment is going to work. Not going to say the company name (though you might be able to figure it out based on my post history, I ask that you don't paste it here if you figure it out), but I was working for a bigass megacorporation where management told me upon being hired "we don't play the blame game here, things are going to break". That sounded cool, and about three months later, I committed some "risky" code (which passed the PR review process evidently), and it broke production for about two hours. I then got VoIP call with that same manager who yelled at me telling me that I need to be more careful because this is unacceptable. Presumably this manager was frustrated and just needed someone to blame, so fair enough, but it's not something I ever really forgave if I'm being honest.
- jostmey 5y ago
  Haha! An experiment you know will work is not an experiment!
  - tombert 5y ago

kristjansson 5y ago

The causal relation to WFH is ... tenuous ... but now can see how losing all internal comms to an incident could make managers more nervous about WFH. Of course announcing a policy change now is closing barn doors behind horses.

lamontcg 5y ago
When I was at Amazon in 2001 outages were done via everyone sitting at their desks on a POTS conference call. The engineers working the issue would be spread out around building at least, and the people who were actually on-call would generally be at home. Remote hands were often in a datacenter that was on the other side of continent. The whole idea of piling everyone into a "war room" doesn't really scale anyway. The bigger problem here was just no out of band comms.
- kristjansson 5y ago
  Yeah, knocking out their backup IRC (or maybe it was the authentication to the backup IRC?) along with everything else was the key.
  - duskwuff 5y ago
    Word-of-mouth I've heard is that the backup IRC server was off-site working fine. The problem was that, because it was only used in emergencies and wasn't hooked into FB's central authentication, a lot of employees simply didn't remember how to log in to it.

maccard 5y ago

This headline is misleading at best, and outright false at worst. It's technically correct, but the article doesn't actually say that the the end of the WFH forever is _because_ of the outage, just that they are going back.

modeless 5y ago
It's not even technically correct. Any employee who doesn't want to return has the option to request full time remote work.

lbriner 5y ago

Isn't this another example of "how does a company with 1000s of the most able employees not notice a massive risk to business continuity sitting there waiting to happen"? DNS hosted on your own registrar, within your own data center for a whole host of domains. Somehow integrating your office security so that it also depends on that same DNS and running BGP updates without an emergency "undo" plan? I hate to over-simplify but Single point of failure is the first thing you ever learn about Business Continuity. We shouldn't need to learn these lessons from a big mistake, that's what college is for.

fksadfji12 5y ago
Spoken like a typical MBA freshman who thinks he has all the answers lol
marcosdumay 5y ago
Nobody expects a BGP issue. It's main weapon is surprise. But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.
- midasuni 5y ago
  I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.
dijit 5y ago
Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight. And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
- tharkun__ 5y ago
  Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back. The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time. Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like: External power goes out for whatever reason (don't remember) Batteries take over Diesel generators start up Batteries keep draining Diesels are spinning All Servers shut down What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.

stuaxo 5y ago

Covid still isn't over.

jaywalk 5y ago
When will you consider Covid to be over? What's the trigger?
- micromacrofoot 5y ago
  it will be endemic, but we need kids to get vaccines at least
- spookthesunset 5y ago
  You’ll never get an answer to this. Covid is over when you move on from it.
  - ImPostingOnHN 5y ago
    looks like they got an answer to it
- smileysteve 5y ago
  Hospital ICUs being below 85% occupancy. A test positivity rate below 2.5%, where contact tracing definitively works. According to the April 16, 2020 3 Phase Reopening Plan for the United States, each state should follow, "14 days of declining cases[/positivity rate] per phase and hospital capacity that exists in case you have a rebound" were the criteria for each phase; The 8 worst states for Delta variant have declining positivity rates starting about 3 weeks ago.
  - lamontcg 5y ago
    Really its all about ICUs. The virus is never going away and will spread epidemically again in the future. At some point the population has enough T-cells though that the infection fatality rate and infection hospitalization rate is that of influenza or colds, and even though it spreads people don't wind up in the hospital that much more than usual for cold and flu season. And we're still probably going to have vaccine stragglers for years winding up finally getting their Herman Cain Awards at some level so it'll never be zero.

NotAnOtter 5y ago

Before I even read this thread it was pretty obvious this meant they never planned on offering WFH forever. If they wanted some on site employees to handle these kind of outages, they could just pay their top 1% to come to office or move within 15 minutes of the office or something.

tombert 5y ago
That's my thought as well. These corporations get a ton of free publicity for being "the cool companies that are going to make remote permanent." Then all they have to do to more or less save face is wait for something bad to happen, show some vague graphs about how this could have been prevented if we weren't remote, and that they are forced to implement a RTO order.

vkou 5y ago

The correct way to read this title is 'Facebook gives itself three months to push RTO back another few months.'

yepthatsreality 5y ago

Everyone I talk to in tech has a RTO date of January 2022. However they also made fully and partially remote an official offering (I say official because of all the tech companies I’ve worked for it was never out of the question, just arranged and not offered). That’s fine, but most of my coworkers are now either partial or fully remote (to the point where we have almost no existing NYC team that would come into the office). So what does this January 2022 RTO really mean for any company?

michaelcampbell 5y ago
The firm I currently work for (in the 4 digits employee size) has pushed theirs back to Mar-2022. Many recruiter emails I have received are pushing "100% remote" - for how long I can't say as I don't answer them, but it feels like RTO is becoming more in the minority.
throwawayboise 5y ago
I had a RTO of August 2021. Mandated vaccine in an exchange for no masks (among other things, such as getting to remain employed). Well they reneged on the no mask rule. I can't undo my vaccine, so I'm working from home until they can provide a safe work environment.

Sebb767 5y ago

Assuming they can get into the building ;-) (context: https://mobile.twitter.com/sheeraf/status/1445099150316503057 https://mobile.twitter.com/sheeraf/status/144509915031650305... )

softwaredoug 5y ago

> Now, Facebook is pointing the finger at remote working. I’m no Facebook fan, but is there actual evidence of “Facebook pointing the finger at remote working”?

sylens 5y ago

3 months to collect more documents to hand over to federal regulators before allowing yourself to be poached by other tech firms

throw8932894 5y ago

Good! Usual digital nomad spots are getting too crowded. I welcomed covid initially, it normalised remote work. But it drives cost of nice places too high.

noobermin 5y ago

This is probably somewhat insensitive but given everything recently, if this leads some workers from quitting facebook, it probably will be good for them (the workers that is), or at the very least, a silver lining is easy to see. I don't think or know if fb will fall or decay (or ever will) but after watching their company get so many beatings and if those people are lucky enough to have options, it's not hard to see people wanting to get out. And good for them.

rickspencer3 5y ago

I've been wondering how these huge companies with town-sized campuses, and all of that capital investment would respond. I guess it would take a really special kind of company to recognize that these campuses are a sunk cost and pivot to true remote.

lamontcg 5y ago

"never let a crisis go to waste"

m-p-3 5y ago

Non-AMP link: https://www.techradar.com/uk/news/facebook-ends-wfh-forever-rule-after-mega-outage https://www.techradar.com/uk/news/facebook-ends-wfh-forever-...

cheshire137 5y ago
I hadn’t even realize the HN link was to an AMP page. Your comment made me finally get Amplosion for my iPad. I’d been on the fence before because I was thinking, “I never load AMP pages, I don’t see amp. in my URL bar much.” I didn’t consider that a site could just have /amp in the URL path and not a specific subdomain.

roflchoppa 5y ago

God damn it this is gonna ruin traffic on highway 84 again, it was so nice not to have traffic on the Dumbarton.