13 ms·
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get peo
by kiernanmcgowan 5y ago
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
- secondcoming 5y agoUnlikely, PagerDuty was invented for this kind of thing
- kiernanmcgowan 5y agoOh I'm sure everyone knows whats wrong, but how am I supposed to send an email, find a coworkers phone number, get the crisis team on video chat etc etc if all of those connections rely on the facebook domain existing?
- ralphm 5y agoHence the suggestion for PagerDuty. It handles all this, because responders set their notification methods (phone, SMS, e-mail, and app) in their profiles, so that when in trouble nobody has to ask those questions and just add a person as a responder to the incident.
- korethr 5y agoYes, but Facebook is not a small company. Could PagerDuty realistically handle the scale of notifications that would be required for Facebook's operations?
- Spooky23 5y agoSure. As long as you plan for disaster. The place where I worked had failure trees for every critical app and service. The goal for incident management was to triage and have an initial escalation for the right group within 15 minutes. When I left they were like 96% on target overall and 100% for infrastructure.
- robalfonso 5y agoEven if it can’t, it’s trivial to use it for an important subset, ie is Facebook.com down, is the ns stuff down etc. So there is an argument to be made for still using an outside service as a fallback
- jfrunyon 5y agoI guarantee you that every single person at Facebook who can do anything at all about this, already knows there's an issue. What would them receiving an extra notification help with?
- robalfonso 5y agoWe kind of got off topic, I was arguing that if you were concerned about internal systems being down (including your monitoring/alerting) something like pager duty would be fine as a backup. Even at huge scale that backup doesn’t need to watch everything. I don’t think it’s particularly relevant to this issue with fb. I suspect they didn’t need a monitoring system to know things were going badly.
- anigbrowl 5y agoSure, if you're... - not arrogant - or complacent - haven't inadvertently acquired the company - know your tech peers well enough to have confidence in their identity during an emergency - do regular drills to simulate everything going wrong at once Lots of us know what should be happening right now, but think back to the many situations we've all experienced where fallback systems turned into a nightmarish war story, then scale it up by 1000. This is a historic day, I think it's quite likely that the scale of the outage will lead to the breakup of the company because it's the Big One that people have been warning about for years.
- antoinealb 5y agoPagerDuty does not solve some of the problems you would have at FB's scale, like how do you even know who to contact ? And how do they login once they know there is a problem ?
- still_grokking 5y agoYou mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.
- oconnor663 5y agoI think the issue there is that in exchange for solving the "one fat finger = outage" problem, you lose the ability to update the server fleet quickly or consistently.
- badrequest 5y agoWhat do you think all those superfluous chat apps were for?
- l9i 5y agoI can assure you that Google has a procedure in place for that.
- knorker 5y agoFor those who don't know who he is: l9i would know this. Just clarifying that this is not an Internet nobody guessing.
- sam_lowry_ 5y agoHe is still an anonymous dude to me.
- deleted 5y ago[deleted]
- danhak 5y agoHN Profile -> Personal Website -> LinkedIn -> Over 10 years experience as Google Site Reliability Engineer
- justinzollars 5y agoWhat do you think will be the impact on WFH and office requirements?
- threevox 5y agoLOL - score one against building out all tooling internally (a la Amazon and apparently Facebook too)
- vineyardmike 5y agoThe rate at which some amazon services lately go done because other AWS services went down proves that this is an unsustainable house of cards anyways.
- kevin_thibedeau 5y agoNetflix knows how to build on top of a house of cards.
- bickeringyokel 5y agoThere's a joke here somewhere about how bad the final season was
- rStar 5y agotime to start working at your mfing desk again, johnson
- gocartStatue 5y agoThey supposedly can't enter facebook office right now. Their cards don't work.
- eskathos 5y agosource?
- jjulius 5y agoNYT reporter on Twitter. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
- BrianKamrany 5y agoSheera Frenkel @sheeraf Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
- eskathos 5y ago"Something went wrong. Try reloading." its not loading for me. could you say what it said?
- lynx234 5y agoSaw this earlier: https://twitter.com/disclosetv/status/1445100931947892736 https://twitter.com/disclosetv/status/1445100931947892736
- BrianKamrany 5y agoDisclose.tv @disclosetv JUST IN - Facebook employees reportedly can't enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)
- okwubodu 5y agoI don’t know how true it is but a few reports claim employees can’t get into the building with their badges.
- deleted 5y ago[deleted]
- cududa 5y agoI remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"
- Bluecobra 5y agoIn case of emergency, break glass... ...the doors are glass right?
- cududa 5y agoZucks personal conference room has 3 glass walls, so I’ve been amusing myself imagining him just throwing a chair through one of the walls
- deleted 5y ago[deleted]
- hellbannedguy 5y agoI don't think he has the strength.
- thrwyoilarticle 5y agoDo they (you?) call him that at FB?
- xapata 5y agoYes, "Zuck".
- strulovich 5y agoThose communications are done over irc at FB for exactly this purpose.