10 ms·
He just deleted all his updates. user: https://old.reddit.com/user/ramenporn https://old.reddit.com/user/ramenporn some messages: * This is a global outage
by guidopallemans 5y ago
He just deleted all his updates.
user:
https://old.reddit.com/user/ramenporn https://old.reddit.com/user/ramenporn
some messages:
* This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team).
* Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls.
* Update 1440 UTC: \
As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.
- yholio 5y agoEssentially, they locked themselves out with an uninspired command line at the exact moment the datacenter was being hijacked by ape-people. Yup, corporate comms won't love these status updates.
- wtf-is-ur-prblm 5y agoSorry, are you referring to data center technicians as “ape people”?
- korethr 5y agoI mean, when I last worked in a NOC, we used to call ourselves "NOC monkeys", so yeah. IF you're in the NOC, you're a NOC monkey, if you're on the floor, you're a floor monkey. And so on.
- meowface 5y agoSame with "SOC monkeys". (Which carries the additional pun of sounding like the "sock monkey" toy.)
- z-nexx 5y agoAs a former data center technician, I wouldn't say it's too far off
- ticklemyelmo 5y agoBut we're all ape people.
- samstave 5y agohttps://i.imgur.com/O4yEget.png https://i.imgur.com/O4yEget.png
- samstave 5y agoAre you fucking kidding me? We even had a site and operation for a long while called: "NOC MONKEY .DOT ORG" We called all of ourselves NOC MONKEYS. [[Remote Hands]] Yeah, that was a term used widely. I'm 46. I assume you are < # --- Where were you in 1997 building out the very first XML implementations to replace EDI from AS400s to FTP EDI file retrievals via some of the first Linux FTP servers based in SV? I was there? Remember LinuxCare?
- eska 5y agoAre you ok, Sir?
- ShamelessC 5y agoWeren't able to get their ego-fill on facebook like normally.
- ogma 5y agoJust a typical long-time HN poster.
- Narushia 5y agoThe 1440 UTC update is also archived on the Wayback Machine: https://web.archive.org/web/20211004171424/https://old.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfda42z/ https://web.archive.org/web/20211004171424/https://old.reddi... And archive.today: https://archive.ph/sMgCi https://archive.ph/sMgCi
- Ueland 5y agoAnd there his account went poof, thanks for archiving.
- rodgerd 5y agoIf it was actually someone in Facebook, their job is gone by now, too.
- treesknees 5y agoThey were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the people who know the router configuration speaks to an organization that hasn't actually thought through all its failure modes. People aren't going to like that. It's not uncommon to have the datacenter techs with access and the actual software folks restricted, but that being the reason one of the most popular services in the world has been down for nearly 3 hours now will raise a lot of questions. Edit: I also hope this doesn't damage prospects for more Work From Home. If they couldn't get anyone who knew the configuration in because they all live a plane ride away from the datacenters, I could see managers being reluctant to have a completely remote team for situations where clearly physical access was needed.
- fidesomnes 5y ago> I hope they don't lose their job. I do! Oh well!
- swayson 5y ago> I hope they don't lose their job. FB has such poor integrity, I'd not be surprised if they take such extreme measures.
- polote 5y ago> an organization that hasn't actually thought through all its failure modes Thinking about any potential things that can happen is impossible
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- meragrin_ 5y agoThe account has been deleted as well.
- DaiPlusPlus 5y agoWhat are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being helpful in their outreach to my users in the wider community.
- handmodel 5y agoSeems reasonable that at a company of 60k, with hundreds who specialize in PR, you do not want a random engineer making the choice himself to be the first to talk to the press by giving a PR conference on a random forum.
- ric2b 5y agoFacebook is well known for having really good PR, if they go after this guy for sharing such basic info that's yet another example of their great PR teams.
- OskarS 5y agoHonestly, from a PR perspective, I’m not sure it’s so bad. Giving honest updates showing Facebook hard at work is certainly better PR for our kind of crowd than whatever actual Facebook PR is doing.
- dschiavu 5y agoIt's time to decentralize and open up the Internet again, as it once was (ie. IRC, NNTP and other open protocols) instead of relying on commercial entities (Google, Facebook, Amazon) to control our data and access to it.
- mlindner 5y agoI'll throw in Discord into that mix, the thing that basically mostly killed IRC. Which is yet again centralized despite pretending that it is not centralized.