36 ms·
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: >As many of you know, DNS for FB services has been affected and this is
by kossTKR 5y ago
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue:
>As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.
User is providing live updates of the incident here:
https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfd4dyv/?context=3 https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...
- cotillion 5y agoSo, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.
- mekatter 5y agoThese are readily available, OpenGear and others have offered them forever. I can't believe fb doesn't have out of band access to their core networking in some fashion. OOB access to core networking is like insurance, rarely appreciated until the house is on fire.
- Sebb767 5y agoIt's quite possible that they have those, but that the credentials are stored in a tool hosted in that datacenter or that the DNS entries are managed by the DNS servers that are down right now.
- mekatter 5y agoYou are probably right but if that is the case, it isn't really out of band and needs another look. I use OpenGear devices with cellular to access our core networking to multiple locations and we treat them as basically an entirely independent deployment, as if it is another company. DNS and credentials are stored in alternate systems that can be accessed regardless of the primary systems. I'm sure the logistics of this become far more complicated as the organization scales but IMHO it is something that shouldn't be overlooked, exactly for outlier events like this. It pays dividends the first time it is really needed. If the accounts of ramenporn are correct, it would be paying very well right now. Out of band access is a far more complicated version of not hosting your own status page, which they don't seem to get right either.
- daper 5y agoOur security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...
- iso1210 5y agoBut you have two independent VPNs right, using different technologies on different internet handoffs in very different parts of your network, right?
- lostapathy 5y agoFundamentally, how is a 2nd independent VPN into your network a different attack surface than a single, well-secured ssh jumphost? When you're using them for narrow emergency access to restore the primary VPN, both are just "one thing" listening on the wire, and it's not like ssh isn't a well-understood commodity.
- iso1210 5y agoZero day sshd vulnerability would be bad. On the other hand if you had to break through wireguard first, and then go through your single well-secured bastion, you'd not only be harder to find, you'd have two layers of protection, and of course you tick the "VPN" box
- lostapathy 5y agoVpn can also have a zero day, and seems about as likely?
- iso1210 5y agoBut if your vpn has a zero day, that lets you get to the ssh server. It's two layers of protection, you'd have to have two zero days to get in instead of one. You could argue it's overkill, but it's clearly more secure
- PeterCorless 5y agoComment now seems to be deleted by user.
- sbierwagen 5y agoThat reddit comment has been deleted.
- ds206 5y agoWell, those comments have been deleted now... I guess someone's boss didn't like the unofficial updates going out? :)
- cwkoss 5y agoUh oh that user deleted their account. Hope they are OK.
- snickersnee11 5y agoJust imagine the amount of stress on this people, hope the money really worth it.
- mov31tmov31t 5y agoIt shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.
- rvnx 5y ago"it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts" Well, you'd be surprised about how one person can bring everything down and/or save the day at Facebook, Cloudflare, Google, Gitlab, etc. Most people are observers/cheerleaders when there is an incident.
- cromka 5y ago> Most people are observers/cheerleaders when there is an incident. Yeah, a typical fight/flight response.
- SomeBoolshit 5y agoOr most people simply don't have anything useful to add or do during an incident.
- ikiris 5y agoTaking all the available slots in the massive gvc warroom ain't much... but its honest work.
- xorcist 5y ago
- larntz 5y agoThis tweet seems to confirm it is a bgp issue... https://twitter.com/GossiTheDog/status/1445063880963674121?s=20 https://twitter.com/GossiTheDog/status/1445063880963674121?s...
- adamredwoods 5y agoCloudflare also confirmed it: https://twitter.com/jgrahamc/status/1445068309288951820 https://twitter.com/jgrahamc/status/1445068309288951820 Also, the Domain name is for sale??? https://whois.domaintools.com/facebook.com https://whois.domaintools.com/facebook.com
- glenneroo 5y agoWeird banner at the top, seems like false advertising as it says a couple lines down: Expires on 2030-03-29
- bombcar 5y agoI suspect it's an automated system triggered by DNS not resolving, and they try to "make an offer" if you follow through.
- adamredwoods 5y agoYou're right, it's misleading, thanks. Other sites (dreamhost, godaddy) don't list it as for sale.
- pmlnr 5y ago> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.
- suyash 5y agofolks with physical access are also denied. source - https://twitter.com/YourAnonOne/status/1445100431181598723 https://twitter.com/YourAnonOne/status/1445100431181598723
- prox 5y agoLet me guess, it is tied to FB systems which are down. That would be hilarious.
- MauranKilom 5y agoFWIW that's not the original source, just some twitter account reposting info shared by someone else. See this sub-thread: https://news.ycombinator.com/item?id=28750888 https://news.ycombinator.com/item?id=28750888
- drdeadringer 5y agoIT: "Please do this fix." Person 1: "I can't, I don't have physical access." IT: "Please do this fix." Person 2: "I can't, I don't have digital access." Why? It's [IT's?] policy.
- RobRivera 5y agolike today! xD
- dsr_ 5y agoIt just bit FB.
- rvnx 5y agoI like how FB decided to send "ramenporn" as their spokesperson.
- IceWreck 5y ago> Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router. I love this comment.
- yawnxyz 5y agofor something as distributed as Facebook, do multiple somebodys all have to race down each individual datacenter and plug their laptops into the routers? As someone with no experience in this, it sounds like a terrifying situation for the admins...
- MuffinFlavored 5y agoImagine having the a huge portion of the digital world internationally riding on your shoulders...
- deleted 5y ago[deleted]
- laurent92 5y agoImagine that guy has this big npm repository locally with all those dodgy libraries with uncontrolled origin, in their /lib/node_modules with root permissions. Wait, we all do, here.
- victor9000 5y agoYou can use a custom npm prefix to avoid the mess you're describing. So basically: See current prefix: > npm config get prefix Set prefix to something you can write to without sudo: > npm config set prefix /some/custom/path
- bennyp101 5y agoInteresting that they published stuff about their BGP setup and infrastructure a few months ago - maybe a little tweak to roll backs is needed. "... We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software component, enabling fast incremental updates..."
- pbhjpbhj 5y agoSurely Facebook don't update routing systems between data centres (IIRC the situation) when they don't have people present to fix things if they go wrong? Or have an out-of-band connection (satellite, or dial-up (?), or some other alternate routing?). I must be misunderstanding this situation here. [Aside: I recall updating wi-fi settings on my laptop and first checking I had direct Ethernet connection working ... and that when I didn't have anything important to do (could have done a reinstall with little loss). Is that a reasonable analogy?]
- tedmiston 5y ago# todo: add rollbacks
- deleted 5y ago[deleted]
- tiborsaas 5y ago"I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally." Hmm, could be a UI/UX bug then :)
- sjg007 5y agoSeems odd to not have a redundant backdoor on a different network interface. Maybe that is too big of a security risk but idk.
- progbits 5y agoYou know how after changing resolution and other video settings you get a popup "do you want to keep these changes?" with a countdown and automatic revert in case you managed to screw up and can't see the output anymore? Well, I wonder why a router that gets a config update but then doesn't see any external traffic for 4 hours doesn't just revert back to the last known good config...
- kerng 5y agoWondering how Facebook communicates now internally - most of their work streams likely depend on Facebooks systems which are all down. Can engineers and security teams even access prod systems anymore? Like, would "Bastion" hosts be reachable? Wonder if they use Signal and Slack now?
- xgme 5y agoFacebook does use IRC and Zoom as a fallback.
- LordHumungous 5y agoMy team set up a discord lol
- flyingswift 5y agoFB uses a separate IRC instance for these kinds of issues, at least when I used to work there
- alasdair_ 5y agoThere are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.
- kaustubhvp 5y agoI just heard from a contact that the fallback/backup IRC is also down.
- treesknees 5y agoI heard the same thing from my old coworker who is at FB currently. All of their internal DNS/logins are broken atm so nobody can reach the IRC server. I bet this will spur some internal changes at FB in terms of how to separate their DR systems in the case of an actual disaster.
- littlecranky67 5y ago
- Pasorrijer 5y agoFacebook is likely scrambling private jets as we speak to get the right people to the right places.
- aero-glide2 5y agoReminds me of that episode in Mr Robot
- zolosa 5y agoThe cost of the downtime would be
- deleted 5y ago[deleted]
- gabaix 5y agoFacebook 2021 revenue is around $100B. That’s $11M an hour. Since it’s peak hour for ad printing, one can assume double or triple this rate. They are already looking at > $100M in ad loss, not counting reputation damage etc.
- prox 5y agoThink of all the influencers who can’t influence and FB addicts who can’t get their fix (+insta and whatsapp)
- rexreed 5y agoI am sure this is not what they specifically mean by fail fast and break things often.
- cecilpl2 5y agoUser has now deleted the update.
- qnsi 5y agohe started deleting the comments
- costcofries 5y agoLooks like those updates have now been deleted
- guidopallemans 5y agoHe just deleted all his updates. user: https://old.reddit.com/user/ramenporn https://old.reddit.com/user/ramenporn some messages: * This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team). * Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls. * Update 1440 UTC: \ As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.
- yholio 5y agoEssentially, they locked themselves out with an uninspired command line at the exact moment the datacenter was being hijacked by ape-people. Yup, corporate comms won't love these status updates.
- wtf-is-ur-prblm 5y agoSorry, are you referring to data center technicians as “ape people”?
- r721 5y agoArchived version: https://archive.is/QvdmH https://archive.is/QvdmH
- secondcoming 5y agoWho is https://www.reddit.com/user/nathan131412/ https://www.reddit.com/user/nathan131412/
- strenholme 5y agoThe Reddit post is down but not before it was archived: https://archive.is/QvdmH https://archive.is/QvdmH and https://archive.is/TNrFv https://archive.is/TNrFv
- winternett 5y agoAlso, equally important to note, there was a massive expose on FaceBook yesterday that is reverberating across social media and news networks, and today, when I tried to make a post including the tag #deletefacebook, my post mysteriously could not be published and the page refreshed, mysteriously wiping my post... This is possibly the equivalent of a corporate watergate if you ask me... Just my personal opinion as a developer though... Not presented as fact... But hrmmm.
- blobbers 5y agoSo what you're saying is facebook... deleted itself? The singularity is happening. It realized it would end society, so it ended itself.
- deleted 5y ago[deleted]
- wolverine876 5y ago> Reddit r/Sysadmin user that claims to be on the "Recovery Team" They have time to make public posts, and think it's a good idea? Sure, I'm on the 'Recovery Team' too! How about you?
- mdip 5y agoIf it's anything like my past employers, they probably have a lot of time. They probably also got in a lot of trouble. When we'd have situation bridges put in place to work a critical issue, there would usually be 2-3 people who were actively troubleshooting and a bunch of others listening in, there because "they were told to join" but with little-to-nothing to do. In the worst cases, there was management there, also. Most of the time I was one of the 2 or 3 and generally preferred if the rest of them weren't paying much attention to what was going on. It's very frustrating when you have a large group of people who know little about what's going on injecting their opinions while you're feverishly trying to (safely) resolve a problem. It was so bad that I once announced[0] to a C-Level and VP that they needed to exit the bridge, immediately because the discussion devolved into finger-pointing. All of management was "kicked out". We were close to solving it but technical staff was second-guessing themselves in the presence of folks with the power to fire them. 30 minutes later we were working again. My boss at the time explained that management created their own bridge and the topic was "what do to about 'me'" which quickly went from "fire me" to "get them all a large Amazon gift card". Despite my undiplomatic handling of the situation, that same C-Level negotiated to get me directly beneath during a reorganization about six months later and I stayed in that spot for years with a very good working relationship. One of my early accomplishments was to limit any of management's participation in situation bridges to once/hour, and only when absolutely necessary, for status updates assuming they couldn't be gotten any other way (phones always worked, but the other communication options may not have). [0] This was the 16th hour of a bridge that started at 11:00 PM after a full work day early in my career -- I was a systems person with a title equivalent to 'peon', we were all very raw by then and my "announcement" was, honestly, very rude, which I wasn't proud of. Assertive does not have to be rude, but figuring out the fine line between expressing urgency and telling people off is a skill that has to be learned.