17 ms·
What Happened to Facebook, Instagram, and WhatsApp?
- yamrzou 5y agoStrangely, https://research.fb.com/ https://research.fb.com/ is UP.
- kazinator 5y ago> revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online. [Ed note: our readers can't be relied up on to know DNS is, but let's see what they make of this techno-babble.]
- i_like_apis 5y agoKrebs just says it's anybody's guess as to whether it's an internal screw up or a hack.
- r721 5y agoThe now infamous /u/ramenporn said in his latest update that they don't consider attack hypothesis (yet?): https://twitter.com/atoonk/status/1445084833017843721 https://twitter.com/atoonk/status/1445084833017843721
- i_like_apis 5y agoBecause of the level of internal access required to do this intentionally I would assume it isn't a hack, but it could be. The timing is interesting with the whistle-blower news though. But then the timing with regard to China sending 50 military aircraft over Taiwan today is also interesting... FB and communication infrastructure would go down first in times of tension, if you want to go full tin-foil hat. Ok ... enough news reading for me today!
- wolverine876 5y ago> timing with regard to ... No matter when something happens, other things will be happening in the world around the same time. That doesn't establish a correlation (China has been doing that for awhile), much less causation.
- deleted 5y ago[deleted]
- PanosJee 5y agoOne should use codebgp.com
- a-dub 5y agohttps://downdetector.com/ https://downdetector.com/ lol! it's like the bicycle, appliance and consumer toilet paper shortages that resulted from changed consumer behavior during last year's lockdowns, but instead with internet distractions. (even HN is creaking under the load, hah!)
- reilly3000 5y agoOps folks: do you have dedicated networking hardware you can push config changes to as a sandbox of prod? Does Facebook? Do they get simulated or shadowed traffic for pre-prod testing? My guess is no, but I’ve never really worked in a big DC.
- notyourday 5y agoFB's mistake was using this kind of complicated over-engineered setup. It works great when it works... but when it does not, it blows up everything and its complexity means recovery is extremely complicated.
- wolverine876 5y agoThey've been pretty successful.
- pid-1 5y agoMy first job was working in data centers for telecos and my impression was that everything was one cable trip away from never working again. Networks were really complex, nothing was documented nor deployed as code, most equipment were untestable black boxes, people who deployed stuff moved on, etc... Just thinking about working with on prem again gives me chills. But maybe FB knows better than telecos, IDK.
- Aissen 5y ago> Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057 Apparently the people planning the heist went a bit overboard with their misdirection.
- indianpianist 5y ago> The mass outage comes just hours after CBS’s 60 Minutes aired a much-anticipated interview with Frances Haugen, the Facebook whistleblower who recently leaked a number of internal Facebook investigations showing the company knew its products were causing mass harm, and that it prioritized profits over taking bolder steps to curtail abuse on its platform — including disinformation and hate speech. I'm hoping that this is just a coincidence
- badmadrad 5y agoI'm sure the disinformation and hate speech is about steel dossier and all the mean things said about Trump supporters. Or do you mean they prioritized profits over censorship?
- azta6521 5y agoI'm always try to side with Occam, but let me speculate here: This may be a sign of resistance from within? A hacker group so good, they were hired by FB only to carry out a huge, clandestine hack that results in FB being down for hours, if not days?
- indianpianist 5y agoWell, it seems Facebook is back up now.
- serialkoder 5y agoThat's why the gym is so busy today.
- Max_aaa 5y agoTechnical issue (most probably the case) or coverup? It is amazing how far some people go with trying to explain this.
- tofuahdude 5y agoCoverup? I'm so curious what you mean. How could you come to the conclusion that it is anything other than a technical issue?
- jader201 5y agoWhile it's certainly possible (likely?) that it's "just" a technical issue, the article talks about this: > The mass outage comes just hours after CBS’s 60 Minutes aired a much-anticipated interview with Frances Haugen, the Facebook whistleblower who recently leaked a number of internal Facebook investigations showing the company knew its products were causing mass harm, and that it prioritized profits over taking bolder steps to curtail abuse on its platform — including disinformation and hate speech. > We don’t know how or why the outages persist at Facebook and its other properties, but the changes had to have come from inside the company, as Facebook manages those records internally. Whether the changes were made maliciously or by accident is anyone’s guess at this point. I think we can't completely rule out the possible connection to this. Again, likely isn't, but answering the question how one might come to the conclusion.
- yurishimo 5y agoI mean, Cambridge Analytica is the example here. Facebook has been privy to some shady shit at the very least. Is it likely that they purposefully took down all their revenue making machines to distract from the 60 minutes piece? No, probably not. But they've demonstrated that they can't be trusted so it's at least worth investigating.
- nashadelic 5y agoYes, conspiracy theories amok. Mine: considering the 60 minutes piece, could be a disgruntled, internal employee as well?
- optimalsolver 5y agoI know this is tinhat territory, but it's weird this happens right after the FB whistleblower interview on 60 minutes. The outage has pretty much buried that story, and perhaps more importantly, stopped its spread on FB networks. That said, I can't see how FB managers and engineers would actually agree to carry out something like this intentionally.
- rootusrootus 5y agoI assume this outage is costing millions per hour. And it's not exactly great advertising for Facebook, either. I doubt very much they would do something like this on purpose.
- ironlion624 5y agoUnless “they” were one or two disgruntled employees with the access, know-how, and motive to execute a “mistake”. Emphasis added.
- bawolff 5y agoIf it was intentional, that's serious jail time territory. That's a high price to pay for such limited downtime. I'm pretty sure an intentionally malicious actor with that type of access could do much worse things.
- cheschire 5y agoI’m pretty sure the vast majority of entry level spy craft is about convincing people to do highly illegal and destructive things from a place of fear. Not saying this is the work of spies, just that it’s not unimaginable to think some middle manager could convince themselves or a subordinate to do something drastically illegal out of some fear that terrible things would happen otherwise.
- xkeysc0re 5y agoI'm curious as to what law, exactly, they would be breaking. Sabotage in the US code is defined mostly in terms of war material and damages done to physical "national defense" properties. Certainly an employee would be fired and sued by the company, but is deliberately changing a routing policy (and not something like a worm or virus that deletes or otherwise degrades hardware and software) a crime?
- Ne02ptzero 5y agoYou know it's a big outage when other people are writing status updates for you
- mrkramer 5y agoToo bad Facebook can't write status updates on Facebook. They have Twitter account tho.
- deleted 5y ago[deleted]
- mrkramer 5y agoSo somebody messed up Facebook's BGP records and traffic couldn't be routed to Facebook servers. I wouldn't be surprised if some angry insider(employee) got his revenge on Facebook for whatever reason.
- polynomial 5y agoCould one single person really inflict that much damage without a balance/check or code deployment review process in place?
- bink 5y agoWith the right access it wouldn't be hard for someone to configure some key routers in such a way that all traffic is blocked and no one can get into them over the network. They'd need to send someone physically to the sites to reset or replace them. I'm still leaning heavily towards an "oopsie" with routing that accomplished the same thing, however.
- notjesse 5y agoI think if you were smart enough, you may be able to mask some needed changes under some legitimate tickets. You make certain changes that you know will break stuff, but you assign a reviewer who doesn't know enough about the particular thing that they may think it seems fine. I am talking in a very generalized sense, not for this particular issue. But I don't think the code review/deployment process is entirely safe against internal bad actors.
- turbinerneiter 5y agoMaybe Zuck watched the interview, agreed and pulled the plug.
- AzzieElbab 5y agobasically facebook deleted itself
- adolph 5y agoSaid "this is why we can't have nice things"
- WORMS_EAT_WORMS 5y agoIn all seriousness there is definitely some poor soul out there stressing their brains out that probably pushed the button to set this all off. Just a reminder this is not a failure of a single person though and of the organization as a whole and policies in place.
- s5300 5y agoHmm... I'm always reminded of my professor telling me that it's never the fault of who pressed the button, responsibility lay upon who decided to make them able to press a button that can cause such catastrophic issues. Somebody from my engineering class had an internship at DuPonts main facility/production line. Was implementing something that managed to complety shut down production for an entire shift & cause a large fire, ended up being something in the millions worth of damages from production loss and fire damage. Intern wasn't even yelled at IIRC. He actually went on to do some very helpful things the rest of the internship. But man, did the person who let an intern be in the position to single handedly cause such a mess get absolutely fucked by his superiors.
- manquer 5y agoWhat about the middle manager who gave that supervisor the power to put an intern in a such critical position without review?. You can keep going up like that . In most companies someone is the fall guy depending on much impact is there. It is not uncommon for that some guy to be the CEO if the fuck up is big enough .
- sschueller 5y agoI wonder if they managed to get caught in a catch-22 where they can not access the systems to fix it because the access control is the system.
- TravisHusky 5y agoYeah; it sounds like that is maybe the case. Reminds me of the concern around a "dark start" if the power grid goes down where you can't bring up certain power plants because they need power to start.
- mckirk 5y agoAnybody who's ever played Satisfactory knows to keep a few old-school biomass generators around for that reason... I'm guessing Facebook upgraded everything to the highest level tech and inadvertently got thrown back into the stone age.
- anderiv 5y agoI do know that many plants which require power to bootstrap themselves maintain emergency generation facilities (with battery backup for the diesel/natural gas engine starters). Hopefully there's a sufficient number of these to make the "dark start" issue not much of a concern.
- Havoc 5y agoSome can bootstrap yes, but it's still a giant mess and takes time to build back up. Jumpstarting it off hydro or neighbours is preferred
- jamincan 5y agoI know that in Ontario, the Bruce Nuclear Plant (with about 8GW capacity) is designed to run indefinitely through a power outage and did during the Northeast blackout in 2003. I assume that sort of power would be enough to bootstrap the grid in Ontario.
- Sebb767 5y ago
- pytlicek 5y agoNot only security. Also privacy! I started to see messages that I know 100% that I deleted days or weeks ago?! https://twitter.com/Pytlicek/status/1445072626729242637 https://twitter.com/Pytlicek/status/1445072626729242637
- spuz 5y agoSo it appears that WhatsApp are in the process of restoring from backup? Why would they need to do that if it was just a DNS issue? And why would the server be accessible while backup restoration was still in progress? I feel like there is going to be a lot more to this story when it all shakes out.
- pytlicek 5y agoWho knows. I use PiHole where all DNS records are cached. Maybe this is the reason why it happens to me. And regards Twitter (obviously), I'm not the only one who is facing this weird behaviour.
- spaceywilly 5y agoOnce the DNS is back up they need to basically reboot every service. Once server one can’t talk to server two, everything is out of sync and they need to resolve this somehow. They probably have mitigation plans for a few data centers going down, but when it’s all of them at once, that’s going to be a huge pain.
- 0xFF0123 5y agoWhatsApp is (at least supposed to be) e2e, so unless they're restoring from every user's personal backup, it seems an unlikely course of action
- sn_master 5y agoe2e does not mean FB isn't keeping the messages on their server. it only means what is kept pn their server is encrypted, that's all. e2e has no impact on message expiration.
- 5y ago
- deleted 5y ago[deleted]
- r721 5y agoInteresting side effects: >Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com http://facebook.com https://twitter.com/jgrahamc/status/1445066136547217413 https://twitter.com/jgrahamc/status/1445066136547217413 >Our small non profit also sees a huge spike in DNS traffic. It’s really insane. https://twitter.com/awlnx/status/1445072441886265355 https://twitter.com/awlnx/status/1445072441886265355 >This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled. https://twitter.com/TheodoreBaschak/status/1445073229970763781 https://twitter.com/TheodoreBaschak/status/14450732299707637...
- el-salvador 5y agoAnother side effect: Two of our local mobile operators are experiencing issues with phone calls due to network overload. https://twitter.com/claroelsalvador/status/1445081933331959831 https://twitter.com/claroelsalvador/status/14450819333319598...
- paganel 5y agoThe same happened in Romania with two of our mobile operators immediately after FB&all went down.
- karencarits 5y agoPossibly in Norway too (internet though, not phone calls) https://www.nrk.no/nyheter/internett-trobbel-hos-telia-1.15677753 https://www.nrk.no/nyheter/internett-trobbel-hos-telia-1.156...
- synaesthesisx 5y agoBelieve it or not, there are places in the world where FB products (WhatsApp specifically) are used as the primary communication platform for most people.
- hansel_der 5y agooh the irony! can't use the phone network to place a call b/c of fb-errors clogging the pipe
- Hokusai 5y agoHow it can be allowed that two of the most used messaging apps inn the world fall at the same time? The regulators in many countries that allowed the purchase failed to protect customers and competition and helped to create a more fragile world prone to systemic disruptions.
- julianlam 5y agoRegulators don't _see_ every single facet of an acquisition. I'd bet they didn't even think about a scenario like this. Their concerns were probably more along the lines of anti-monopoly, preservation of competition, etc. That said, who wants to go back to ICQ?
- trutannus 5y agoWhile this is a massive inconvenience, I don't see how messaging apps like this are a government problem if they go offline. These are not state run businesses.
- Sebb767 5y agoImagine the SWIFT network (handles all bank transfers) going down. _Technically_, it's a private company, but it can wreck havoc on a country. Similarly, these messaging services are quite essential for some people and this dependency is only going to become stronger. So it can absolutely make sense for a country to have a fallback.
- trutannus 5y agoBanking and the ability to use Facebook messenger are not even remotely close to the same thing. SMS services are still alive, so are phone and other actually critical services. Messenger is nice, but really, outside of being able to send stickers and whatnot (which is nice) this isn't critical infrastructure.
- manquer 5y agoNot in the U.S. Perhaps, However many services even government ones depends on WhatsApp business accounts to provide essential services. A ton of apps also use fb login as their only or primary authentication mechanism some of them would definitely be essential services. Maybe building on Facebook was not best idea for these apps , but essential services would have been impacted nonetheless.
- alexeiz 5y ago"Someone at Facebook screwed up" - intern Chris, you're fired!
- Yoofie 5y agoSurprised no one went with the typical "state sponsored" actor as cause for outages yet.
- paxys 5y agoIt would be in every government's best interests to have Facebook up and running
- SMAAART 5y ago>In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded. SinglePointOfFailure.NoRedundancies.FB
- seaman1921 5y agoNice try, but they have separate communication channels for SREs so don't worry.
- jrodthree24 5y agoYou mean they don't receive all their alerts through facebook messenger?
- regnull 5y agoHaving worked for a similar company, I remember there were some good old IRC servers up and running to communicate in an emergency just like that.
- makeworld 5y agoFacebook has this too, but they require facebook.com DNS to work, so they are also down.
- coldcode 5y agoThanos snapped his fingers and Zuckerberg vanished with the keys. My (very large) employer had a worldwide outage a few years ago where a single bad DNS update stopped everything in its tracks (at the time many things were still in our own data centers, now more is in Amazon/etc). It took most of the day to restart everything. But it's not something most people would have noticed like FB. Thankfully I worked in mobile so not involved.
- manquer 5y agoIt is hard to balance dogfooding (good) with SPOF (bad), many big companies do get it wrong (AWS with S3, Slack in the recent past) all the time. It is easy to get it wrong if your company provides internet services that every developer typical depends in their workflows and to keep educating your own developers on how to use them and when not to use your own services.
- us0r 5y agoThis is costing them not less then $170,000 PER MINUTE. I highly doubt it's some kind of cover up.
- AWildC182 5y agoAt their market cap it would take in excess of 10 years to zero out. 170,000, if that's the actual number, is absolutely nothing to them
- heartbreak 5y agoNot all minutes are equal, so the real number is likely well north of $200k/min. And that is a lot, even for Facebook, with a large enough number of minutes. We're at >4 hours. So they've likely lost north of $50 million in advertising revenue today.
- samhw 5y agoTheir market cap doesn't have anything to do with how much cash they have.
- AWildC182 5y agoThe 170,000 doesn't have a precise unit, it could be losses in market cap, cash reserves, advertiser revenue, operating costs, literally anything. I'm merely putting the number in perspective. 6 digits isn't really something that's particularly concerning on any Facebook spreadsheet.
- samhw 5y agoI'm not sure what you mean by that. The 170,000 does have a precise unit - it's dollars lost in cash, as stated in the comment you were replying to. The amount of time taken for Facebook to fold in minutes is represented by $TOTAL_CASH ÷ $170,000. There's nothing mystical about it. But $TOTAL_CASH is entirely different from, and unrelated to, market cap. Market cap does not come into this at all, in any way.
- 5y ago
- TedShiller 5y agoToday, Facebook made the world a better place. For real, this time.
- toss1 5y agoTrue Although, to be fair, that is kind of like praising the arsonist after he put out the fire he started (which had already smoke-damaged the whole neighborhood).
- Andrew_nenakhov 5y agoI wonder if Oculus Quests are working?
- david_allison 5y agoLeaderboards are down, but functional otherwise for me
- jareklupinski 5y agothere was a great tweet earlier about all the people who were in VR when it happened now being stuck in limbo ;)
- mewpmewp2 5y agoThey can't even take off their headsets?
- deleted 5y ago[deleted]
- purple_ferret 5y agoCan't buck the zuck
- deleted 5y ago[deleted]
- brewdad 5y agoMaybe they shouldn't have hired that new guy, Bobby Tables.
- jpomykala 5y ago„commit on the first day”
- dynamite-ready 5y agoWhatsApp going down is a total shock. I thought that ran on completely separate infrastructure.
- _Understated_ 5y agoIf the issues is DNS-related or routing-related then since it's all owned by FB, they own the records too so it's likely that literally everything that has a DNS record under the FB cost code will be hit. The WhatsApp servers will be sitting idle most likely!
- latortuga 5y agoLeaving aside whether or not this is true, why would you apply DNS changes to all of your properties at once? Especially if it's a sensitive change? Seems much more likely to me that WA/IG/Oculus somehow rely on facebook.com behind the scenes than that all FB domains were affected by a config change.
- notwedtm 5y agoA misconfiguration of BGP would absolutely do it.
- toast0 5y agoI worked at WhatsApp until Aug 2019. WhatsApp hasn't been on completely separate infrastructure for quite some time. It's in FB datacenters, so if FB BGP is messed up, so is WA. There is a separate WA ASN that's used for DNS, but it's still FB infra, and announced through the FB ASN, so that doesn't help either. Instagram still has their DNS with AWS Route 53, so their DNS is still up, but their site isn't because it runs on FB Infra too. Within the FB datacenters, WhatsApp is somewhat isolated (at least when I left, chat was run on dedicated hosts allocated to WhatsApp only, but using FB container orchestration tools, and FB specified hardware, etc). Edit to add: WhatsApp has some mitigations against DNS not working, but in this case, it looks like DNS being dead was a symptom of something (probably BGP config error, if the Reddit post earlier was accurate) and that something also broke the underlying servers.
- _Understated_ 5y agoI assume this means that anyone that uses their FB account to login somewhere can't login once their token expires... Hahahahaha!
- deleted 5y ago[deleted]
- sebow 5y agoSomething good, for once
- winternett 5y agoI wrote about what is going on today with FaceBook and many other social media sites long ago. Market-driven social media platforms end up becoming destructive in behavior on their user base over time because profit demand from investors grows over time driving bad practices. Tom from Myspace really had the concept right. There's no reason why he shouldn't be on CNN right now speaking about what is going on as an informed consultant. They may possibly be covertly cleaning up obviously harmful content and evidence behind the curtains now that they are closed. Just speculation/opinion, not proven fact in any way though... Many sites and apps on the Internet also rely on FaceBook for authentication and analytical tracking, so that may explain some cases of service and site outages, but all social media sites operate under the same cloud of non-transparent and profit driven mystery. Congress is overdue in protecting citizens from psychological, financial, and emotional manipulation, but first they need THE RIGHT people educating them about how to recognize the underlying issues in modern IT and algorithms. This is a major point in the Internet's history, a point where everything may change. I wrote about today's revelations a while back (in 2017) and many times in other ways since ( http://circuitbored.com/communicate/ http://circuitbored.com/communicate/ ) - Preparing for the GIANT "Unplug": http://circuitbored.com/communicate/viewtopic.php?f=3&t=23 http://circuitbored.com/communicate/viewtopic.php?f=3&t=23
- hartator 5y agoIsn't Facebook.com still down?
- wazoox 5y agoYep, still down.
- dm319 5y agoAnyone noticed Signal seems to be down too?
- crocal 5y agoNegative in my case. It’s functioning properly.
- robbedpeter 5y agoSignal is up - everything seems slow online, though. Getting normal congestion at the ISP level, looks like the side-effects on the web in general are noticeable when facebook goes hard down like this.
- MarcellusDrum 5y agoAnd Telegram is extremely slow. Probably the number of new users.
- basilgohar 5y agoIt's not for me. But several services seem to be a lot slower, possibly due to network effects (haha) on DNS as many devices are repeatedly trying lookup FB domains and not finding them, so they just try again and again.
- wolverine876 5y agoI had an hour+ video meeting on it a little while ago.
- annoyingnoob 5y agoThe world should do 'a day without facebook' more often.
- caeril 5y ago1. This is mostly speculation. 2. Please don't boost Krebs. He is a known CIA cutout whose entire career is predicated on taking geopolitical sides on matters of network security.
- stareatgoats 5y agoNot experiencing schadenfreude (much), but I have to say that I never thought "move fast and break things" was a good guiding principle...
- 1vuio0pswjnm7 5y ago"As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page." Not quite. Many DoH servers are working fine. DNS isn't a problem for the browser, but it seems to be a problem for Facebook's internal setup. It's like their proxy configuration is 100% reliant on DNS lookups in order to find backends. The FB content servers are reachable. It is only the Facebook DNS servers that are unreachable. Don't take my word for it, try for yourself www.facebook.com 1 IN A 179.60.192.3 (content) static.facebook.com 1 IN A 157.240.21.16 (content) a.ns.facebook.com 1 IN A 129.134.30.12 (DNS) ping -c1 157.240.21.16 |grep -A1 statistics --- 157.240.21.16 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms ping -c1 179.60.192.3|grep -A1 statistics --- 179.60.192.3 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms ping -c1 -W2 129.134.30.12 |grep -A1 statistics --- 129.134.30.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms The browser, i.e., client, here, curl, has an idea where to find Facebook.com curl -HUser-Agent --resolve www.facebook.com:443:179.60.192.3 https://www.facebook.com|sed windex.htm Wait... links -dump index.htm [IMG] Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can. Go Back Facebook (c) 2020 . Help Center grep HTTP index.htm HTTP/1.1 503 No server is available for the request
- rbobby 5y agoIf I do: nslookup > server 8.8.8.8 Default Server: dns.google Address: 8.8.8.8 > facebook.com Server: dns.google Address: 8.8.8.8 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to dns.google timed-out Weird.
- chrononaut 5y agoAt the time, OP likely pulled cached DNS records to interrogate the associated IPs directly for their application (HTTP) level resources.
- 5y ago
- beprogrammed 5y agoI like to think that it went down, and they just decided, you know what? It's probably for the best.
- r721 5y ago>Facebook has dispatched a small team to one of its California data centers to try and manually reset its servers in an attempt to fix the problem. >(It's chaos to even try to contact folks, but people are resorting to zoom, discord etc) https://twitter.com/MikeIsaac/status/1445118465258364928 https://twitter.com/MikeIsaac/status/1445118465258364928
- stevenhubertron 5y agoTalk about a tactical attack. Whistleblower interview goes up. BGP weakness likely hacked. Facebook down. Facebook internal tools for communicating problem and fix also down. Everyone is WFH because of COVID. The fix may have been easy, all the tools and comms down you need to fix is making it hard. It's all so interesting. Good riddance to Facebook.
- nickthemagicman 5y agoVery curious much money is being lost by these FB sites at the moment.
- _joel 5y agoLooks like DNS resolution is back now, but the site's still down.
- deleted 5y ago[deleted]
- Xxplosive 5y agoDue to DNS being busted, all internal FB services/tooling that they'd use to push DNS config updates are probably completely inaccessible. Someone at FB will have to manually SSH into a production host (assuming they can even identify the right one), and issue some commands to repopulate the DNS records. They'll probably have to do this without any access to internal wikis, documentation, or code. Keeping those poor network engineers in our thoughts.
- huijzer 5y agoI strongly dislike how we are forced into centralizing our online life into a few big corporations. Therefore, it is somewhat nice to read that even the access cards don’t work at Facebook HQ due to them running everything via the Facebook domains. Still, let‘s hope that this gets fixed soon for the engineers and users involved
- moneywoes 5y agoHow much revenue could they have lost
- New_California 5y agoWhy the article claims the change originated at Facebook? Updates to BGP routing are not authenticated. BGP hijacking is a real thing. To the best of my understanding, other well-positioned AS could publish this evil update to BGP routing tables.
- bob229 5y agoIf only the cancer apps could be shut down permanently
- CodeWriter23 5y agoI’m just gonna say this. Disclaimer I have no knowledge nor evidence whatsoever that this may be the reality. But speculation seems to be the order of the day… Seems like cutting their ASN off from the world would be a great way to cut off any would-be Discovery Volunteers that might try to collect evidence 4chan style to support the whistleblower’s case.
- babypuncher 5y agoAww man they're back. What a shame.
- mjevans 5y agoThis is why, even with keycards, you need __key__ disaster recovery employees to have real keys that really go through the locks and let them in to do what must be done.
- Chicken3gg 5y agoHuman error when prepairing the update, maybe some kind of security system in the code should have this excetion catched and not let it run. So => Facebook fault - also for hundreds of other online companies that lost money
- Chicken3gg 5y agoHuman error when preparing such a big update and schedule it for the (European) afternoon and the Asian evening. Maybe some kind of security system in the code should have caught this exception and not let the computer take control of this. So → Facebook's fault — also for hundreds of other online companies that lost money.
- Ice_cream_suit 5y agoThis will have a positive effect on the mental health of Facebook users. Could Mr Zuckerberg please arrange for this to be weekly event ?
- gannon- 5y agohttps://engineering.fb.com/2021/08/09/connectivity/backbone-management/ https://engineering.fb.com/2021/08/09/connectivity/backbone-...
- jdlyga 5y agoImagine if Google went down like this for 8 hours. No Gmail, Google Search, Google Maps, Google Drive, YouTube? I thought these companies were a little more fault tolerant.
- sektek 5y agoFaulty configuration? - who makes a significant change during prime US/UK hours?