55 ms·

Facebook-owned sites were down

2589 points by nabeards 5y ago

GDC7 5y ago
Maybe intentional? Zuck trying to give an example of what a world without FB would look like, kinda saying to detractors what would happen if they had it their way.
- forix 5y ago
  That would be classic Zuck right there
- gmiller123456 5y ago
  Maybe "intentional" in quotes. My money is on a major security breach and they've shut everything down until they can deal with it. Even if you go to Instagram by the IP address [1], you get a 400 error. So it looks like things are off line because they want them off line for now. https://31.13.65.174/ https://31.13.65.174/
ballenf 5y ago
One real potential cost to FB here is breaking people's addictions to FB and IG. This might just be the little finger-snap to wake up a sizable chunk of the user base that they life is just a little better during the outage.
deleted 5y ago
[deleted]
tacker2000 5y ago
https://www.status.fb.com/ https://www.status.fb.com/ is back online now
mrfusion 5y ago
Thought experiment: what if they were down for a week and the world completely healed itself?
asduoihfijnu 5y ago
add comment
susahahhaha 5y ago
add comment
ur-whale 5y ago
> Facebook-owned sites are down And the world rejoiced.
sparrish 5y ago
Seems to be DNS related. None of the listed facebook nameservers are resolvable or reachable: a.ns.facebook.com b.ns.facebook.com c.ns.facebook.com d.ns.facebook.com
- zulln 5y ago
  In the beginning it responded but gave server errors.
  - jaywalk 5y ago
    Which seems to indicate a massive infrastructure failure.
    - sakisv 5y ago
      Actually I'd argue that the biggest problem would be to wait for the TTL to expire after you've fixed the problem.
      - jaywalk 5y ago
        The TTL was most likely very low, so I don't see that as being an issue.
- sparrish 5y ago
  Looks like the routing is goofed. Loops over and over - DDoS attacking themselves. mtr -r -c10 -w -b a.ns.facebook.com Start: 2021-10-04T10:02:50-0600 Loss% Snt Last Avg Best Wrst StDev ... 4.|-- ae-2-rur101.cosprings.co.denver.comcast.net (162.151.51.125) 0.0% 10 12.6 11.9 9.6 19.0 2.9 5.|-- 24.124.155.233 0.0% 10 9.3 10.2 9.1 12.4 1.1 6.|-- 96.216.22.45 0.0% 10 12.0 14.0 11.6 31.3 6.1 7.|-- be-36041-cs04.1601milehigh.co.ibone.comcast.net (96.110.43.253) 20.0% 10 14.6 13.5 11.6 20.5 3.0 8.|-- be-3402-pe02.910fifteenth.co.ibone.comcast.net (96.110.38.126) 0.0% 10 12.2 12.0 11.5 13.2 0.5 9.|-- 173.167.59.170 0.0% 10 13.8 17.8 12.0 34.7 8.4 10.|-- 129.134.40.74 0.0% 10 15.3 12.6 11.4 15.3 1.1 11.|-- 129.134.43.226 0.0% 10 18.9 15.3 12.6 20.3 3.0 12.|-- 129.134.98.166 0.0% 10 12.5 14.2 12.5 20.4 2.3 13.|-- 129.134.54.61 0.0% 10 34.2 30.8 28.9 34.2 1.8 14.|-- 129.134.53.61 0.0% 10 29.8 31.1 28.9 36.5 2.7 15.|-- 129.134.53.61 90.0% 10 31.9 31.9 31.9 31.9 0.0
ivanmontillam 5y ago
One thing that triggers my OCD is leaving the Facebook session open, though it's my own computer. Maybe it's DNS. It's always DNS.
- debacle 5y ago
  Facebook went down (error page, then 503) before DNS went down.
  - marbex7 5y ago
    Same for me.
- WallyFunk 5y ago
  > Maybe it's DNS If it is, close Facebook as there's probably a BGP hijack going on that is siphoning off personal data and or secrets
raimille1 5y ago
Whatsapp down as well
milankragujevic 5y ago
Also Speedtest.net for me is showing a 503 error page. Seems a large CDN might be having problems. Their status page shows all green. FB and their other sites are also down. edit: I see it's back up and I've been getting downvoted, here's a screenshot of the error for clarity https://i.imgur.com/wvhOwwL.png https://i.imgur.com/wvhOwwL.png
- chki 5y ago
  If Facebook and WhatsApp and Instagram fails there are probably a lot of people checking whether their Internet works. That might be why Speedtest was overwhelmed.
nabeards 5y ago
Seems to be affecting all Facebook properties.
- mrweasel 5y ago
  Instagram just returns a 503. Crazy how closely everything seems to be integrated. I’d guess internal networking issues, but the insane that something can bring down all of Facebooks properties.
tut-urut-utut 5y ago
Facebook down, WhatsApp down, but Signal still works. Time for a change? EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.
- WallyFunk 5y ago
  > Signal still works https://old.reddit.com/r/thehatedone/comments/f160jh/is_signal_centralized/ https://old.reddit.com/r/thehatedone/comments/f160jh/is_sign... Signal is still centralized and uses AWS. So if AWS was to go down, it would affect not just Signal but vast swathes of the Internets.
- fsflover 5y ago
  Signal had their own share of downtime. How about going to a federated system instead of repeating the same mistakes? (https://matrix.org https://matrix.org)
  - JCWasmx86 5y ago
    But still a part would be down, if a server has an outage. How about a system, where every device that is used for chatting is a server at the same time? I wonder whether something like that already exists. Bundle it together with bigger servers to handle the load. If the bigger servers experience outages, the service can still continue, although a bit slower
    - fsflover 5y ago
      Matrix P2P already exists.
      - JCWasmx86 5y ago
        Thanks, didn't know that!
- Arathorn 5y ago
deleted 5y ago
[deleted]
jleyank 5y ago
Somebody published vogon poetry and pictures and the internet routed around the damage...
- poetaster 5y ago
  I have not laughed so loud since before the virus that shall have no vogon name.
deleted 5y ago
[deleted]
kaustubhvp 5y ago
it is always DNS!
bubblehack3r 5y ago
Their stock is down 5% too. Everything is down for them today ;)
- i_like_apis 5y ago
  Great dip to buy. (I'm not a facebook zealot, but you know it will recover today or tomorrow once the DNS is sorted in an hour or two) Of course there is the whistle-blower issue too...
  - derwiki 5y ago
    I don’t think stock dip is related to downtime; anecdotally, I’ve never seen a company’s stock affected by downtime (unless that downtime destroys the business)
    - i_like_apis 5y ago
      You may be right, but theres a Reuters article about the downtime, this is making the news today. I would say Facebook is different because of their scale. Looks like there are a few problems with fb in the news today ...
deleted 5y ago
[deleted]
koprulusector 5y ago
Reddit wasn’t working a few min ago. Broader issue?
- blowski 5y ago
  Reddit goes down every 10 mins anyway.
  - stemc43 5y ago
    this person uses reddit
m_coder 5y ago
I unblocked Facebook right now from my hosts file so I could message someone and couldn't figure out why Facebook failed to load. I tested HN and viola I see that the entire world has sent Facebook requests to 0.0.0.0 lol
- interestica 5y ago
  You broke it. I didn't receive expected WhatsApp messages and am only now realizing there's no indication within the app that there is even a problem. It only becomes (somewhat) apparent when sending a message never gets a single check mark. Not a graceful failure for the user view.
ricardo81 5y ago
Doesn't seem too clever that Facebook's NS servers are a.ns.facebook.com, b.ns.facebook.com etc. IIRC that kind of setup requires some glue records.
- gorgoiler 5y ago
  If you mean because the name servers are in the same zone, this is very common. When an NS is returned for a zone, you also get an “additional” A and AAAA to resolve the NS name. It’s called glue. dig NS example.com ; ANSWER example.com. NS ns1.example.com. ; ADDITIONAL ns1.example.com. A 1.2.3.4 Edit: I didn’t see your glue comment when I wrote this.
  - ricardo81 5y ago
    Cheers, I'd edited my post. Thought the common wisdom nowadays was to use nameservers on different TLDs and sub-labels for the best resilience. /added, they seem to have glue records so I'd assume it's the nameservers themselves having issues. $ dig NS @g.gtld-servers.net. a.ns.facebook.com ;; AUTHORITY SECTION: facebook.com. 172800 IN NS a.ns.facebook.com. facebook.com. 172800 IN NS b.ns.facebook.com. facebook.com. 172800 IN NS c.ns.facebook.com. facebook.com. 172800 IN NS d.ns.facebook.com. ;; ADDITIONAL SECTION: a.ns.facebook.com. 172800 IN A 129.134.30.12 a.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35 b.ns.facebook.com. 172800 IN A 129.134.31.12 b.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35 c.ns.facebook.com. 172800 IN A 185.89.218.12 c.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35 d.ns.facebook.com. 172800 IN A 185.89.219.12 d.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35
nwatson 5y ago
I guess the "prophets" at Victory Channel / Flashpoint called down Holy Fire on the Facebook infrastructure in retribution ... https://youtu.be/FbSkFuvqFdA?t=1127 https://youtu.be/FbSkFuvqFdA?t=1127 . (I'm an Evangelical Christian but those folks are nuts ... Mario Murillo, Lance Wallnau, Hank Kunneman, Gene Bailey, etc.)
jangrahul 5y ago
makes me think, why dont porn sites ever go down ?
- gmiller123456 5y ago
  Phrasing!
chasd00 5y ago
can productivity (or emotional stability) for the overall US economy be tracked on a daily basis? I wonder if a wholesale facebook outage would show up on that graph as a brief blip in the positive direction.
bhartzer 5y ago
Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only. It's so important to diversify, such as building a website.
- tinza123 5y ago
  Well assuming they do that, likely they will host websites on AWS or Azure. Then AWS is down, what are you gonna do?
  - K5EiS 5y ago
    There's always Oracle Cloud ;)
  - jressey 5y ago
    Move your resources to the one you didn't pick and point your domain registrar there.
- deleted 5y ago
  [deleted]
nicbou 5y ago
Alle Störungen shows a massive spike in problems for every service it keeps track of: https://allestörungen.de/ https://xn--allestrungen-9ib.de/
- TravisHusky 5y ago
  Interesting; I have been noticing a lot of service are unstable today. I wonder if there is a larger outage.
pwenzel 5y ago
I noticed that some websites are loading slowly due to the third party script https://connect.facebook.net/en_US/fbevents.js https://connect.facebook.net/en_US/fbevents.js timing out. When uBlock Origin is running, this script gets blocked and pages return to feeling snappy.
monkaiju 5y ago
Gotta post this every time theres a big DNS issue, which seems daily now. Check out Dug! Its a global DNS propagation/monitoring toolon the CLI: https://github.com/unfrl/dug/ https://github.com/unfrl/dug/
LuisMondragon 5y ago
Some Oculus Quest owners can't use their device https://www.reddit.com/r/OculusQuest/comments/q18xwy/facebookoculus_outage/ https://www.reddit.com/r/OculusQuest/comments/q18xwy/faceboo...
russellbeattie 5y ago
It's not hyperbole to say that this is going to literally save lives. Cutting off Facebook's firehouse of hate and misinformation for just a couple hours is going to have a obvious positive effect on millions of people. At this scale, at least one person will get vaccinated today because they didn't see the wall of ignorance that is FB's news feed. Maybe we should introduce "digital blue laws", where one day a week, social media is shut down for the overall good of society.
xmpir 5y ago
Just thinking about all the conspiracy theories you could make of this. Yesterday pandora papers, today the internet stops working.
elzbardico 5y ago
Rejoice! The revolution has started! Yes, I know humor is not welcome in HN
matsemann 5y ago
What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days. Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.
74d-fe6-2c6 5y ago
I'm pretty sure this has been building up since the morning (Germany). I've had odd connectivity problems to a number of sites including slack for a moment.
joelbondurant 5y ago
LOL! They should have used the USA Fact-Check Algorithm from the Science Ministry.
TrackerFF 5y ago
Anyone wanna estimate the cost of total downtime for facebook and instagram, as far as lost ad revenue goes - per minute?
- devenvdev 5y ago
  Looks like someone built a counter: https://facebookadloss.facebookadloss.repl.co/ https://facebookadloss.facebookadloss.repl.co/ Edit: the counter just jumped from 10B to 60M, so I doubt it's any reliable :)
EastOfTruth 5y ago
We can only hope that they will be gone forever... and HN is having major issues at the same time!
nurhdmsx 5y ago
Its always a DNS problem
Animats 5y ago
New York Times has coverage.[1] "A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo." [1] https://archive.is/iBzs3 https://archive.is/iBzs3
peanut_worm 5y ago
I am sure it is just a symptom of the Facebook outage, but it seems like every website I am going on is slower than usual today.
- Redoubts 5y ago
  I've seen a couple fail to log in, because their SSO is broken through this. (even if FB login is merely an option)
- donatj 5y ago
  Every Facebook App on every phone is DDoS-ing the DNS system https://twitter.com/blazejkrajnak/status/1445063232486531099 https://twitter.com/blazejkrajnak/status/1445063232486531099
pmlittle 5y ago
This is all left brain implimentation with looping and classic complexity coming home to roost. As we move through time, we build off of solutions of the past which are solving a problem, but complexity keeps adding on and this is a classic programming/computer science delemma.
qualudeheart 5y ago
If it bleeds we can kill it!
waltbosz 5y ago
I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook. For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.
Bwild 5y ago
They must have shut it off and turned it back on
rvnx 5y ago
Fixed
deleted 5y ago
[deleted]
MrPatan 5y ago
Oh no
oldabc 5y ago
https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
rkagerer 5y ago
The internet felt a little more fresh and clean today.
husamia 5y ago
do we know what caused the DNS outage at FB?
deleted 5y ago
[deleted]
cheese_van 5y ago
Sir, I don't care who you are, you must open a ticket.
- durnygbur 5y ago
  Rajeesh FFS, get off HN! We have the world on fire!
cblconfederate 5y ago
It's like a nuclear bomb exploded on the internet.
nabeards 5y ago
Was receiving an error page, now just a server not responding error.
- kawsper 5y ago
  I can't even resolve Facebook.com
  - daitangio 5y ago
    Mobile app hangs too (I am from Italy btw)
TremendousJudge 5y ago
Whataspp too. This seems pretty big
neb_b 5y ago
Also instagram and messenger
zip1234 5y ago
DNS?
- SSLy 5y ago
  No, I'm getting their error page, so Load Balancers or whatever is behind. EDIT: Or at least not /just/ DNS.
  - rootinier 5y ago
    Definitely DNS. facebook.com might be in your local dns cache.
  - andyjohnson0 5y ago
    I'm getting an error page with a dead image link and a 2020 copyright date (uk)
- deleted 5y ago
  [deleted]
RedShift1 5y ago
Nothing of value was lost
- ozfive 5y ago
  This cannot be said enough.
  - blowski 5y ago
    On the contrary, it's said far too much. Facebook is extremely valuable for a lot of people. I dislike Facebook as much as most people on here, but saying "it's totally pointless" is silly and it's not surprising that those who say it are ignored by those who use Facebook.
    - rawoke083600 5y ago
      Never underestimate software that is 'just good enough'
    - belter 5y ago
      In what ways is Facebook "extremely valuable for a lot of people"?
      - can16358p 5y ago
        Communication for many out there. Many will be just fine without commenting on cat photos or bragging with their likes or followers. Many will be in trouble if they use WhatsApp/Instagram/Messenger/Marketplace to do business and any important communication.
      - blowski 5y ago
        * A friend of mine runs a posh burger van that moves around a lot, and he puts "today's location" on Facebook. * My wife talks with her family in Brazil through Facebook, sharing photos * My Church receives a lot of help requests from people in trouble through Facebook * Some abuse charities talk give support to victims through Facebook etc You could argue that it would be nice if there were alternatives, or that these organisations shouldn't be using Facebook at all. Sign me up for your campaign, I agree with you. But if you say "Facebook has no value" then you will never understand the value proposition you need to offer in order to kill Facebook.
lewich 5y ago
Hopefully permanently.
- aaomidi 5y ago
  Yeah let's keep it that way.
rootinier 5y ago
nslookup www.facebook.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 * server can't find www.facebook.com: SERVFAIL
- aduitsis 5y ago
  dig +trace messenger.com shows that all is well with the root DNS servers and dig @a.ns.facebook.com messenger.com ;; connection timed out; no servers could be reached and also ping a.ns.facebook.com 3 packets transmitted, 0 packets received, 100.0% packet loss shows that something's wrong with facebook.
tomerbd 5y ago
No like for you
aritraghosh007 5y ago
The down page shows copyright from 2020 smh
new-day-rising 5y ago
Thoughts and prayers...
husainhz7 5y ago
Whatsapp's down too. Tough month for FB, especially with the leak.
- decrypt 5y ago
  What leak are you referring to?
  - Wistar 5y ago
    https://www.cbsnews.com/news/facebook-whistleblower-frances-haugen-misinformation-public-60-minutes-2021-10-03/ https://www.cbsnews.com/news/facebook-whistleblower-frances-...
  - samwilliams 5y ago
    Whistleblower that spoke to WSJ.
  - nerbert 5y ago
    https://www.nytimes.com/2021/10/03/technology/whistle-blower-facebook-frances-haugen.html https://www.nytimes.com/2021/10/03/technology/whistle-blower...
    - dmix 5y ago
      The actual leak was published on WSJ “The Facebook Files” https://www.wsj.com/articles/the-facebook-files-11631713039 https://www.wsj.com/articles/the-facebook-files-11631713039
  - orangepurple 5y ago
    Facebook Whistleblower Claims Profit Was Prioritized Over Clamping Down on Hate Speech A Facebook whistleblower, who is due to testify before Congress on Tuesday, has accused the Big Tech company of repeatedly putting profit before doing “what was good for the public,” including clamping down on hate speech. Frances Haugen, who told CBS’s “60 Minutes” program that she was recruited by Facebook as a product manager on the civic misinformation team in 2019, said she and her attorneys have filed at least eight complaints with the U.S. Securities and Exchange Commission. During her appearance on the television program on Sunday, Haugen revealed that she was the whistleblower who provided the internal documents for a Sept. 14 exposé by The Wall Street Journal that claims Instagram has a “toxic” impact on the self-esteem of young girls. That investigation claimed that the social media giant knows about the issue but “made minimal efforts to address these issues and plays them down in public.” “The thing I saw at Facebook over and over again was there were conflicts of interest between what was good for the public and what was good for Facebook. And Facebook, over and over again, chose to optimize for its own interests, like making more money,” said Haugen. She explained that Facebook did so by “picking out” content that “gets engagement or reaction,” even it that content is hateful, divisive, or polarizing, because “it’s easier to inspire people to anger than it is to other emotions.” “Facebook has realized that if they change the algorithm to be safer, people will spend less time on the site, they’ll click on less ads, they’ll make less money,” she claimed. Haugen is expected to to testify at a Senate hearing on Oct. 5 titled “Protecting Kids Online,” about Facebook’s knowledge regarding the photo sharing app’s allegedly harmful effects on children. During her appearance on the television program, Haugen also accused Facebook of lying to the public about the progress it made to rein in hate speech on the social media platform. She further accused the company of fueling division and violence in the United States and worldwide. “When we live in an information environment that is full of angry, hateful, polarizing content it erodes our civic trust, it erodes our faith in each other, it erodes our ability to want to care for each other. The version of Facebook that exists today is tearing our societies apart and causing ethnic violence around the world,” she said. She added that Facebook was used to help organize the breach of the U.S. Capitol building on Jan. 6, after the company switched off its safety systems following the U.S. presidential elections. While she believed no one at Facebook was “malevolent,” she said the company had misaligned incentives. “Facebook makes more money when you consume more content,” she said. “People enjoy engaging with things that elicit an emotional reaction. And the more anger that they get exposed to, the more they interact and the more they consume.” Shortly after the televised interview, Facebook spokesperson Lena Pietsch released a statement pushing back against Haugen’s claims. “We continue to make significant improvements to tackle the spread of misinformation and harmful content,” said Pietsch. “To suggest we encourage bad content and do nothing is just not true.” Separately, Facebook Vice President of global affairs Nick Clegg told CNN before the interview aired that it was “ludicrous” to assert social media was to blame for the the events that unfolded on Jan. 6. The Epoch Times has reached out to Facebook for additional comment. https://www.theepochtimes.com/facebook-whistleblower-claims-profit-was-prioritized-over-clamping-down-on-hate-speech_4030295.html https://www.theepochtimes.com/facebook-whistleblower-claims-...
m0guz 5y ago
Ironic enough; status.fb.com also down.
- ricardo81 5y ago
  It uses the same nameservers as facebook.com, same point of failure. Seems like another poster posted finer details regarding BGP/peering which is ultimately causing the issue.
- devenvdev 5y ago
  Reminds me of S3 outage a couple of years ago when AWS status page went down because it was relying on... S3...
- zoobab 5y ago
  LOL
  - rd_police 5y ago
    very based response, sir
- awinter-py 5y ago
  need a status.status.fb.com to indicate the status of the status and / or an S3 bucket with a json blob the apps can pull to at least tell users 'here's what's up'
OnceUponADevops 5y ago
Confirming that we're seeing a major outage with all of our integrations with FB products.
boramalper 5y ago
Duplicates: https://news.ycombinator.com/item?id=28748233 https://news.ycombinator.com/item?id=28748233 https://news.ycombinator.com/item?id=28748199 https://news.ycombinator.com/item?id=28748199 https://news.ycombinator.com/item?id=28748159 https://news.ycombinator.com/item?id=28748159 https://news.ycombinator.com/item?id=28748246 https://news.ycombinator.com/item?id=28748246 https://news.ycombinator.com/item?id=28748131 https://news.ycombinator.com/item?id=28748131
akshayrajp 5y ago
As are Instagram and WhatsApp
deleted 5y ago
[deleted]
attende_domine 5y ago
% ping whatsapp.com ping: whatsapp.com: Name or service not known % ping web.whatsapp.com ping: web.whatsapp.com: Name or service not known % ping facebook.com ping: facebook.com: Name or service not known % ping instagram.com PING instagram.com (31.13.65.174) 56(84) bytes of data. 64 bytes from 31.13.65.174 (31.13.65.174): icmp_seq=1 ttl=53 time=110 ms
underscore_ku 5y ago
good
mef 5y ago
Looks like the routes to their hosted nameservers are down, e.g. A.NS.FACEBOOK.COM
glanzwulf 5y ago
Oh no... Anyway...
- TremendousJudge 5y ago
  WhatsApp is pretty important infrastructure for most of the world
  - jszymborski 5y ago
    Which is regrettable when secure alternatives exist like Signal and Matrix whose business model doesn't involve selling your data.
    - TremendousJudge 5y ago
      Yeah I'm not saying I like it
  - pc86 5y ago
    For some egregiously loose definitions of "infrastructure," maybe.
    - TremendousJudge 5y ago
      The same definition that includes phone lines also includes the messaging service everybody uses
    - drcongo 5y ago
      And "important"
    - nicoburns 5y ago
      In parts of South America it's used for all sorts of things. Want to know when your bus is arriving? The bus company likely only knows because the driver is WhatsApp'ing them status updates.
rocho 5y ago
For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL: $ dig facebook.com ; <<>> DiG 9.16.21 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23982 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;facebook.com. IN A ;; Query time: 16 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Oct 04 17:53:00 CEST 2021 ;; MSG SIZE rcvd: 41
- john37386 5y ago
  Even the Name servers are not returning any values. That's bad. dig @8.8.8.8 +short facebook.com NS These are usually anycasted, meaning that 1 ip return in NS are in fact several servers spread in several regions. They are distributed to closer match through agreements with ISP with the BGP protocol. Very interesting, because it seems that it took 1 DNS entry misconfiguration to withdraw M$ worth of devices from the internet.
  - variant 5y ago
    BGP goof? https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445056923309649926?s...
- simlevesque 5y ago
  Maybe they tried everything else before that. At first it was working but they couldn't serve responses: https://i.imgur.com/UaCtOiX.png https://i.imgur.com/UaCtOiX.png Notice the "2020"
  - rvnx 5y ago
    The servers struggle to reply a basic 5xx answer. Two possibilities: - the DNS services internally have issues (most likely, as this could explain the snowball effect) - it could be also a core storage issue and all their VMs are relying on it and so they don't want to block third-party websites and think it will last for a long time, so they prefer to answer nothing for now in the DNS (so it will fail instantly to the client, and drain the application/database servers so they can reboot with less load)
natas 5y ago
good ridance
rainboiboi 5y ago
Just wondering - would the engineer who made the mistake be fired?
- tomelliott 5y ago
  Nope: https://www.usenix.org/conference/lisa19/presentation/turner https://www.usenix.org/conference/lisa19/presentation/turner
- yodon 5y ago
  If a single person can cause the failure during the course of their normal tasks, it's not the fault of that person it's the fault of designers of the systems and processes used by that person.
- aaomidi 5y ago
  If they are then Facebook is worse than I thought.
- baby 5y ago
  That’s not the culture at facebook
  - babuskov 5y ago
    "Move fast and break things". Yeah, it's exactly the opposite. The person should be promoted ;)
- deleted 5y ago
  [deleted]
- pc86 5y ago
  What makes you think it was a mistake? What makes you think an engineer did it? Sometimes things just break and take time to fix.
- jaywalk 5y ago
  How could anyone answer that question? We don't even know that an engineer made a mistake in the first place, much less what the mistake was and what led up to it.
deleted 5y ago
[deleted]
cryptodan 5y ago
Hope it's permanent
deleted 5y ago
[deleted]
kingkool68 5y ago
https://www.whatsmydns.net/#A/facebook.com https://www.whatsmydns.net/#A/facebook.com
_joel 5y ago
Everything is a f*king Facebook problem
poorjohnmacafee 5y ago
Teen depression and suicide rates plummeting right now
drcongo 5y ago
Let's hope it's permanent.
- mrstumpy 5y ago
  You win for best comment
- jose-cl 5y ago
  here's your upvote master
- hkai 5y ago
  Why?
  - elevenoh 5y ago
    b/c most humans are on the wrong end of fb's covert, exploitative attention-manipulation
  - dessant 5y ago
    Because combined with the abysmal state of education in most places, and a general lack of government action, Facebook is an actual threat to our civilization.
    - forgetfulness 5y ago
      People unfortunately love the upsides of misinformation, or perhaps it's the format that makes it easy to build community around shared (misinformed) values, to rally in battles that rage for hours or days for a cause you deeply believe in and can follow by digesting 30-second soundbites on social midea and 30-minute videos on YouTube. People will do this wherever they can talk in a group online, not just Facebook properties. It's... pretty bad actually, I think the only tool that exists right now is censorship, because the bullshit gets created, spread, and wholeheartedly received way faster than debunking will. And censorship is a power that can't be safely entrusted to nobody.
    - hkai 5y ago
v4ult 5y ago
Signal FTW
WallyFunk 5y ago
This would be a golden opportunity to launch your 'Facebook Killer' app. Preferably a social network where people don't pay with their data, but with, you know, a thing called Money.
- smt88 5y ago
  Who would pay money to be the first user of a new social network?
Paianni 5y ago
The level of intelligence of the comments in this thread kinda confirm my suspicions that the armchair experts from Reddit et al (including myself) are discovering this site.
faramarz 5y ago
What are some of the possible scnarios beyond the DNS issue suggested? (and might it be an attack?)
- coolspot 5y ago
  A BGP routing mistake that can cascade into a hard-to-recover-from state of the network where inter-dependencies lock each other.
- heegemcgee 5y ago
  This is what i came to comments for. Unfortunately, we have literally dozens of comments that amount to nothing more than schadenfreude, and another handful of non-FANG employees speculating how one of the largest internet operations in existence could improve their game (lol)
- cwkoss 5y ago
  I doubt this is the case, but someone on twitter was speculating "what if this is fb's infra team going on strike"
randomperson_24 5y ago
World productivity just grew by 10%
- frederikvs 5y ago
  or it went down by another 20%, everybody at first thinking there's something wrong with their internet connection.
mnd999 5y ago
Somebody moved fast.
- drummer 5y ago
  And broke everything
walrus01 5y ago
downdetector looks like a real mess for it. I'm going to parrot the other comment here and say nothing of value was lost. https://downdetector.com/status/facebook/ https://downdetector.com/status/facebook/
baby 5y ago
Wow. I can’t remember the last time whatsapp was done. I pretty much use messenger/instagram/whatsapp to talk to most of my friends and family. I’m happy that I do use other platforms otherwise I would be completely cut off from my parents right now.
fartingflamingo 5y ago
The whistleblower said she wanted to fix Facebook. Mission accomplished, I'd say. For now at least.
deleted 5y ago
[deleted]
emptysea 5y ago
Interesting, even some open source sites like: https://fbinfer.com https://fbinfer.com are down but https://glean.software https://glean.software and https://reactjs.org https://reactjs.org aren't
- jaywalk 5y ago
  Anything hosted on Facebook's infrastructure is down. The two sites that you note are up aren't hosted by Facebook.
  - Graffur 5y ago
    What's react hosted on?
    - jaywalk 5y ago
      Looks like Cloudflare nameservers and Vercel hosting.
- mcintyre1994 5y ago
  According to https://lookup.icann.org/lookup https://lookup.icann.org/lookup both glean and reactjs have Cloudflare nameservers. fbinfer has ns.facebook.com nameservers which are presumably down.
deleted 5y ago
[deleted]
yawnxyz 5y ago
Funny enough, I went to https://www.isitdownrightnow.com/ https://www.isitdownrightnow.com/ to check if Facebook is down, and isitdownrightnow is down itself... probably from the massive number requests coming to check if Facebook is down
- homeskool 5y ago
  yep down for me too
- cecilpl2 5y ago
  https://downforeveryoneorjustme.com/facebook https://downforeveryoneorjustme.com/facebook
  - aaronharnly 5y ago
    Amusingly, that returns: > Is Facebook down right now? > Uh oh! Something went wrong on our side. It's not you, it's us. Feel free to contact us if this persists.
- skizm 5y ago
  https://downdetector.com/ https://downdetector.com/ seems to be working for me at least.
  - horsellama 5y ago
    'Unusual traffic patterns detected' now
  - spiantino 5y ago
    It's amusing that the top 3 trending reports are the FB sites that are down, and then the mobile carriers themselves, presumably because when FB doesn't load they assume it's their mobile network's fault. People really do think FB is the internet
    - Sebb767 5y ago
j3th9n 5y ago
I like it, it feels like it's 1999.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
htrp 5y ago
Facebook is proving that it's systemically important by taking the entire site down..... Zuckerberg is taking his ball and going home unless you stop writing mean things about him /s
- spaceywilly 5y ago
  Let’s hope it’s this. Everyone will just shrug and move onto the next hopefully less evil site
  - hkai 5y ago
    Which one?
    - julianlam 5y ago
      lobste.rs? mastodon?
- oneeyedpigeon 5y ago
  Used to have a manager that we swore did exactly that. Every time he was away on holiday, mysterious site problems to prove his worth!
  - gagege 5y ago
    I had the exact opposite and it was hilarious. Every time my manager (a great guy and really good at what he did) was away for a week the sprint would go very smoothly.
- jimkleiber 5y ago
  Lol exactly what I was thinking. I'm trying to keep my tinfoil hat in the closet and yet it seems odd that after there is a huge FB whistleblower story on 60 Minutes last night, all of FB goes down today. I really hope it's just some internal technical error and not a "see, despite the bad things of FB, you really need us" move. It's probably trivial, the timing just seems weird to me.
totaldude87 5y ago
wonder how much of internet traffic as a whole is down now..
dkarp 5y ago
Someone save a risky release for 9am on a Monday morning? Decided Friday afternoon was too risky?
- grayhatter 5y ago
  0830 actually :/ But to be fair... seems like it was a good call to not do it Friday night :D
  - dkarp 5y ago
    If they chose 8:30, then it must have been really risky! ;)
    - euroderf 5y ago
      The Honolulu office is getting ready for a long night :)
XiS 5y ago
https://www.cyberciti.biz/humour/a-haiku-about-dns/ https://www.cyberciti.biz/humour/a-haiku-about-dns/
rootusrootus 5y ago
And a lot of people seem to be coming to HN to find out why, judging by how laggy HN is getting right now...
- moffkalast 5y ago
  Well it sure is the place to find out for sure.
tinyprojects 5y ago
Oculus is also down
- ourcat 5y ago
  Indeed. People seem to forget that when Facebook goes down, it's not just your feed of depressing posts, photos and messages that go away, but also the entire Oculus VR platform, since they demanded a FB account to use Quest headsets.
orthecreedence 5y ago
Good riddance.
tannhaeuser 5y ago
Expecting to get messages on WhatsApp alternatives tonight ...
johnbaker92 5y ago
Let this be permanent - not a huge loss for humanity.
thedudeabides5 5y ago
Checked isitdownrightnow.com and said Netflix was also down. Any chance these are related?
- robjan 5y ago
  Netflix is up
deleted 5y ago
[deleted]
CodeGlitch 5y ago
Looks like HN is being hit pretty hard right now?
yawnxyz 5y ago
I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)? How could they all go down at the same time, if they have different teams of engineers running each product separately? Could anyone with some background (or person familiar with the matter) explain how their system's set up?
- vodkapump 5y ago
  Seems unrelated to their infrastructure, the DNS records for facebook.com, instagram.com, whatsapp.com and all derivative domains are wiped clean it seems edit: though saying that, they do run their own registrar... Might've fucked something up over there.
- toast0 5y ago
  WhatsApp and Instagram are both in FB infra. As I understand it, Instagram is fairly integrated with FB services; when I left in 2019, WhatsApp was less so, it was mostly WhatsApp specific containers running with FB's container orchestration on FB machines dedicated to WhatsApp (there was and probably is some dependence on FB systems for some parts of the app, for example the server side of multimedia is mostly a FB system with some small tweaks and specific settings, but chat should be relatively isolated). Inbound connection loadbalancing is shared though. FWIW, WhatsApp (on phones) should be resiliant to a DNS only outage, the clients contain fallback IPs to use when DNS doesn't work, and internal services don't use DNS as far as I remember. At one time, WhatsApp had actually separate infrastructure at SoftLayer (IBM Cloud now), but that hasn't been in place for quite some time now. When I left, it was mostly just HAProxy to catch older clients with SoftLayer IPs as their DNS fallback.
elboru 5y ago
Is it just me or HN also feels kinda laggy?
- amelius 5y ago
  Yes, it's slow here as well, and posting this comment failed the first and second time.
- amelius 5y ago
  Yes, it's slow here as well, and posting this comment failed the first and second and third time.
- amelius 5y ago
  Yes, it's slow here as well, and posting this comment failed the first and second and third and fourth time.
  - bradenb 5y ago
    This is either a hilarious accident or genius comedy.
    - tzs 5y ago
      This is not too rare when HN is being slow and giving those "We're having some trouble serving your request. Sorry!" pages. If you get one of those on your comment submission you have no way to know if the trouble stopped it from accepting the comment or if it accepted the comment and ran into trouble then trying to display the updated thread. For some reason I can't even begin to guess at HN does not seem to have protection against multiple submissions of the same form, so if after getting "We're having some trouble serving your request. Sorry!" on your comment submission you hit refresh again to display the page and the form gets resubmitted, you get a duplicate comment.
      - saagarjha 5y ago
        Earlier today when I was getting these I went to go check the page to see if the comment was posted there. More than once it said it failed but I was able to stop myself from trying again because it was actually there.
klik99 5y ago
Is this related to the outages from lets encrypt root cert expiring? Probably not since this looks like a DNS issue, but still it's a crazy coincidence that two major internet breaking events happen in the same week
- treesknees 5y ago
  There is zero reason to believe it's related at all. It's perfectly reasonable to have multiple large and unrelated failures in the same week. I also wouldn't classify the loss of 1 company, and the expiration of some TLS certificates, as the interconnected network of networks being broken. The Internet has continued to function even if some larger players were unreachable or having issues.
deleted 5y ago
[deleted]
smashah 5y ago
Good, now I can go for lunch.
AzzieElbab 5y ago
it is probably unrelated but HN is crawling
pixelgeek 5y ago
I would like that say that after my "burn it down" comments on another Facebook related post that I had nothing to do with this.
jamespwilliams 5y ago
Is this caused by missing glue records? I can’t resolve any of FB’s nameservers. Anyone know how that could happen?
- wut42 5y ago
  the glues are still there-- it's not a DNS issue but a network one. Their ASN has mostly been withdrawn from everywhere.
- zekica 5y ago
  The glue records are fine from my end: dig -t NS facebook.com @a.gtld-servers.net
carrja99 5y ago
Good.
keithnoizu 5y ago
somewhere an engineer is begging a mnesia instance to come back online.
html5web 5y ago
Great time to take a break from Facebook and Instagram. Use Telegram instead of WhatsApp
hotz 5y ago
The joy that people are getting from this is quite shitty. I hate social media but there are people earning a living working for these companies. Like others have pointed out, businesses and neighborhood watches rely on tech like this. At some point we've all had sites/apps go down, in a situation like that the last thing you want is people enjoying it. The lack of empathy in this thread is telling.
- sasaf5 5y ago
  It pales in comparison to the lack of empathy facebook has shown to its user herd.
yholio 5y ago
I had problems with my internet connection and loaded my ISPs site. Strangely, my bill was paid. Even stranger, some sites load while others do not. Then it hit me: I am so dependent on Facebook owned properties (Whatsapp, Facebook, insta) that a Facebook failure looks to me like an internet failure.
rvz 5y ago
From [0] > ...there is no limit to the scandals, leaks, whistleblowers, lawsuits or penalties that will bring the Facebook mafia down. Fine. 'Literally' bringing the Facebook mafia down like that would do. But only for now. [0] https://news.ycombinator.com/item?id=28742179 https://news.ycombinator.com/item?id=28742179
wejick 5y ago
Would be very interesting if they release the RCA to the public
keithnoizu 5y ago
some poor engineer is sobbing over a split brain mnesia cluster right now praying to get the thing back up.
throwawaylolx 5y ago
When I noticed HN was loading slowly, I already knew FB was down.
rcarmo 5y ago
Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.
- wyager 5y ago
  Can you not use an Oculus headset if FB servers are down? That’s absurd.
  - rcarmo 5y ago
    Some preloaded apps work (like YouTube, Firefox), but stuff like settings, the lobby, etc., are very slow or display "Unable to Load" messages. Any game that relies on your friends list seems to freeze for a while, then try to carry on.
  - Reubachi 5y ago
    Yep. We've come full circle, where techies are rediscovering the original hatred for the Oculus, that it is tied to a social media walled garden, for some reason.
    - boston_clone 5y ago
      I think you may need a refresher on their history - https://en.wikipedia.org/wiki/Oculus_(brand)#History https://en.wikipedia.org/wiki/Oculus_(brand)#History When FB announced they would be buying Oculus, they promised that no social media integration would be required. FB breaking that promise is not the same as Oculus having that requirement from the get-go. What original hatred are you talking about??
      - Reubachi 5y ago
        Originally, you did not need a facebook account to use oculus after purchase. They framed this as "you do not need to integrate social media/facebook". This ^ means fuck-all, because at that time (day 1), their oculus services where hosted in the same infrastructure as their social media services. Last year, they got rid of "you do not need a facebook account". But in all situations since inception, all of your data is passing through the same infrastructure as facebook data. It may not be being exposed, or targeted for advertising, but this WAS a huge point of contention years back.
gprasanth 5y ago
Suspecting it might be related to the recent letsencrypt cert authority expiring? Was just debugging an issue earlier today and just couldn't help wondering how much of the internet is secured by letsencrypt. All of the static hosts providing free SSL: vercel, netlify, render, firebase hosting, github pages, heroku etc. ... It does work on modern browsers and devices but goes terribly broken on a lot of old devices.
- jaywalk 5y ago
  Obviously not possible to check right now to provide proof, but I feel quite confident in saying that Facebook does not use Let's Encrypt. It's also clearly not an SSL issue.
  - gprasanth 5y ago
    You're right. Fb doesn't seem to be using letsencrypt. https://crt.sh/?q=facebook.com https://crt.sh/?q=facebook.com On a side note, the amount of phishing sites using letsencrypt and having a domain similar to facebook.com is quite appalling.
  - deleted 5y ago
    [deleted]
sAbakumoff 5y ago
rich people serve their revenge for pandora papers
foobaw 5y ago
This is taking a longer time than expected for a company like Facebook - must be serious where a rollback isn't possible or trivial.
- chasd00 5y ago
  from what i understand (take with grain of salt) remote access to the routers affected is down. So they need to be physically plugged in to address the issue. hence some of the other "scrambling private jets" comments referring to getting the right people physically plugged in to the right routers.
- derwiki 5y ago
  I wonder if everyone refreshing the sites/apps trying to get it to load is contributing to the problem
  - XCSme 5y ago
    Probably not, from other comments it looks like there was a wrong configuration rolled out, and now they are logistically struggling to get access to fix them.
em3rgent0rdr 5y ago
How much revenue does facebook loose per hour down?
- dmoy 5y ago
  This is very hard to get exactly right, because traffic isn't constant at all times, and you don't know if people won't just make up for lost time using facebook at another time of the day, etc. So you can't really know. But, a good rule of thumb right now is about $10,000,000 per hour.
- missedthecue 5y ago
  Facebook made $29 billion last quarter which translates to $315,217,391 per day. Divide that over 24 hours in a day, and it's ~$13 million per hour. Of course, depends on the hour of day. Facebook likely makes more ad money when North America is awake than when Asia is awake for instance.
throwaway78981 5y ago
Signal is welcoming everyone: https://nitter.mailstation.de/signalapp/status/1445062426739855366 https://nitter.mailstation.de/signalapp/status/1445062426739...
- maxxxxxx 5y ago
  Ironically, I cannot send messages on Signal right now. They can’t handle the extra load?
  - coolspot 5y ago
    Signal replicates each message to NSA and FB, so when one is down, Signal’s backend fails with a timeout error.
    - lavp 5y ago
      Source?
- kitkat_new 5y ago
  to also face a failure in the single point of failure - system?
- MajorSauce 5y ago
  Still an American centralized platform. Federated Matrix is the way to go!
unusximmortalis 5y ago
for me even this site loads very very slowly. pinging google name server is fast as usual. it could be a more wide problem not just FB related.
wly_cdgr 5y ago
...Including that one
caturopath 5y ago
What would a full day of WhatsApp outage mean for the world?
TedShiller 5y ago
This solves the disinformation problem
deleted 5y ago
[deleted]
amir-h 5y ago
Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?
- o10449366 5y ago
  Either many HN users are in glee over FB's potential demise ...or many HN users are also avid FB users (and now have to resort to backup sources of entertainment)
- uyt 5y ago
  If I wanted to know if a site is down for everyone or just me, I would check twitter/hn first before checking the down detector sites
kossTKR 5y ago
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: >As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures. User is providing live updates of the incident here: https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfd4dyv/?context=3 https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...
- cotillion 5y ago
  So, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.
  - mekatter 5y ago
    These are readily available, OpenGear and others have offered them forever. I can't believe fb doesn't have out of band access to their core networking in some fashion. OOB access to core networking is like insurance, rarely appreciated until the house is on fire.
    - Sebb767 5y ago
      It's quite possible that they have those, but that the credentials are stored in a tool hosted in that datacenter or that the DNS entries are managed by the DNS servers that are down right now.
      - mekatter 5y ago
        You are probably right but if that is the case, it isn't really out of band and needs another look. I use OpenGear devices with cellular to access our core networking to multiple locations and we treat them as basically an entirely independent deployment, as if it is another company. DNS and credentials are stored in alternate systems that can be accessed regardless of the primary systems. I'm sure the logistics of this become far more complicated as the organization scales but IMHO it is something that shouldn't be overlooked, exactly for outlier events like this. It pays dividends the first time it is really needed. If the accounts of ramenporn are correct, it would be paying very well right now. Out of band access is a far more complicated version of not hosting your own status page, which they don't seem to get right either.
calebm 5y ago
"When you are strong, appear weak."
deleted 5y ago
[deleted]
amir-h 5y ago
Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB? [I'm also getting server error trying to submit this comment]
jangrahul 5y ago
makes me wonder, why dont porn sites ever seem to go down ?
mrkickling 5y ago
When I do dig instagram.com I get an A response for this IP: 31.13.65.174 or similar addresses, which leads to an empty page.
shkkmo 5y ago
I keep getting non-dns errors from Hackner News as well. There appears to be some sort of broader incident happening? It's not just lag, I keep getting the "We're having some trouble serving your request. Sorry!" page. Edit: HN related thread https://news.ycombinator.com/item?id=28749476 https://news.ycombinator.com/item?id=28749476
deleted 5y ago
[deleted]
EB66 5y ago
Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached [root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: Name: downforeveryoneorjustme.com Address: 172.67.166.187 Name: downforeveryoneorjustme.com Address: 104.21.91.48 [root@app ~]# Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.
- uo21tp5hoyg 5y ago
  Yeah Cloudflare have said they're being flooded with Facebook DNS retries and had to get extra hands on deck to deal with the influx.
- rstupek 5y ago
  Yes same issue for me with 8.8.8.8, errors for everything but big domains.
- pixxel 5y ago
  No idea if it’s related but a lot of Tor websites have also been offline all day (BBC, ProtonMail etc).
- r721 5y ago
  See this thread (with replies): >Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com http://facebook.com https://twitter.com/jgrahamc/status/1445066136547217413 https://twitter.com/jgrahamc/status/1445066136547217413 >Our small non profit also sees a huge spike in DNS traffic. It’s really insane. https://twitter.com/awlnx/status/1445072441886265355 https://twitter.com/awlnx/status/1445072441886265355 >This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled. https://twitter.com/TheodoreBaschak/status/1445073229970763781 https://twitter.com/TheodoreBaschak/status/14450732299707637...
zrail 5y ago
hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook. I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.
- poetaster 5y ago
  I used to do this properly. One vanity got the better of me. Got some work to do. TGF SQL.
- chasd00 5y ago
  There will be so many meetings over this. If powerpoint was listed on the stock exchange i'd say now's a good time to buy hah.
mdani 5y ago
Is this in some way connected to the Facebook data leak of 1.5 billion users? The timing seems quite odd that both these things happen around the same time.
AtNightWeCode 5y ago
So they managed to remove facebook.com from 1.1.1.1 and 8.8.8.8. That is impressive. Not something anyone can achieve in such short time by even trying.
HellDunkel 5y ago
I wish it would stay that way.
slackfan 5y ago
So how much do we need to pitch in to keep it down?
slackfan 5y ago
So how much cash do we need to pitch in to keep it down?
ProjectArcturis 5y ago
Have they tried turning it off and then turning it back on again?
mindcrime 5y ago
Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don't you get that?
- GreenWatermelon 5y ago
  At this point in time, I doubt this holds true. Facebook is just too big and pervasive that such an outage would be treated by its users like an internet outage or a power outage. Once it's back online, everyone will forget.
- dgb23 5y ago
  Is this a quote?
  - mindcrime 5y ago
    Yes, it's from "The Social Network". It's a scene where Mark Z. is explaining to Eduardo how important it is that the servers stay up all the time. Of course it was, as far as I know, ficitonalized in the first place, although it rings true (in context) to some extent. What I wonder is, how much is that true now? That is, how much downtime would FB have to experience for enough users to start leaving, to the point that it might prompt a serious exodus.
  - qnsi 5y ago
    google suggest its from a social network movie
jbverschoor 5y ago
It's funny that hackernews is now overloaded with distracted people ;-)
deleted 5y ago
[deleted]
jurajmlich 5y ago
DNS servers of a major internet provider in the Czech Republic are down now. Probably not a coincidence (other DNS server's stats show increased traffic so my guess is that Vodafone's DNS servers were unable to cope with the increased traffic and crashed https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099). It's crazy that half the country doesn't have internet because Facebook stopped working.
john37386 5y ago
Imagine if they need access to fb.com email to re-enable the access for the on-site technician.
Ristovski 5y ago
Discord [1] is taking a toll from the increased traffic as well: "We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service. We have temporarily disabled typing notifications. We expect these to be re-enabled soon." [1] https://discordstatus.com/ https://discordstatus.com/
- Le_Dook 5y ago
  Yeah, it seems like a lot of places online are facing the same. Even here has been bad for me
Animats 5y ago
DNS configuration is becoming a single point of failure. A few weeks ago, many services running out of AWS West 2 failed because the within-the-datacenter DNS system broke down somehow.
jurajmlich 5y ago
It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it (https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099). Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)
- blntechie 5y ago
  All most people have to do now is install an app and it takes care. But messaging need to go from media and news.
- megous 5y ago
  If only the news reporting was not as stupid as "internet is not working at UPC", instead of DNS resolvers at UPC crashed, here's what you can do... Anyway, I didn't even notice since I run knot-resolver at home. I wonder what it will be like connecting Facebook back to the internet, thundering herd and everything...
  - yourad_io 5y ago
    I suspect that the DNS aspect will be fine. The middle DNS servers only need one valid response to cache it for $TTL, but they can't cache SERVFAIL.
    - megous 5y ago
      I mean connecting your company to the internet when you have billions of devices waiting in the line to fetch updates, or whatever. Will not that be an issue? Re-enabling routing to such a massive internet service...
      - yourad_io 5y ago
        I suspect it will slowly get there. We'll find out, it is recovering now.
Sophira 5y ago
Somebody just had their very own "onosecond". https://www.youtube.com/watch?v=X6NJkWbM1xk https://www.youtube.com/watch?v=X6NJkWbM1xk The video is one that Tom Scott published in June 2020 about the worst typo he ever made in one of his prior jobs, and while the Facebook mistake is almost certainly not going to be anything irrecoverable like this one, you can bet that Facebook pride themselves on being available all the time.
dizzant 5y ago
Reading the thread, I'm surprised at the number of nearly identical "How much do we have to pay to keep it down? xD" posts I'm seeing, often from throwaway accounts. Some accounts with multiple near-identical posts within the same minute. Could this be a coordinated smear in HN comments?
- cwkoss 5y ago
  I think a lot of people just think facebook is bad for society. I do.
midnightdiesel 5y ago
Hopefully it never comes back up!
_yo2u 5y ago
value of fb directly proportional to traffic flow curious about why fb is down ;)
StapleHorse 5y ago
That explain the new contacts in Telegram.
mcintyre1994 5y ago
I love how understated companies always are about things like this. > Facebook said: "We are aware some people are having trouble accessing our apps and products. We are working to get things back to normal as quickly as possible and apologise for any inconvenience." https://www.bbc.co.uk/news/technology-58793174 https://www.bbc.co.uk/news/technology-58793174
teddyh 5y ago
Here we can see why you should not have all your DNS servers in the same AS (in this case, AS32934).
shockeychap 5y ago
I feel for the sysadmins who are fighting ulcers and migraines at the moment, but I can't shake feeling that the world is just a little bit better for this small window of time.
not2b 5y ago
This is a great argument for the antitrust authorities to break up Facebook. Allowing the big social media companies to buy each other creates a single point of failure. If Instagram and WhatsApp were separate companies, a technical disaster at one would not take out the other two.
chungy 5y ago
Even Facebook's Onion site isn't working: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion/ http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t... Fascinating simply that it's apparently not just a DNS issue.
- alexvoda 5y ago
  Any idea what is the explanation for this?
  - deleted 5y ago
    [deleted]
dclaw 5y ago
Delete Facebook / Instagram / WhatsApp when it comes back up. They are all trash.
ionwake 5y ago
Did I just read that the Facebook IRC fallback went down too?!? I was about to say what’s wrong with freenode ( but yeah on 2nd thoughts let’s not talk about freenode )
taftster 5y ago
What I think is interesting is the effects of this type of thing across peripheral news sites, like HN. I wonder how much spike HN gets with people rushing here to find out what's going on and to read the (articulate) related discussions.
ionwake 5y ago
Why is this not the first post on hn? It’s double the points and has over 590 comments?
imalerba 5y ago
Seems like Telegram went down with a big whatsapp-is-down hug of death.
- blntechie 5y ago
  Yep, Telegram stopped sending messages a while back and not loading at all for me now.
marstall 5y ago
between outage and whistleblower, this has got to be the worst day in facebook's life
blobbers 5y ago
well... this is unfortunate: ; <<>> DiG 9.10.6 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36072 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
iseanstevens 5y ago
Interesting Timing!
aero-glide2 5y ago
Does this mean Oculus is unusable now?
shmiga 5y ago
A lot of addict's will now feel how is it to live in the reality not scrolling dump images. I hope this will be tradition at least once a month.
- areactnativedev 5y ago
  And here we are, looking at the xth top-level comment on HN...
- qnsi 5y ago
  I am an addict. I refresh this thread waiting for information: fb is back online. I need help but this is too hard for me. I uninstall social media once a week but install two days later. I should probably go to therapy with this, but I am not sure I wouldnt be laughed at
  - ppqqrr 5y ago
    https://heyfocus.com/ https://heyfocus.com/ worked for me, maybe it’ll help (if you’re on Mac). Addiction to social media is a real problem; thousands of engineers are paid to make sure that these products ensnare your attention. It wouldn’t be odd if it takes a few bucks of your own to rescue yourself. Don’t hesitate to seek help, no one will laugh at you.
teddyh 5y ago
Interestingly, their .onion site¹ is also down. https://en.wikipedia.org/wiki/Facebook_onion_address https://en.wikipedia.org/wiki/Facebook_onion_address
- fluxem 5y ago
  This is expected. I guess, the internal DNS is down, so the whole infrastructure is broken
- coolspot 5y ago
  The onion site is just a reverse-proxy to the main web-site. So if the main site is down (due to internal DNS or BGP issues) onion reverse-proxy can’t get to it as well.
EGreg 5y ago
Anyone able to Connect w WhatsApp at the moment?
alkonaut 5y ago
Here is a handy troubleshooting flowchart for megacorp outages: > Is it a DNS issue? -> yes It can be used in reverse as a postmortem too.
nycdatasci 5y ago
Ignore.
- wrycoder 5y ago
  19 hours ago??
scumcity 5y ago
hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.
geocrasher 5y ago
Once again, it's DNS. https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tune-of-let-it-be https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...
clipradiowallet 5y ago
In other news, a bunch of people got a lot more work done today than normal I suspect...
suyash 5y ago
Facebook hacked?
pat-jay 5y ago
Finally! A small (or not so small) outage for FB, a large benefit for mankind :)
deleted 5y ago
[deleted]
bitzlab 5y ago
This morning around 10:00 AM I've been posting on a forum a message in to which I have said that I'm hoping Facebook will get shut down.. Now, Facebook situation you see it might be my fault. Sorry folks, didn't meant that for real.. I still need FB's products! =)) https://natrmd.com https://natrmd.com
chadlavi 5y ago
I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.
- wooger 5y ago
  Also amusingly I think quite a lot of employers use WhatsApp for part of their disaster communications plan. If this happened then some wider issue (github down) there’d be chaos.
  - phatfish 5y ago
    The UK government will grind to a halt if WhatsApp is still down tomorrow. Well, more than usual that is.
swayson 5y ago
This really is a fascinating case-study of what is truly resilient systems. More often than not, they are not centralized.
throwaway123x2 5y ago
The cynic in me wonders if this is related to the Pandora Papers leak
jose-cl 5y ago
In this context, I remember youtube+pakistan issue[1]. I also wonder how an AS/BGP manager do his/her job... I imagine a guy/girl changing a text file in a old console. Anyone knows? [1] https://www.infoworld.com/article/2648947/youtube-outage-underscores-big-internet-problem.html https://www.infoworld.com/article/2648947/youtube-outage-und...
platz 5y ago
[redacted]
- treesknees 5y ago
  What evidence is there to suggest this is due to Facebook being down?
fartingflamingo 5y ago
From the archived ramenporn reddit comment thread at [0]: > This must be incredibly stressful so for your sake I hope you sort it out quickly... but for the world's sake, I hope you fail and make the problem worse before jumping ship followed by every other engineer, leaving it to Zuckerberg to fix himself. But I still hope it's not too stressful for you! https://archive.is/Idsdl https://archive.is/Idsdl
blablablub 5y ago
Glad to report that facebook's dns in china is not affected. You can dig facebook.com and the depth of the internet happily reply with a random ip address as usual.
liendolucas 5y ago
Does anyone have a reasonable guess on how much money they have already lost?
- gmiller123456 5y ago
  Likely $0. Ad views lost now will likely be made up for later. And even if there is a reduction in views, it just makes other views more valuable. Facebook doesn't have real competitors, so the money isn't going anywhere else.
jeffrom 5y ago
Is facebook being down causing hacker news to get the hug of death??
liendolucas 5y ago
Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.
- agilob 5y ago
  that's not how post-mortems work
6510 5y ago
Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.
- DreamScatter 5y ago
  Do you mean Jacque Fresco?
  - 6510 5y ago
    yes
Animats 5y ago
There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms 5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms 6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms 7 * * * ... So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.
- john37386 5y ago
  Yeah the patch to fix BGP to reach the DNS is sent by email to @facebook.com. Ooops no DNS to resolve the MX records to send the patch to fix the BGP routers.
  - yoelo 5y ago
    Seriously? Is that how it works?
    - john37386 5y ago
      I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
      - boshomi 5y ago
        Sheera Frenkel: Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
jmfldn 5y ago
Yet another reason to not over-rely on a few big tech companies for the majority of the planet's communication. Forget concerns about competition, monopolies and so on for now (as important as they are), what we want are many social networks, video conferencing apps, messenger apps. Every country should strive to build their own Google or FB, or certainly many more should. State-backed if needed. It's a question of resilience and security as much as anything.
koksik202 5y ago
my home connection with ISP is down Vodafone Ireland, so I guess they have such a big churn in Vodafone from FB BGP routes that it blew Vodafone network. Is it DNS or routing issue?
sabujp 5y ago
India runs on whatsapp. They'll have more backups now.
marchingvehicle 5y ago
Where could the physical data centers be that they need to access? How far away could it be?
- chippy 5y ago
  maybe, there are reports (i.e. unverified tweets) that employees cannot access sites due to the security systems also being down. I imagine email, and messaging for employees would also be down too. It may be very hard for employees to get to the physical boxes, and/or bypass any physical or software security systems.
MrYellowP 5y ago
I keep trying to submit to HN but I keep getting an error. What's wrong with the internet? FaceBook is down. My friend from Slovenia is having trouble with discord. It eats his messages. I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received. TrackMania players have talked about having input lag. ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this) ycombinator turned out to be giving only errors. Some sites I've found via google results seem to report that they are suffering from slow connections. Do you have anything to add to this?
- user3939382 5y ago
  Big ISP outages in NYC right now
- saltyfamiliar 5y ago
  I'm having issues with telegram as well. Images won't send and the app continuously says "updating" on the top status bar. How could facebook dns issues cause this?
  - karlzt 5y ago
    Because everyone is using Telegram instead of Whatsapp due to WA being down.
MrYellowP 5y ago
I keep trying to submit to HN but I keep getting an error. What's wrong with the internet? FaceBook is down. My friend from Slovenia is having trouble with discord. It eats his messages. I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received. TrackMania players have talked about having input lag. ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this) ycombinator turned out to be giving only errors, but now seems to be working occasionally. I can not submit anything, though. Some sites I've found via google results seem to report that they are suffering from slow connections. Do you have anything to add to this?
- tiluha 5y ago
  Some of those can probably be explained by facebooks traffic being redistributed to other services, overloading them
- agilob 5y ago
  A few wordpress blogs crashed because addon facebook pixel is crashing. Very intensive lesson for the internet!
- Animats 5y ago
  What's wrong with the internet? With Facebook down, some large DNS servers seem to be struggling with the extra load of failing requests to look up "facebook.com". Cloudflare reports overload with their DNS server at 1.1.1.1, although that's working for me. Billions of things worldwide are trying to connect to Facebook. The lookup which normally returns the IP address for facebook.com on the first try now requires trying a.ns.facebook.com, b.ns.facebook.com, etc. several times each before giving up. Probably several times a minute for everyone who has a Facebook app in their phone turned on. That may be using a big fraction of world DNS resources. Vodaphone Ireland seems to be struggling with a DNS overload right now, per the Irish Independent. Also, their status page can't find "Dublin" as a city.
lazlee 5y ago
Hopefully forever.
pytlicek 5y ago
Who else sees their deleted messages on WhatsApp that shouldn't be there? https://news.ycombinator.com/item?id=28749652 https://news.ycombinator.com/item?id=28749652
freediver 5y ago
Terrible day for many people. Both working for Facebook and those depending on their services.
polynomial 5y ago
"It's always DNS."
asduoihfijnu 5y ago
ok mom i commented
htrp 5y ago
In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.
- mkr-hn 5y ago
  I'm genuinely not sure if the reports I heard of employees being locked out of the systems they need to fix it because their network is down are jokes or true.
the-dude 5y ago
It is about time to speculate about sabotage, a disgruntled employee or something more exotic. All this BGP talk is boring.
- sheepybloke 5y ago
  Like this being tweeted: https://twitter.com/YourAnonOne/status/1445082304393719818?s=20 https://twitter.com/YourAnonOne/status/1445082304393719818?s...
_yoqn 5y ago
It's about damn time. Hopefully they stay down. It will do the world some good (long term) to have some time away from this platform and platforms like it.
emmap21 5y ago
Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.
- jy3 5y ago
  No. All the ones you mentioned are up.
- _fizz_buzz_ 5y ago
  Youtube and google are definitely working for me without any problems (haven't tested telegram or zoom).
pytlicek 5y ago
Who else sees their deleted messages on WhatsApp that shouldn't be there? https://twitter.com/Pytlicek/status/1445072626729242637 https://twitter.com/Pytlicek/status/1445072626729242637
- reilly3000 5y ago
  That is wild and definitely newsworthy. Capture as many screenshots and data as you can. FWIW it seems possible that the messages remain cached locally on your device but deleted from their servers, and with their outage they aren’t being updated to delete?
  - reilly3000 5y ago
    This is something to get in front of a tech journalist who covers Facebook. It’s a major breach of trust. Probably hit one of your favs with a tweet, but they also tend to list their contact info on author pages of the sites they publish for.
deleted 5y ago
[deleted]
cestith 5y ago
Out of band management is an important feature for the reliability of your network.
todd-davies 5y ago
Perhaps allowing Facebook, WhatsApp and Instagram to merge was efficient after all - now that they have synchronized outages, people finally have a chance to get on with their lives, free of clickbait news and misinformation.
mupuff1234 5y ago
Is there any site that tracks number of users for messaging apps? I'd be really curious to see if signal\telegram\etc are seeing a big bump.
lucidbee 5y ago
The timing of this is so rich in irony I can't help but wonder if there is an element of internal sabotage. How many FB employees hate FB right now? The latest expose of FB is both effective and truly awful. I can't imagine feeling good about a FB job. And it's gotten worse! Now they look like they can't even keep their websites up.
- greeklish 5y ago
  Can we really ever know? There are million of $ at stake!
- ttobbaybbob 5y ago
  Perhaps we'll find out. As fun as internal sabotage would be, schadenfreude-wise, i think it much more likely this will turn out to be a time when Hanlon's Razor applies
baalimago 5y ago
Aren't there places around poorer countries where Facebook is basically an ISP? What about them? They have literally 0 info. https://tcrn.ch/3kOHco1 https://tcrn.ch/3kOHco1
LuisMondragon 5y ago
Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
baalimago 5y ago
Aren't there places around poorer countries where Facebook is basically an ISP? What about them? https://tcrn.ch/3kOHco1 https://tcrn.ch/3kOHco1 2Africa cable, as an example
codediesel 5y ago
still down, people going to signal to have a chat
dghughes 5y ago
Twitter seems to be a big buggy now too maybe just a coincidence. User comments under posts are not appearing.
jontro 5y ago
Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.
emmap21 5y ago
There is a global outage this morning starting from 8:00 AM. This list has Google, Tiktok, Zoom, Slack and of course FB products and services.
decrypt 5y ago
From this tweet: https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099 "Because of missing DNS records for http://Facebook.com http://Facebook.com, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."
jleyank 5y ago
I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I can never remember...
- aenis 5y ago
  most of what they do, they do with in house tools, and custom-everything, including hardware. as a consequence, for some classes of problems there are no experts - not at facebook, not anywhere. i feel for their netops people. uncharted territory with the whole world watching and, no doubt, a lot of morons from management trying to be "helpful" in getting this nice crisis resolved. for any crisis there is always a bunch of clowns with MBAs that consider it their golden opportunity to shine (nearly always at someone elses expense)
firstSpeaker 5y ago
Is there any place to see how the overall internet bandwidth usage has changed during this outage?
zanethomas 5y ago
hahaha, good riddance
38932ur98u 5y ago
This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.
jacke 5y ago
They made they own BGP tools and looks like it failed https://www.youtube.com/watch?v=wHfYUbKNEyc https://www.youtube.com/watch?v=wHfYUbKNEyc
typingmonkey 5y ago
If it is an DNS error, why is the .onion site also offline? - https://en.wikipedia.org/wiki/Facebook_onion_address https://en.wikipedia.org/wiki/Facebook_onion_address - facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion
- detaro 5y ago
  It's not just/primarily a DNS error.
- ApolIllo 5y ago
  My guess is that the FB backend also required DNS. The .Onion site isn't backed by a backend built on a onion native stack (is that a thing?)
- lifthrasiir 5y ago
  DNS outage is an outcome of faulty BGP updates. As such not only the Internet can't see the FB network, there is also no connectivity from the FB network to the Internet right now.
littlecranky67 5y ago
The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?
- mdavidn 5y ago
  Sounds like automation deployed a configuration update to most of Facebook's peering routers simultaneously. Something similar brought down Google in 2019.
  - littlecranky67 5y ago
    If so, then it would simply be a BGP issue - no FB servers reachable, as all routes are down. But media+claims a combo of BGP/DNS. Hard to believe world-wide border routers, only responsible for networks containing DNS servers, are misconfigured. I am rely curious about that post-mortem :)
    - mdavidn 5y ago
      I think it was only a BGP issue. The DNS servers apparently shared the same peering routers as the rest of Facebook's infrastructure. Everyone focused on DNS because that's the first sign of failure to an end user.
vvpan 5y ago
Not an indepth technical comment here but: seeing a tech megacorp go offline for a day makes me very very happy.
MrStonedOne 5y ago
Rumormill is suggesting that facebook badge readers are also down causing issues with trying to get to the servers to manually fix them. https://twitter.com/sheeraf/status/1445099150316503057?s=21 https://twitter.com/sheeraf/status/1445099150316503057?s=21
mkaszkowiak 5y ago
This outage is huge. I'm waiting for the write-up, assuming they release one
TedShiller 5y ago
This is honestly the best feature Facebook has ever developed. I hope it's permanent. It has the following effects: you feel better about yourself, you can spend more time with your family, you are more productive.
mro_name 5y ago
frankly, who cares? Seriously. Those services are toxic for years now and everybody knows that. Who still uses them occasionally let alone relies on them can't be helped, can they?
deleted 5y ago
[deleted]
Animats 5y ago
Facebook outage is now the top story on CNN and Fox. Facebook stock down 5%. Facebook is not returning calls from Fox, or CNN.
someonehere 5y ago
When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt. I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.
tomohawk 5y ago
Many local governments use FB to get info out. Events like this show they should use multiple outlets instead of the big monopoly. Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.
muthdra 5y ago
Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow and "having trouble serving requests".
Animats 5y ago
Outage is top story on CNN and Fox. Facebook is not returning their calls. Sheera Frenkel at the New York Times has been able to get a little more info, but not much. Now Twitter is starting to have problems with overload.
deleted 5y ago
[deleted]
muthdra 5y ago
Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow.
babuskov 5y ago
Is HN hit by something as well? It's loading really slow for me.
- runawaybottle 5y ago
  I think the mods throttled logged in users to discourage over-discussion and thread creation. There’s no rhyme or reason for read-only to be snappy and logged-in to be crawling.
erjjones 5y ago
Does everyone just buy in that this is just a network change gone wrong? OR could they be mitigating a breach/hack? OR could it be some other theory?
polack 5y ago
After changing the screen resolution all operating systems will prompt the user if the applied settings where correct, otherwise it will time out and reset to last known good setting. Maybe time for the core internet infrastructure to implement something similar? :)
- thealistra 5y ago
  You can’t really make a system that will unboil an egg.
barbs 5y ago
I find myself a little bit happy that it's down. I use Facebook quite often, but mostly because everyone else I know uses it. If everyone is forced to find an alternative, that'd be fine by me.
DebtDeflation 5y ago
I assume all of the tertiary sites that use "Login with Facebook" are broken now too? So glad I never adopted that.
pmlittle 5y ago
This is all total left brain looping and complexity coming home to roost.
quattrofan 5y ago
And right now, the world is a slightly better place
EGreg 5y ago
Telegram seems down too, is it down for you?
jacobwilliamroy 5y ago
Should say "Facebook owned. Sites are down."
antocv 5y ago
Its alive! drill @1.1.1.1 www.facebook.com ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2172 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com. star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36
- kblev 5y ago
  "Sorry, something went wrong. Facebook © 2020"
  - simonklitj 5y ago
    Yes, even Facebook falls prey to the wrong copyright year. Anyway, I got further now to a page that says "Account Temporarily Unavailable." and has the old Facebook layout. Would love a peek inside the Facebook codebase to see how this happens, hah!
    - ultrarunner 5y ago
      Definitely not recent, but you might find this interesting nonetheless: https://gist.github.com/nikcub/3833406 https://gist.github.com/nikcub/3833406
- strenholme 5y ago
  Kinda sorta. There are four DNS servers for Facebook: 129.134.30.12, 129.134.31.12, 185.89.218.12, and 185.89.219.12. Of those, only 185.89.219.12 is up right now (Edit All four DNS servers are now up). For people who want to add Facebook to hosts.txt, the A record (IP) I’m getting right now is 157.240.11.35 (it was 31.13.70.36)
- daniellehmann 5y ago
  See e.g. https://www.digwebinterface.com/?hostnames=facebook.com&ns=all https://www.digwebinterface.com/?hostnames=facebook.com&ns=a... for responses from different nameservers.
david_acm 5y ago
pings to a.ns.facebook.com are no longer timing out
shreezus 5y ago
It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections. I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.
gunshai 5y ago
Posting this comment will be like farting into a hurricane, but here goes. Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity. What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct. Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.
- osrec 5y ago
  Margin calls don't really exist as far as loans are concerned. Once you have agreed collateral, and an agreed schedule of payment, you only get in trouble if you miss a payment, regardless of how the collateral fluctuates in value.
  - gunshai 5y ago
    Okay, but as the value of the collateral approaches 0 your lender asks you to increase your collateral correct?
    - odonnellryan 5y ago
      Yes. But banks won't loan 100% on equities. This scenario would just be basically impossible.
      - gunshai 5y ago
        I am not sure you understand my question / hypothetical. A bank is not the only form of a lender first off, second the reason there isn't a 100% loan on an equity is that it's understood that the value of the underlying collateral can fluctuate. These are called over collateralized loans.
cphoover 5y ago
now if only tiktok would fail
jader201 5y ago
FB seems to be finally loading for me, after nearly 6 hours. This will be a highly discussed topic for a bit.
korethr 5y ago
At 21:44 UTC, facebook.com resolves for me.
amediauk1 5y ago
tracert 129.134.30.12 Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254] 2 3 ms 3 ms 3 ms 172.16.14.63 3 * 5 ms 3 ms 213.121.98.145 4 5 ms 3 ms 4 ms 213.121.98.144 5 17 ms 8 ms 18 ms 87.237.20.142 6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149] 7 * * * Request timed out. 8 * * * Request timed out. 9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185] 10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38] 11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221] 12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173] 13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129] 14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102] 15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97] 17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143] 18 91 ms 93 ms 90 ms 157.240.36.97 19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12] Trace complete. this is what i got now
amediauk1 5y ago
tracert 129.134.30.12 Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254] 2 3 ms 3 ms 3 ms 172.16.14.63 3 \* 5 ms 3 ms 213.121.98.145 4 5 ms 3 ms 4 ms 213.121.98.144 5 17 ms 8 ms 18 ms 87.237.20.142 6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149] 7 \* \* \* Request timed out. 8 \* \* \* Request timed out. 9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185] 10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38] 11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221] 12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173] 13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129] 14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102] 15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97] 17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143] 18 91 ms 93 ms 90 ms 157.240.36.97 19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12] Trace complete.
ctur 5y ago
It's back as of approx 14:47 PST.
- tcarn 5y ago
  Agreed, website loading here, still no whatsapp though.
josalhor 5y ago
facebook.com resolves again! > ping facebook.com PING facebook.com (31.13.83.36) 56(84) bytes of data. 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=1 ttl=54 time=12.2 ms 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=2 ttl=54 time=12.1 ms 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=3 ttl=54 time=11.7 ms Can't yet traceroute to a.ns.facebook.com tho
nazgulsenpai 5y ago
I just got the login page. It was fun while it lasted.
rstupek 5y ago
Looks like someone found the light switch and turned everything back on!
stonks 5y ago
Facebook and Messenger are working now. Instagram and WhatsApp - not yet.
- sss111 5y ago
  I was hoping it would stay down for longer haha
spicybright 5y ago
Still amazes me their infra team is supposedly the best in the world, and compensated as such, yet things like this happen. Personally I'm glad FB went down for a few hours, but it's hard to imagine how that would happen in the first place.
deleted 5y ago
[deleted]
spurgu 5y ago
They seem to be back now.
ExtraE 5y ago
Any way to read this without an account?
markus_zhang 5y ago
Just a sidenote I think outlook is down as well.
dnndev 5y ago
I am soooo happy I did not even notice! Don’t use any of these apps.
avodonosov 5y ago
When I click the HN link I am presented with facebook login page. I don't have an account so can not proceed. Is the article link just https://facebook.com/ https://facebook.com/ ?
- lom 5y ago
  No, it’s back up by now. The link was to show that facebook was down.
cemoktra 5y ago
Imagine this happening to AWS
infologics 5y ago
I want to to thank you for this good read!! I definitely enjoyed every little bit of it. I've got you book-marked to look at new stuff you post… http://infocampus.co.in/ui-development-training-in-bangalore.html http://infocampus.co.in/ui-development-training-in-bangalore... https://www.firstenquiry.com/ui-development-training-in-bangalore https://www.firstenquiry.com/ui-development-training-in-bang... http://infocampus.co.in/web-designing-training-in-bangalore.html http://infocampus.co.in/web-designing-training-in-bangalore.... http://infocampus.co.in/web-development-training-in-bangalore.html http://infocampus.co.in/web-development-training-in-bangalor... http://infocampus.co.in/react-js-training-in-bangalore http://infocampus.co.in/react-js-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.htm...
Nasreddin_Hodja 5y ago
Ha-ha!
segah 5y ago
My father (https://bit.ly/3acZAAI https://bit.ly/3acZAAI), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake. Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.

2589 points by nabeards 5y ago

GDC7 5y ago

Maybe intentional? Zuck trying to give an example of what a world without FB would look like, kinda saying to detractors what would happen if they had it their way.

forix 5y ago
That would be classic Zuck right there
gmiller123456 5y ago
Maybe "intentional" in quotes. My money is on a major security breach and they've shut everything down until they can deal with it. Even if you go to Instagram by the IP address [1], you get a 400 error. So it looks like things are off line because they want them off line for now. https://31.13.65.174/ https://31.13.65.174/

ballenf 5y ago

One real potential cost to FB here is breaking people's addictions to FB and IG. This might just be the little finger-snap to wake up a sizable chunk of the user base that they life is just a little better during the outage.

deleted 5y ago

[deleted]

tacker2000 5y ago

https://www.status.fb.com/ https://www.status.fb.com/ is back online now

mrfusion 5y ago

Thought experiment: what if they were down for a week and the world completely healed itself?

asduoihfijnu 5y ago

add comment

susahahhaha 5y ago

add comment

ur-whale 5y ago

> Facebook-owned sites are down And the world rejoiced.

sparrish 5y ago

Seems to be DNS related. None of the listed facebook nameservers are resolvable or reachable: a.ns.facebook.com b.ns.facebook.com c.ns.facebook.com d.ns.facebook.com

zulln 5y ago
In the beginning it responded but gave server errors.
- jaywalk 5y ago
  Which seems to indicate a massive infrastructure failure.
  - sakisv 5y ago
    Actually I'd argue that the biggest problem would be to wait for the TTL to expire after you've fixed the problem.
    - jaywalk 5y ago
      The TTL was most likely very low, so I don't see that as being an issue.
sparrish 5y ago
Looks like the routing is goofed. Loops over and over - DDoS attacking themselves. mtr -r -c10 -w -b a.ns.facebook.com Start: 2021-10-04T10:02:50-0600 Loss% Snt Last Avg Best Wrst StDev ... 4.|-- ae-2-rur101.cosprings.co.denver.comcast.net (162.151.51.125) 0.0% 10 12.6 11.9 9.6 19.0 2.9 5.|-- 24.124.155.233 0.0% 10 9.3 10.2 9.1 12.4 1.1 6.|-- 96.216.22.45 0.0% 10 12.0 14.0 11.6 31.3 6.1 7.|-- be-36041-cs04.1601milehigh.co.ibone.comcast.net (96.110.43.253) 20.0% 10 14.6 13.5 11.6 20.5 3.0 8.|-- be-3402-pe02.910fifteenth.co.ibone.comcast.net (96.110.38.126) 0.0% 10 12.2 12.0 11.5 13.2 0.5 9.|-- 173.167.59.170 0.0% 10 13.8 17.8 12.0 34.7 8.4 10.|-- 129.134.40.74 0.0% 10 15.3 12.6 11.4 15.3 1.1 11.|-- 129.134.43.226 0.0% 10 18.9 15.3 12.6 20.3 3.0 12.|-- 129.134.98.166 0.0% 10 12.5 14.2 12.5 20.4 2.3 13.|-- 129.134.54.61 0.0% 10 34.2 30.8 28.9 34.2 1.8 14.|-- 129.134.53.61 0.0% 10 29.8 31.1 28.9 36.5 2.7 15.|-- 129.134.53.61 90.0% 10 31.9 31.9 31.9 31.9 0.0

ivanmontillam 5y ago

One thing that triggers my OCD is leaving the Facebook session open, though it's my own computer. Maybe it's DNS. It's always DNS.

debacle 5y ago
Facebook went down (error page, then 503) before DNS went down.
- marbex7 5y ago
  Same for me.
WallyFunk 5y ago
> Maybe it's DNS If it is, close Facebook as there's probably a BGP hijack going on that is siphoning off personal data and or secrets

raimille1 5y ago

Whatsapp down as well

milankragujevic 5y ago

Also Speedtest.net for me is showing a 503 error page. Seems a large CDN might be having problems. Their status page shows all green. FB and their other sites are also down. edit: I see it's back up and I've been getting downvoted, here's a screenshot of the error for clarity https://i.imgur.com/wvhOwwL.png https://i.imgur.com/wvhOwwL.png

chki 5y ago
If Facebook and WhatsApp and Instagram fails there are probably a lot of people checking whether their Internet works. That might be why Speedtest was overwhelmed.

nabeards 5y ago

Seems to be affecting all Facebook properties.

mrweasel 5y ago
Instagram just returns a 503. Crazy how closely everything seems to be integrated. I’d guess internal networking issues, but the insane that something can bring down all of Facebooks properties.

tut-urut-utut 5y ago

Facebook down, WhatsApp down, but Signal still works. Time for a change? EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.

WallyFunk 5y ago
> Signal still works https://old.reddit.com/r/thehatedone/comments/f160jh/is_signal_centralized/ https://old.reddit.com/r/thehatedone/comments/f160jh/is_sign... Signal is still centralized and uses AWS. So if AWS was to go down, it would affect not just Signal but vast swathes of the Internets.
fsflover 5y ago
Signal had their own share of downtime. How about going to a federated system instead of repeating the same mistakes? (https://matrix.org https://matrix.org)
- JCWasmx86 5y ago
  But still a part would be down, if a server has an outage. How about a system, where every device that is used for chatting is a server at the same time? I wonder whether something like that already exists. Bundle it together with bigger servers to handle the load. If the bigger servers experience outages, the service can still continue, although a bit slower
  - fsflover 5y ago
    Matrix P2P already exists.
    - JCWasmx86 5y ago
      Thanks, didn't know that!
Arathorn 5y ago

deleted 5y ago

[deleted]

jleyank 5y ago

Somebody published vogon poetry and pictures and the internet routed around the damage...

poetaster 5y ago
I have not laughed so loud since before the virus that shall have no vogon name.

deleted 5y ago

[deleted]

kaustubhvp 5y ago

it is always DNS!

bubblehack3r 5y ago

Their stock is down 5% too. Everything is down for them today ;)

i_like_apis 5y ago
Great dip to buy. (I'm not a facebook zealot, but you know it will recover today or tomorrow once the DNS is sorted in an hour or two) Of course there is the whistle-blower issue too...
- derwiki 5y ago
  I don’t think stock dip is related to downtime; anecdotally, I’ve never seen a company’s stock affected by downtime (unless that downtime destroys the business)
  - i_like_apis 5y ago
    You may be right, but theres a Reuters article about the downtime, this is making the news today. I would say Facebook is different because of their scale. Looks like there are a few problems with fb in the news today ...

deleted 5y ago

[deleted]

koprulusector 5y ago

Reddit wasn’t working a few min ago. Broader issue?

blowski 5y ago
Reddit goes down every 10 mins anyway.
- stemc43 5y ago
  this person uses reddit

m_coder 5y ago

I unblocked Facebook right now from my hosts file so I could message someone and couldn't figure out why Facebook failed to load. I tested HN and viola I see that the entire world has sent Facebook requests to 0.0.0.0 lol

interestica 5y ago
You broke it. I didn't receive expected WhatsApp messages and am only now realizing there's no indication within the app that there is even a problem. It only becomes (somewhat) apparent when sending a message never gets a single check mark. Not a graceful failure for the user view.

ricardo81 5y ago

Doesn't seem too clever that Facebook's NS servers are a.ns.facebook.com, b.ns.facebook.com etc. IIRC that kind of setup requires some glue records.

gorgoiler 5y ago
If you mean because the name servers are in the same zone, this is very common. When an NS is returned for a zone, you also get an “additional” A and AAAA to resolve the NS name. It’s called glue. dig NS example.com ; ANSWER example.com. NS ns1.example.com. ; ADDITIONAL ns1.example.com. A 1.2.3.4 Edit: I didn’t see your glue comment when I wrote this.
- ricardo81 5y ago
  Cheers, I'd edited my post. Thought the common wisdom nowadays was to use nameservers on different TLDs and sub-labels for the best resilience. /added, they seem to have glue records so I'd assume it's the nameservers themselves having issues. $ dig NS @g.gtld-servers.net. a.ns.facebook.com ;; AUTHORITY SECTION: facebook.com. 172800 IN NS a.ns.facebook.com. facebook.com. 172800 IN NS b.ns.facebook.com. facebook.com. 172800 IN NS c.ns.facebook.com. facebook.com. 172800 IN NS d.ns.facebook.com. ;; ADDITIONAL SECTION: a.ns.facebook.com. 172800 IN A 129.134.30.12 a.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35 b.ns.facebook.com. 172800 IN A 129.134.31.12 b.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35 c.ns.facebook.com. 172800 IN A 185.89.218.12 c.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35 d.ns.facebook.com. 172800 IN A 185.89.219.12 d.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35

nwatson 5y ago

I guess the "prophets" at Victory Channel / Flashpoint called down Holy Fire on the Facebook infrastructure in retribution ... https://youtu.be/FbSkFuvqFdA?t=1127 https://youtu.be/FbSkFuvqFdA?t=1127 . (I'm an Evangelical Christian but those folks are nuts ... Mario Murillo, Lance Wallnau, Hank Kunneman, Gene Bailey, etc.)

jangrahul 5y ago

makes me think, why dont porn sites ever go down ?

gmiller123456 5y ago
Phrasing!

chasd00 5y ago

can productivity (or emotional stability) for the overall US economy be tracked on a daily basis? I wonder if a wholesale facebook outage would show up on that graph as a brief blip in the positive direction.

bhartzer 5y ago

Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only. It's so important to diversify, such as building a website.

tinza123 5y ago
Well assuming they do that, likely they will host websites on AWS or Azure. Then AWS is down, what are you gonna do?
- K5EiS 5y ago
  There's always Oracle Cloud ;)
- jressey 5y ago
  Move your resources to the one you didn't pick and point your domain registrar there.
deleted 5y ago
[deleted]

nicbou 5y ago

Alle Störungen shows a massive spike in problems for every service it keeps track of: https://allestörungen.de/ https://xn--allestrungen-9ib.de/

TravisHusky 5y ago
Interesting; I have been noticing a lot of service are unstable today. I wonder if there is a larger outage.

pwenzel 5y ago

I noticed that some websites are loading slowly due to the third party script https://connect.facebook.net/en_US/fbevents.js https://connect.facebook.net/en_US/fbevents.js timing out. When uBlock Origin is running, this script gets blocked and pages return to feeling snappy.

monkaiju 5y ago

Gotta post this every time theres a big DNS issue, which seems daily now. Check out Dug! Its a global DNS propagation/monitoring toolon the CLI: https://github.com/unfrl/dug/ https://github.com/unfrl/dug/

LuisMondragon 5y ago

Some Oculus Quest owners can't use their device https://www.reddit.com/r/OculusQuest/comments/q18xwy/facebookoculus_outage/ https://www.reddit.com/r/OculusQuest/comments/q18xwy/faceboo...

russellbeattie 5y ago

It's not hyperbole to say that this is going to literally save lives. Cutting off Facebook's firehouse of hate and misinformation for just a couple hours is going to have a obvious positive effect on millions of people. At this scale, at least one person will get vaccinated today because they didn't see the wall of ignorance that is FB's news feed. Maybe we should introduce "digital blue laws", where one day a week, social media is shut down for the overall good of society.

xmpir 5y ago

Just thinking about all the conspiracy theories you could make of this. Yesterday pandora papers, today the internet stops working.

elzbardico 5y ago

Rejoice! The revolution has started! Yes, I know humor is not welcome in HN

matsemann 5y ago

What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days. Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.

74d-fe6-2c6 5y ago

I'm pretty sure this has been building up since the morning (Germany). I've had odd connectivity problems to a number of sites including slack for a moment.

joelbondurant 5y ago

LOL! They should have used the USA Fact-Check Algorithm from the Science Ministry.

TrackerFF 5y ago

Anyone wanna estimate the cost of total downtime for facebook and instagram, as far as lost ad revenue goes - per minute?

devenvdev 5y ago
Looks like someone built a counter: https://facebookadloss.facebookadloss.repl.co/ https://facebookadloss.facebookadloss.repl.co/ Edit: the counter just jumped from 10B to 60M, so I doubt it's any reliable :)

EastOfTruth 5y ago

We can only hope that they will be gone forever... and HN is having major issues at the same time!

nurhdmsx 5y ago

Its always a DNS problem

Animats 5y ago

New York Times has coverage.[1] "A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo." [1] https://archive.is/iBzs3 https://archive.is/iBzs3

peanut_worm 5y ago

I am sure it is just a symptom of the Facebook outage, but it seems like every website I am going on is slower than usual today.

Redoubts 5y ago
I've seen a couple fail to log in, because their SSO is broken through this. (even if FB login is merely an option)
donatj 5y ago
Every Facebook App on every phone is DDoS-ing the DNS system https://twitter.com/blazejkrajnak/status/1445063232486531099 https://twitter.com/blazejkrajnak/status/1445063232486531099

pmlittle 5y ago

This is all left brain implimentation with looping and classic complexity coming home to roost. As we move through time, we build off of solutions of the past which are solving a problem, but complexity keeps adding on and this is a classic programming/computer science delemma.

qualudeheart 5y ago

If it bleeds we can kill it!

waltbosz 5y ago

I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook. For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.

Bwild 5y ago

They must have shut it off and turned it back on

rvnx 5y ago

Fixed

deleted 5y ago

[deleted]

MrPatan 5y ago

Oh no

oldabc 5y ago

https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/

rkagerer 5y ago

The internet felt a little more fresh and clean today.

husamia 5y ago

do we know what caused the DNS outage at FB?

deleted 5y ago

[deleted]

cheese_van 5y ago

Sir, I don't care who you are, you must open a ticket.

durnygbur 5y ago
Rajeesh FFS, get off HN! We have the world on fire!

cblconfederate 5y ago

It's like a nuclear bomb exploded on the internet.

nabeards 5y ago

Was receiving an error page, now just a server not responding error.

kawsper 5y ago
I can't even resolve Facebook.com
- daitangio 5y ago
  Mobile app hangs too (I am from Italy btw)

TremendousJudge 5y ago

Whataspp too. This seems pretty big

neb_b 5y ago

Also instagram and messenger

zip1234 5y ago

DNS?

SSLy 5y ago
No, I'm getting their error page, so Load Balancers or whatever is behind. EDIT: Or at least not /just/ DNS.
- rootinier 5y ago
  Definitely DNS. facebook.com might be in your local dns cache.
- andyjohnson0 5y ago
  I'm getting an error page with a dead image link and a 2020 copyright date (uk)
deleted 5y ago
[deleted]

RedShift1 5y ago

Nothing of value was lost

ozfive 5y ago
This cannot be said enough.
- blowski 5y ago
  On the contrary, it's said far too much. Facebook is extremely valuable for a lot of people. I dislike Facebook as much as most people on here, but saying "it's totally pointless" is silly and it's not surprising that those who say it are ignored by those who use Facebook.
  - rawoke083600 5y ago
    Never underestimate software that is 'just good enough'
  - belter 5y ago
    In what ways is Facebook "extremely valuable for a lot of people"?
    - can16358p 5y ago
      Communication for many out there. Many will be just fine without commenting on cat photos or bragging with their likes or followers. Many will be in trouble if they use WhatsApp/Instagram/Messenger/Marketplace to do business and any important communication.
    - blowski 5y ago
      * A friend of mine runs a posh burger van that moves around a lot, and he puts "today's location" on Facebook. * My wife talks with her family in Brazil through Facebook, sharing photos * My Church receives a lot of help requests from people in trouble through Facebook * Some abuse charities talk give support to victims through Facebook etc You could argue that it would be nice if there were alternatives, or that these organisations shouldn't be using Facebook at all. Sign me up for your campaign, I agree with you. But if you say "Facebook has no value" then you will never understand the value proposition you need to offer in order to kill Facebook.

lewich 5y ago

Hopefully permanently.

aaomidi 5y ago
Yeah let's keep it that way.

rootinier 5y ago

nslookup www.facebook.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53 * server can't find www.facebook.com: SERVFAIL

aduitsis 5y ago
dig +trace messenger.com shows that all is well with the root DNS servers and dig @a.ns.facebook.com messenger.com ;; connection timed out; no servers could be reached and also ping a.ns.facebook.com 3 packets transmitted, 0 packets received, 100.0% packet loss shows that something's wrong with facebook.

tomerbd 5y ago

No like for you

aritraghosh007 5y ago

The down page shows copyright from 2020 smh

new-day-rising 5y ago

Thoughts and prayers...

husainhz7 5y ago

Whatsapp's down too. Tough month for FB, especially with the leak.

decrypt 5y ago
What leak are you referring to?
- Wistar 5y ago
  https://www.cbsnews.com/news/facebook-whistleblower-frances-haugen-misinformation-public-60-minutes-2021-10-03/ https://www.cbsnews.com/news/facebook-whistleblower-frances-...
- samwilliams 5y ago
  Whistleblower that spoke to WSJ.
- nerbert 5y ago
  https://www.nytimes.com/2021/10/03/technology/whistle-blower-facebook-frances-haugen.html https://www.nytimes.com/2021/10/03/technology/whistle-blower...
  - dmix 5y ago
    The actual leak was published on WSJ “The Facebook Files” https://www.wsj.com/articles/the-facebook-files-11631713039 https://www.wsj.com/articles/the-facebook-files-11631713039
- orangepurple 5y ago
  Facebook Whistleblower Claims Profit Was Prioritized Over Clamping Down on Hate Speech A Facebook whistleblower, who is due to testify before Congress on Tuesday, has accused the Big Tech company of repeatedly putting profit before doing “what was good for the public,” including clamping down on hate speech. Frances Haugen, who told CBS’s “60 Minutes” program that she was recruited by Facebook as a product manager on the civic misinformation team in 2019, said she and her attorneys have filed at least eight complaints with the U.S. Securities and Exchange Commission. During her appearance on the television program on Sunday, Haugen revealed that she was the whistleblower who provided the internal documents for a Sept. 14 exposé by The Wall Street Journal that claims Instagram has a “toxic” impact on the self-esteem of young girls. That investigation claimed that the social media giant knows about the issue but “made minimal efforts to address these issues and plays them down in public.” “The thing I saw at Facebook over and over again was there were conflicts of interest between what was good for the public and what was good for Facebook. And Facebook, over and over again, chose to optimize for its own interests, like making more money,” said Haugen. She explained that Facebook did so by “picking out” content that “gets engagement or reaction,” even it that content is hateful, divisive, or polarizing, because “it’s easier to inspire people to anger than it is to other emotions.” “Facebook has realized that if they change the algorithm to be safer, people will spend less time on the site, they’ll click on less ads, they’ll make less money,” she claimed. Haugen is expected to to testify at a Senate hearing on Oct. 5 titled “Protecting Kids Online,” about Facebook’s knowledge regarding the photo sharing app’s allegedly harmful effects on children. During her appearance on the television program, Haugen also accused Facebook of lying to the public about the progress it made to rein in hate speech on the social media platform. She further accused the company of fueling division and violence in the United States and worldwide. “When we live in an information environment that is full of angry, hateful, polarizing content it erodes our civic trust, it erodes our faith in each other, it erodes our ability to want to care for each other. The version of Facebook that exists today is tearing our societies apart and causing ethnic violence around the world,” she said. She added that Facebook was used to help organize the breach of the U.S. Capitol building on Jan. 6, after the company switched off its safety systems following the U.S. presidential elections. While she believed no one at Facebook was “malevolent,” she said the company had misaligned incentives. “Facebook makes more money when you consume more content,” she said. “People enjoy engaging with things that elicit an emotional reaction. And the more anger that they get exposed to, the more they interact and the more they consume.” Shortly after the televised interview, Facebook spokesperson Lena Pietsch released a statement pushing back against Haugen’s claims. “We continue to make significant improvements to tackle the spread of misinformation and harmful content,” said Pietsch. “To suggest we encourage bad content and do nothing is just not true.” Separately, Facebook Vice President of global affairs Nick Clegg told CNN before the interview aired that it was “ludicrous” to assert social media was to blame for the the events that unfolded on Jan. 6. The Epoch Times has reached out to Facebook for additional comment. https://www.theepochtimes.com/facebook-whistleblower-claims-profit-was-prioritized-over-clamping-down-on-hate-speech_4030295.html https://www.theepochtimes.com/facebook-whistleblower-claims-...

m0guz 5y ago

Ironic enough; status.fb.com also down.

ricardo81 5y ago
It uses the same nameservers as facebook.com, same point of failure. Seems like another poster posted finer details regarding BGP/peering which is ultimately causing the issue.
devenvdev 5y ago
Reminds me of S3 outage a couple of years ago when AWS status page went down because it was relying on... S3...
zoobab 5y ago
LOL
- rd_police 5y ago
  very based response, sir
awinter-py 5y ago
need a status.status.fb.com to indicate the status of the status and / or an S3 bucket with a json blob the apps can pull to at least tell users 'here's what's up'

OnceUponADevops 5y ago

Confirming that we're seeing a major outage with all of our integrations with FB products.

boramalper 5y ago

Duplicates: https://news.ycombinator.com/item?id=28748233 https://news.ycombinator.com/item?id=28748233 https://news.ycombinator.com/item?id=28748199 https://news.ycombinator.com/item?id=28748199 https://news.ycombinator.com/item?id=28748159 https://news.ycombinator.com/item?id=28748159 https://news.ycombinator.com/item?id=28748246 https://news.ycombinator.com/item?id=28748246 https://news.ycombinator.com/item?id=28748131 https://news.ycombinator.com/item?id=28748131

akshayrajp 5y ago

As are Instagram and WhatsApp

deleted 5y ago

[deleted]

attende_domine 5y ago

% ping whatsapp.com ping: whatsapp.com: Name or service not known % ping web.whatsapp.com ping: web.whatsapp.com: Name or service not known % ping facebook.com ping: facebook.com: Name or service not known % ping instagram.com PING instagram.com (31.13.65.174) 56(84) bytes of data. 64 bytes from 31.13.65.174 (31.13.65.174): icmp_seq=1 ttl=53 time=110 ms

underscore_ku 5y ago

good

mef 5y ago

Looks like the routes to their hosted nameservers are down, e.g. A.NS.FACEBOOK.COM

glanzwulf 5y ago

Oh no... Anyway...

TremendousJudge 5y ago
WhatsApp is pretty important infrastructure for most of the world
- jszymborski 5y ago
  Which is regrettable when secure alternatives exist like Signal and Matrix whose business model doesn't involve selling your data.
  - TremendousJudge 5y ago
    Yeah I'm not saying I like it
- pc86 5y ago
  For some egregiously loose definitions of "infrastructure," maybe.
  - TremendousJudge 5y ago
    The same definition that includes phone lines also includes the messaging service everybody uses
  - drcongo 5y ago
    And "important"
  - nicoburns 5y ago
    In parts of South America it's used for all sorts of things. Want to know when your bus is arriving? The bus company likely only knows because the driver is WhatsApp'ing them status updates.

rocho 5y ago

For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL: $ dig facebook.com ; <<>> DiG 9.16.21 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23982 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;facebook.com. IN A ;; Query time: 16 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Oct 04 17:53:00 CEST 2021 ;; MSG SIZE rcvd: 41

john37386 5y ago
Even the Name servers are not returning any values. That's bad. dig @8.8.8.8 +short facebook.com NS These are usually anycasted, meaning that 1 ip return in NS are in fact several servers spread in several regions. They are distributed to closer match through agreements with ISP with the BGP protocol. Very interesting, because it seems that it took 1 DNS entry misconfiguration to withdraw M$ worth of devices from the internet.
- variant 5y ago
  BGP goof? https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445056923309649926?s...
simlevesque 5y ago
Maybe they tried everything else before that. At first it was working but they couldn't serve responses: https://i.imgur.com/UaCtOiX.png https://i.imgur.com/UaCtOiX.png Notice the "2020"
- rvnx 5y ago
  The servers struggle to reply a basic 5xx answer. Two possibilities: - the DNS services internally have issues (most likely, as this could explain the snowball effect) - it could be also a core storage issue and all their VMs are relying on it and so they don't want to block third-party websites and think it will last for a long time, so they prefer to answer nothing for now in the DNS (so it will fail instantly to the client, and drain the application/database servers so they can reboot with less load)

natas 5y ago

good ridance

rainboiboi 5y ago

Just wondering - would the engineer who made the mistake be fired?

tomelliott 5y ago
Nope: https://www.usenix.org/conference/lisa19/presentation/turner https://www.usenix.org/conference/lisa19/presentation/turner
yodon 5y ago
If a single person can cause the failure during the course of their normal tasks, it's not the fault of that person it's the fault of designers of the systems and processes used by that person.
aaomidi 5y ago
If they are then Facebook is worse than I thought.
baby 5y ago
That’s not the culture at facebook
- babuskov 5y ago
  "Move fast and break things". Yeah, it's exactly the opposite. The person should be promoted ;)
deleted 5y ago
[deleted]
pc86 5y ago
What makes you think it was a mistake? What makes you think an engineer did it? Sometimes things just break and take time to fix.
jaywalk 5y ago
How could anyone answer that question? We don't even know that an engineer made a mistake in the first place, much less what the mistake was and what led up to it.

deleted 5y ago

[deleted]

cryptodan 5y ago

Hope it's permanent

deleted 5y ago

[deleted]

kingkool68 5y ago

https://www.whatsmydns.net/#A/facebook.com https://www.whatsmydns.net/#A/facebook.com

_joel 5y ago

Everything is a f*king Facebook problem

poorjohnmacafee 5y ago

Teen depression and suicide rates plummeting right now

drcongo 5y ago

Let's hope it's permanent.

mrstumpy 5y ago
You win for best comment
jose-cl 5y ago
here's your upvote master
hkai 5y ago
Why?
- elevenoh 5y ago
  b/c most humans are on the wrong end of fb's covert, exploitative attention-manipulation
- dessant 5y ago
  Because combined with the abysmal state of education in most places, and a general lack of government action, Facebook is an actual threat to our civilization.
  - forgetfulness 5y ago
    People unfortunately love the upsides of misinformation, or perhaps it's the format that makes it easy to build community around shared (misinformed) values, to rally in battles that rage for hours or days for a cause you deeply believe in and can follow by digesting 30-second soundbites on social midea and 30-minute videos on YouTube. People will do this wherever they can talk in a group online, not just Facebook properties. It's... pretty bad actually, I think the only tool that exists right now is censorship, because the bullshit gets created, spread, and wholeheartedly received way faster than debunking will. And censorship is a power that can't be safely entrusted to nobody.
  - hkai 5y ago

v4ult 5y ago

Signal FTW

WallyFunk 5y ago

This would be a golden opportunity to launch your 'Facebook Killer' app. Preferably a social network where people don't pay with their data, but with, you know, a thing called Money.

smt88 5y ago
Who would pay money to be the first user of a new social network?

Paianni 5y ago

The level of intelligence of the comments in this thread kinda confirm my suspicions that the armchair experts from Reddit et al (including myself) are discovering this site.

faramarz 5y ago

What are some of the possible scnarios beyond the DNS issue suggested? (and might it be an attack?)

coolspot 5y ago
A BGP routing mistake that can cascade into a hard-to-recover-from state of the network where inter-dependencies lock each other.
heegemcgee 5y ago
This is what i came to comments for. Unfortunately, we have literally dozens of comments that amount to nothing more than schadenfreude, and another handful of non-FANG employees speculating how one of the largest internet operations in existence could improve their game (lol)
cwkoss 5y ago
I doubt this is the case, but someone on twitter was speculating "what if this is fb's infra team going on strike"

randomperson_24 5y ago

World productivity just grew by 10%

frederikvs 5y ago
or it went down by another 20%, everybody at first thinking there's something wrong with their internet connection.

mnd999 5y ago

Somebody moved fast.

drummer 5y ago
And broke everything

walrus01 5y ago

downdetector looks like a real mess for it. I'm going to parrot the other comment here and say nothing of value was lost. https://downdetector.com/status/facebook/ https://downdetector.com/status/facebook/

baby 5y ago

Wow. I can’t remember the last time whatsapp was done. I pretty much use messenger/instagram/whatsapp to talk to most of my friends and family. I’m happy that I do use other platforms otherwise I would be completely cut off from my parents right now.

fartingflamingo 5y ago

The whistleblower said she wanted to fix Facebook. Mission accomplished, I'd say. For now at least.

deleted 5y ago

[deleted]

emptysea 5y ago

Interesting, even some open source sites like: https://fbinfer.com https://fbinfer.com are down but https://glean.software https://glean.software and https://reactjs.org https://reactjs.org aren't

jaywalk 5y ago
Anything hosted on Facebook's infrastructure is down. The two sites that you note are up aren't hosted by Facebook.
- Graffur 5y ago
  What's react hosted on?
  - jaywalk 5y ago
    Looks like Cloudflare nameservers and Vercel hosting.
mcintyre1994 5y ago
According to https://lookup.icann.org/lookup https://lookup.icann.org/lookup both glean and reactjs have Cloudflare nameservers. fbinfer has ns.facebook.com nameservers which are presumably down.

deleted 5y ago

[deleted]

yawnxyz 5y ago

Funny enough, I went to https://www.isitdownrightnow.com/ https://www.isitdownrightnow.com/ to check if Facebook is down, and isitdownrightnow is down itself... probably from the massive number requests coming to check if Facebook is down

homeskool 5y ago
yep down for me too
cecilpl2 5y ago
https://downforeveryoneorjustme.com/facebook https://downforeveryoneorjustme.com/facebook
- aaronharnly 5y ago
  Amusingly, that returns: > Is Facebook down right now? > Uh oh! Something went wrong on our side. It's not you, it's us. Feel free to contact us if this persists.
skizm 5y ago
https://downdetector.com/ https://downdetector.com/ seems to be working for me at least.
- horsellama 5y ago
  'Unusual traffic patterns detected' now
- spiantino 5y ago
  It's amusing that the top 3 trending reports are the FB sites that are down, and then the mobile carriers themselves, presumably because when FB doesn't load they assume it's their mobile network's fault. People really do think FB is the internet
  - Sebb767 5y ago

j3th9n 5y ago

I like it, it feels like it's 1999.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

htrp 5y ago

Facebook is proving that it's systemically important by taking the entire site down..... Zuckerberg is taking his ball and going home unless you stop writing mean things about him /s

spaceywilly 5y ago
Let’s hope it’s this. Everyone will just shrug and move onto the next hopefully less evil site
- hkai 5y ago
  Which one?
  - julianlam 5y ago
    lobste.rs? mastodon?
oneeyedpigeon 5y ago
Used to have a manager that we swore did exactly that. Every time he was away on holiday, mysterious site problems to prove his worth!
- gagege 5y ago
  I had the exact opposite and it was hilarious. Every time my manager (a great guy and really good at what he did) was away for a week the sprint would go very smoothly.
jimkleiber 5y ago
Lol exactly what I was thinking. I'm trying to keep my tinfoil hat in the closet and yet it seems odd that after there is a huge FB whistleblower story on 60 Minutes last night, all of FB goes down today. I really hope it's just some internal technical error and not a "see, despite the bad things of FB, you really need us" move. It's probably trivial, the timing just seems weird to me.

totaldude87 5y ago

wonder how much of internet traffic as a whole is down now..

dkarp 5y ago

Someone save a risky release for 9am on a Monday morning? Decided Friday afternoon was too risky?

grayhatter 5y ago
0830 actually :/ But to be fair... seems like it was a good call to not do it Friday night :D
- dkarp 5y ago
  If they chose 8:30, then it must have been really risky! ;)
  - euroderf 5y ago
    The Honolulu office is getting ready for a long night :)

XiS 5y ago

https://www.cyberciti.biz/humour/a-haiku-about-dns/ https://www.cyberciti.biz/humour/a-haiku-about-dns/

rootusrootus 5y ago

And a lot of people seem to be coming to HN to find out why, judging by how laggy HN is getting right now...

moffkalast 5y ago
Well it sure is the place to find out for sure.

tinyprojects 5y ago

Oculus is also down

ourcat 5y ago
Indeed. People seem to forget that when Facebook goes down, it's not just your feed of depressing posts, photos and messages that go away, but also the entire Oculus VR platform, since they demanded a FB account to use Quest headsets.

orthecreedence 5y ago

Good riddance.

tannhaeuser 5y ago

Expecting to get messages on WhatsApp alternatives tonight ...

johnbaker92 5y ago

Let this be permanent - not a huge loss for humanity.

thedudeabides5 5y ago

Checked isitdownrightnow.com and said Netflix was also down. Any chance these are related?

robjan 5y ago
Netflix is up

deleted 5y ago

[deleted]

CodeGlitch 5y ago

Looks like HN is being hit pretty hard right now?

yawnxyz 5y ago

I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)? How could they all go down at the same time, if they have different teams of engineers running each product separately? Could anyone with some background (or person familiar with the matter) explain how their system's set up?

vodkapump 5y ago
Seems unrelated to their infrastructure, the DNS records for facebook.com, instagram.com, whatsapp.com and all derivative domains are wiped clean it seems edit: though saying that, they do run their own registrar... Might've fucked something up over there.
toast0 5y ago
WhatsApp and Instagram are both in FB infra. As I understand it, Instagram is fairly integrated with FB services; when I left in 2019, WhatsApp was less so, it was mostly WhatsApp specific containers running with FB's container orchestration on FB machines dedicated to WhatsApp (there was and probably is some dependence on FB systems for some parts of the app, for example the server side of multimedia is mostly a FB system with some small tweaks and specific settings, but chat should be relatively isolated). Inbound connection loadbalancing is shared though. FWIW, WhatsApp (on phones) should be resiliant to a DNS only outage, the clients contain fallback IPs to use when DNS doesn't work, and internal services don't use DNS as far as I remember. At one time, WhatsApp had actually separate infrastructure at SoftLayer (IBM Cloud now), but that hasn't been in place for quite some time now. When I left, it was mostly just HAProxy to catch older clients with SoftLayer IPs as their DNS fallback.

elboru 5y ago

Is it just me or HN also feels kinda laggy?

amelius 5y ago
Yes, it's slow here as well, and posting this comment failed the first and second time.
amelius 5y ago
Yes, it's slow here as well, and posting this comment failed the first and second and third time.
amelius 5y ago
Yes, it's slow here as well, and posting this comment failed the first and second and third and fourth time.
- bradenb 5y ago
  This is either a hilarious accident or genius comedy.
  - tzs 5y ago
    This is not too rare when HN is being slow and giving those "We're having some trouble serving your request. Sorry!" pages. If you get one of those on your comment submission you have no way to know if the trouble stopped it from accepting the comment or if it accepted the comment and ran into trouble then trying to display the updated thread. For some reason I can't even begin to guess at HN does not seem to have protection against multiple submissions of the same form, so if after getting "We're having some trouble serving your request. Sorry!" on your comment submission you hit refresh again to display the page and the form gets resubmitted, you get a duplicate comment.
    - saagarjha 5y ago
      Earlier today when I was getting these I went to go check the page to see if the comment was posted there. More than once it said it failed but I was able to stop myself from trying again because it was actually there.

klik99 5y ago

Is this related to the outages from lets encrypt root cert expiring? Probably not since this looks like a DNS issue, but still it's a crazy coincidence that two major internet breaking events happen in the same week

treesknees 5y ago
There is zero reason to believe it's related at all. It's perfectly reasonable to have multiple large and unrelated failures in the same week. I also wouldn't classify the loss of 1 company, and the expiration of some TLS certificates, as the interconnected network of networks being broken. The Internet has continued to function even if some larger players were unreachable or having issues.

deleted 5y ago

[deleted]

smashah 5y ago

Good, now I can go for lunch.

AzzieElbab 5y ago

it is probably unrelated but HN is crawling

pixelgeek 5y ago

I would like that say that after my "burn it down" comments on another Facebook related post that I had nothing to do with this.

jamespwilliams 5y ago

Is this caused by missing glue records? I can’t resolve any of FB’s nameservers. Anyone know how that could happen?

wut42 5y ago
the glues are still there-- it's not a DNS issue but a network one. Their ASN has mostly been withdrawn from everywhere.
zekica 5y ago
The glue records are fine from my end: dig -t NS facebook.com @a.gtld-servers.net

carrja99 5y ago

Good.

keithnoizu 5y ago

somewhere an engineer is begging a mnesia instance to come back online.

html5web 5y ago

Great time to take a break from Facebook and Instagram. Use Telegram instead of WhatsApp

hotz 5y ago

The joy that people are getting from this is quite shitty. I hate social media but there are people earning a living working for these companies. Like others have pointed out, businesses and neighborhood watches rely on tech like this. At some point we've all had sites/apps go down, in a situation like that the last thing you want is people enjoying it. The lack of empathy in this thread is telling.

sasaf5 5y ago
It pales in comparison to the lack of empathy facebook has shown to its user herd.

yholio 5y ago

I had problems with my internet connection and loaded my ISPs site. Strangely, my bill was paid. Even stranger, some sites load while others do not. Then it hit me: I am so dependent on Facebook owned properties (Whatsapp, Facebook, insta) that a Facebook failure looks to me like an internet failure.

rvz 5y ago

From [0] > ...there is no limit to the scandals, leaks, whistleblowers, lawsuits or penalties that will bring the Facebook mafia down. Fine. 'Literally' bringing the Facebook mafia down like that would do. But only for now. [0] https://news.ycombinator.com/item?id=28742179 https://news.ycombinator.com/item?id=28742179

wejick 5y ago

Would be very interesting if they release the RCA to the public

keithnoizu 5y ago

some poor engineer is sobbing over a split brain mnesia cluster right now praying to get the thing back up.

throwawaylolx 5y ago

When I noticed HN was loading slowly, I already knew FB was down.

rcarmo 5y ago

Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.

wyager 5y ago
Can you not use an Oculus headset if FB servers are down? That’s absurd.
- rcarmo 5y ago
  Some preloaded apps work (like YouTube, Firefox), but stuff like settings, the lobby, etc., are very slow or display "Unable to Load" messages. Any game that relies on your friends list seems to freeze for a while, then try to carry on.
- Reubachi 5y ago
  Yep. We've come full circle, where techies are rediscovering the original hatred for the Oculus, that it is tied to a social media walled garden, for some reason.
  - boston_clone 5y ago
    I think you may need a refresher on their history - https://en.wikipedia.org/wiki/Oculus_(brand)#History https://en.wikipedia.org/wiki/Oculus_(brand)#History When FB announced they would be buying Oculus, they promised that no social media integration would be required. FB breaking that promise is not the same as Oculus having that requirement from the get-go. What original hatred are you talking about??
    - Reubachi 5y ago
      Originally, you did not need a facebook account to use oculus after purchase. They framed this as "you do not need to integrate social media/facebook". This ^ means fuck-all, because at that time (day 1), their oculus services where hosted in the same infrastructure as their social media services. Last year, they got rid of "you do not need a facebook account". But in all situations since inception, all of your data is passing through the same infrastructure as facebook data. It may not be being exposed, or targeted for advertising, but this WAS a huge point of contention years back.

gprasanth 5y ago

Suspecting it might be related to the recent letsencrypt cert authority expiring? Was just debugging an issue earlier today and just couldn't help wondering how much of the internet is secured by letsencrypt. All of the static hosts providing free SSL: vercel, netlify, render, firebase hosting, github pages, heroku etc. ... It does work on modern browsers and devices but goes terribly broken on a lot of old devices.

jaywalk 5y ago
Obviously not possible to check right now to provide proof, but I feel quite confident in saying that Facebook does not use Let's Encrypt. It's also clearly not an SSL issue.
- gprasanth 5y ago
  You're right. Fb doesn't seem to be using letsencrypt. https://crt.sh/?q=facebook.com https://crt.sh/?q=facebook.com On a side note, the amount of phishing sites using letsencrypt and having a domain similar to facebook.com is quite appalling.
- deleted 5y ago
  [deleted]

sAbakumoff 5y ago

rich people serve their revenge for pandora papers

foobaw 5y ago

This is taking a longer time than expected for a company like Facebook - must be serious where a rollback isn't possible or trivial.

chasd00 5y ago
from what i understand (take with grain of salt) remote access to the routers affected is down. So they need to be physically plugged in to address the issue. hence some of the other "scrambling private jets" comments referring to getting the right people physically plugged in to the right routers.
derwiki 5y ago
I wonder if everyone refreshing the sites/apps trying to get it to load is contributing to the problem
- XCSme 5y ago
  Probably not, from other comments it looks like there was a wrong configuration rolled out, and now they are logistically struggling to get access to fix them.

em3rgent0rdr 5y ago

How much revenue does facebook loose per hour down?

dmoy 5y ago
This is very hard to get exactly right, because traffic isn't constant at all times, and you don't know if people won't just make up for lost time using facebook at another time of the day, etc. So you can't really know. But, a good rule of thumb right now is about $10,000,000 per hour.
missedthecue 5y ago
Facebook made $29 billion last quarter which translates to $315,217,391 per day. Divide that over 24 hours in a day, and it's ~$13 million per hour. Of course, depends on the hour of day. Facebook likely makes more ad money when North America is awake than when Asia is awake for instance.

throwaway78981 5y ago

Signal is welcoming everyone: https://nitter.mailstation.de/signalapp/status/1445062426739855366 https://nitter.mailstation.de/signalapp/status/1445062426739...

maxxxxxx 5y ago
Ironically, I cannot send messages on Signal right now. They can’t handle the extra load?
- coolspot 5y ago
  Signal replicates each message to NSA and FB, so when one is down, Signal’s backend fails with a timeout error.
  - lavp 5y ago
    Source?
kitkat_new 5y ago
to also face a failure in the single point of failure - system?
MajorSauce 5y ago
Still an American centralized platform. Federated Matrix is the way to go!

unusximmortalis 5y ago

for me even this site loads very very slowly. pinging google name server is fast as usual. it could be a more wide problem not just FB related.

wly_cdgr 5y ago

...Including that one

caturopath 5y ago

What would a full day of WhatsApp outage mean for the world?

TedShiller 5y ago

This solves the disinformation problem

deleted 5y ago

[deleted]

amir-h 5y ago

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?

o10449366 5y ago
Either many HN users are in glee over FB's potential demise ...or many HN users are also avid FB users (and now have to resort to backup sources of entertainment)
uyt 5y ago
If I wanted to know if a site is down for everyone or just me, I would check twitter/hn first before checking the down detector sites

kossTKR 5y ago

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: >As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures. User is providing live updates of the incident here: https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfd4dyv/?context=3 https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...

cotillion 5y ago
So, does anyone know where to one can buy an LTE gateway with a serial port interface? Asking for a friend.
- mekatter 5y ago
  These are readily available, OpenGear and others have offered them forever. I can't believe fb doesn't have out of band access to their core networking in some fashion. OOB access to core networking is like insurance, rarely appreciated until the house is on fire.
  - Sebb767 5y ago
    It's quite possible that they have those, but that the credentials are stored in a tool hosted in that datacenter or that the DNS entries are managed by the DNS servers that are down right now.
    - mekatter 5y ago
      You are probably right but if that is the case, it isn't really out of band and needs another look. I use OpenGear devices with cellular to access our core networking to multiple locations and we treat them as basically an entirely independent deployment, as if it is another company. DNS and credentials are stored in alternate systems that can be accessed regardless of the primary systems. I'm sure the logistics of this become far more complicated as the organization scales but IMHO it is something that shouldn't be overlooked, exactly for outlier events like this. It pays dividends the first time it is really needed. If the accounts of ramenporn are correct, it would be paying very well right now. Out of band access is a far more complicated version of not hosting your own status page, which they don't seem to get right either.

calebm 5y ago

"When you are strong, appear weak."

deleted 5y ago

[deleted]

amir-h 5y ago

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB? [I'm also getting server error trying to submit this comment]

jangrahul 5y ago

makes me wonder, why dont porn sites ever seem to go down ?

mrkickling 5y ago

When I do dig instagram.com I get an A response for this IP: 31.13.65.174 or similar addresses, which leads to an empty page.

shkkmo 5y ago

I keep getting non-dns errors from Hackner News as well. There appears to be some sort of broader incident happening? It's not just lag, I keep getting the "We're having some trouble serving your request. Sorry!" page. Edit: HN related thread https://news.ycombinator.com/item?id=28749476 https://news.ycombinator.com/item?id=28749476

deleted 5y ago

[deleted]

EB66 5y ago

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached [root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: Name: downforeveryoneorjustme.com Address: 172.67.166.187 Name: downforeveryoneorjustme.com Address: 104.21.91.48 [root@app ~]# Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.

uo21tp5hoyg 5y ago
Yeah Cloudflare have said they're being flooded with Facebook DNS retries and had to get extra hands on deck to deal with the influx.
rstupek 5y ago
Yes same issue for me with 8.8.8.8, errors for everything but big domains.
pixxel 5y ago
No idea if it’s related but a lot of Tor websites have also been offline all day (BBC, ProtonMail etc).
r721 5y ago
See this thread (with replies): >Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com http://facebook.com https://twitter.com/jgrahamc/status/1445066136547217413 https://twitter.com/jgrahamc/status/1445066136547217413 >Our small non profit also sees a huge spike in DNS traffic. It’s really insane. https://twitter.com/awlnx/status/1445072441886265355 https://twitter.com/awlnx/status/1445072441886265355 >This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled. https://twitter.com/TheodoreBaschak/status/1445073229970763781 https://twitter.com/TheodoreBaschak/status/14450732299707637...

zrail 5y ago

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook. I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

poetaster 5y ago
I used to do this properly. One vanity got the better of me. Got some work to do. TGF SQL.
chasd00 5y ago
There will be so many meetings over this. If powerpoint was listed on the stock exchange i'd say now's a good time to buy hah.

mdani 5y ago

Is this in some way connected to the Facebook data leak of 1.5 billion users? The timing seems quite odd that both these things happen around the same time.

AtNightWeCode 5y ago

So they managed to remove facebook.com from 1.1.1.1 and 8.8.8.8. That is impressive. Not something anyone can achieve in such short time by even trying.

HellDunkel 5y ago

I wish it would stay that way.

slackfan 5y ago

So how much do we need to pitch in to keep it down?

slackfan 5y ago

So how much cash do we need to pitch in to keep it down?

ProjectArcturis 5y ago

Have they tried turning it off and then turning it back on again?

mindcrime 5y ago

Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don't you get that?

GreenWatermelon 5y ago
At this point in time, I doubt this holds true. Facebook is just too big and pervasive that such an outage would be treated by its users like an internet outage or a power outage. Once it's back online, everyone will forget.
dgb23 5y ago
Is this a quote?
- mindcrime 5y ago
  Yes, it's from "The Social Network". It's a scene where Mark Z. is explaining to Eduardo how important it is that the servers stay up all the time. Of course it was, as far as I know, ficitonalized in the first place, although it rings true (in context) to some extent. What I wonder is, how much is that true now? That is, how much downtime would FB have to experience for enough users to start leaving, to the point that it might prompt a serious exodus.
- qnsi 5y ago
  google suggest its from a social network movie

jbverschoor 5y ago

It's funny that hackernews is now overloaded with distracted people ;-)

deleted 5y ago

[deleted]

jurajmlich 5y ago

DNS servers of a major internet provider in the Czech Republic are down now. Probably not a coincidence (other DNS server's stats show increased traffic so my guess is that Vodafone's DNS servers were unable to cope with the increased traffic and crashed https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099). It's crazy that half the country doesn't have internet because Facebook stopped working.

john37386 5y ago

Imagine if they need access to fb.com email to re-enable the access for the on-site technician.

Ristovski 5y ago

Discord [1] is taking a toll from the increased traffic as well: "We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service. We have temporarily disabled typing notifications. We expect these to be re-enabled soon." [1] https://discordstatus.com/ https://discordstatus.com/

Le_Dook 5y ago
Yeah, it seems like a lot of places online are facing the same. Even here has been bad for me

Animats 5y ago

DNS configuration is becoming a single point of failure. A few weeks ago, many services running out of AWS West 2 failed because the within-the-datacenter DNS system broke down somehow.

jurajmlich 5y ago

It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it (https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099). Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)

blntechie 5y ago
All most people have to do now is install an app and it takes care. But messaging need to go from media and news.
megous 5y ago
If only the news reporting was not as stupid as "internet is not working at UPC", instead of DNS resolvers at UPC crashed, here's what you can do... Anyway, I didn't even notice since I run knot-resolver at home. I wonder what it will be like connecting Facebook back to the internet, thundering herd and everything...
- yourad_io 5y ago
  I suspect that the DNS aspect will be fine. The middle DNS servers only need one valid response to cache it for $TTL, but they can't cache SERVFAIL.
  - megous 5y ago
    I mean connecting your company to the internet when you have billions of devices waiting in the line to fetch updates, or whatever. Will not that be an issue? Re-enabling routing to such a massive internet service...
    - yourad_io 5y ago
      I suspect it will slowly get there. We'll find out, it is recovering now.

Sophira 5y ago

Somebody just had their very own "onosecond". https://www.youtube.com/watch?v=X6NJkWbM1xk https://www.youtube.com/watch?v=X6NJkWbM1xk The video is one that Tom Scott published in June 2020 about the worst typo he ever made in one of his prior jobs, and while the Facebook mistake is almost certainly not going to be anything irrecoverable like this one, you can bet that Facebook pride themselves on being available all the time.

dizzant 5y ago

Reading the thread, I'm surprised at the number of nearly identical "How much do we have to pay to keep it down? xD" posts I'm seeing, often from throwaway accounts. Some accounts with multiple near-identical posts within the same minute. Could this be a coordinated smear in HN comments?

cwkoss 5y ago
I think a lot of people just think facebook is bad for society. I do.

midnightdiesel 5y ago

Hopefully it never comes back up!

_yo2u 5y ago

value of fb directly proportional to traffic flow curious about why fb is down ;)

StapleHorse 5y ago

That explain the new contacts in Telegram.

mcintyre1994 5y ago

I love how understated companies always are about things like this. > Facebook said: "We are aware some people are having trouble accessing our apps and products. We are working to get things back to normal as quickly as possible and apologise for any inconvenience." https://www.bbc.co.uk/news/technology-58793174 https://www.bbc.co.uk/news/technology-58793174

teddyh 5y ago

Here we can see why you should not have all your DNS servers in the same AS (in this case, AS32934).

shockeychap 5y ago

I feel for the sysadmins who are fighting ulcers and migraines at the moment, but I can't shake feeling that the world is just a little bit better for this small window of time.

not2b 5y ago

This is a great argument for the antitrust authorities to break up Facebook. Allowing the big social media companies to buy each other creates a single point of failure. If Instagram and WhatsApp were separate companies, a technical disaster at one would not take out the other two.

chungy 5y ago

Even Facebook's Onion site isn't working: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion/ http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t... Fascinating simply that it's apparently not just a DNS issue.

alexvoda 5y ago
Any idea what is the explanation for this?
- deleted 5y ago
  [deleted]

dclaw 5y ago

Delete Facebook / Instagram / WhatsApp when it comes back up. They are all trash.

ionwake 5y ago

Did I just read that the Facebook IRC fallback went down too?!? I was about to say what’s wrong with freenode ( but yeah on 2nd thoughts let’s not talk about freenode )

taftster 5y ago

What I think is interesting is the effects of this type of thing across peripheral news sites, like HN. I wonder how much spike HN gets with people rushing here to find out what's going on and to read the (articulate) related discussions.

ionwake 5y ago

Why is this not the first post on hn? It’s double the points and has over 590 comments?

imalerba 5y ago

Seems like Telegram went down with a big whatsapp-is-down hug of death.

blntechie 5y ago
Yep, Telegram stopped sending messages a while back and not loading at all for me now.

marstall 5y ago

between outage and whistleblower, this has got to be the worst day in facebook's life

blobbers 5y ago

well... this is unfortunate: ; <<>> DiG 9.10.6 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36072 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

iseanstevens 5y ago

Interesting Timing!

aero-glide2 5y ago

Does this mean Oculus is unusable now?

shmiga 5y ago

A lot of addict's will now feel how is it to live in the reality not scrolling dump images. I hope this will be tradition at least once a month.

areactnativedev 5y ago
And here we are, looking at the xth top-level comment on HN...
qnsi 5y ago
I am an addict. I refresh this thread waiting for information: fb is back online. I need help but this is too hard for me. I uninstall social media once a week but install two days later. I should probably go to therapy with this, but I am not sure I wouldnt be laughed at
- ppqqrr 5y ago
  https://heyfocus.com/ https://heyfocus.com/ worked for me, maybe it’ll help (if you’re on Mac). Addiction to social media is a real problem; thousands of engineers are paid to make sure that these products ensnare your attention. It wouldn’t be odd if it takes a few bucks of your own to rescue yourself. Don’t hesitate to seek help, no one will laugh at you.

teddyh 5y ago

Interestingly, their .onion site¹ is also down. https://en.wikipedia.org/wiki/Facebook_onion_address https://en.wikipedia.org/wiki/Facebook_onion_address

fluxem 5y ago
This is expected. I guess, the internal DNS is down, so the whole infrastructure is broken
coolspot 5y ago
The onion site is just a reverse-proxy to the main web-site. So if the main site is down (due to internal DNS or BGP issues) onion reverse-proxy can’t get to it as well.

EGreg 5y ago

Anyone able to Connect w WhatsApp at the moment?

alkonaut 5y ago

Here is a handy troubleshooting flowchart for megacorp outages: > Is it a DNS issue? -> yes It can be used in reverse as a postmortem too.

nycdatasci 5y ago

Ignore.

wrycoder 5y ago
19 hours ago??

scumcity 5y ago

hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.

geocrasher 5y ago

Once again, it's DNS. https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tune-of-let-it-be https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...

clipradiowallet 5y ago

In other news, a bunch of people got a lot more work done today than normal I suspect...

suyash 5y ago

Facebook hacked?

pat-jay 5y ago

Finally! A small (or not so small) outage for FB, a large benefit for mankind :)

deleted 5y ago

[deleted]

bitzlab 5y ago

This morning around 10:00 AM I've been posting on a forum a message in to which I have said that I'm hoping Facebook will get shut down.. Now, Facebook situation you see it might be my fault. Sorry folks, didn't meant that for real.. I still need FB's products! =)) https://natrmd.com https://natrmd.com

chadlavi 5y ago

I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.

wooger 5y ago
Also amusingly I think quite a lot of employers use WhatsApp for part of their disaster communications plan. If this happened then some wider issue (github down) there’d be chaos.
- phatfish 5y ago
  The UK government will grind to a halt if WhatsApp is still down tomorrow. Well, more than usual that is.

swayson 5y ago

This really is a fascinating case-study of what is truly resilient systems. More often than not, they are not centralized.

throwaway123x2 5y ago

The cynic in me wonders if this is related to the Pandora Papers leak

jose-cl 5y ago

In this context, I remember youtube+pakistan issue[1]. I also wonder how an AS/BGP manager do his/her job... I imagine a guy/girl changing a text file in a old console. Anyone knows? [1] https://www.infoworld.com/article/2648947/youtube-outage-underscores-big-internet-problem.html https://www.infoworld.com/article/2648947/youtube-outage-und...

platz 5y ago

[redacted]

treesknees 5y ago
What evidence is there to suggest this is due to Facebook being down?

fartingflamingo 5y ago

From the archived ramenporn reddit comment thread at [0]: > This must be incredibly stressful so for your sake I hope you sort it out quickly... but for the world's sake, I hope you fail and make the problem worse before jumping ship followed by every other engineer, leaving it to Zuckerberg to fix himself. But I still hope it's not too stressful for you! https://archive.is/Idsdl https://archive.is/Idsdl

blablablub 5y ago

Glad to report that facebook's dns in china is not affected. You can dig facebook.com and the depth of the internet happily reply with a random ip address as usual.

liendolucas 5y ago

Does anyone have a reasonable guess on how much money they have already lost?

gmiller123456 5y ago
Likely $0. Ad views lost now will likely be made up for later. And even if there is a reduction in views, it just makes other views more valuable. Facebook doesn't have real competitors, so the money isn't going anywhere else.

jeffrom 5y ago

Is facebook being down causing hacker news to get the hug of death??

liendolucas 5y ago

Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.

agilob 5y ago
that's not how post-mortems work

6510 5y ago

Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.

DreamScatter 5y ago
Do you mean Jacque Fresco?
- 6510 5y ago
  yes

Animats 5y ago

There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms 5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms 6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms 7 * * * ... So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.

john37386 5y ago
Yeah the patch to fix BGP to reach the DNS is sent by email to @facebook.com. Ooops no DNS to resolve the MX records to send the patch to fix the BGP routers.
- yoelo 5y ago
  Seriously? Is that how it works?
  - john37386 5y ago
    I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
    - boshomi 5y ago
      Sheera Frenkel: Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057

jmfldn 5y ago

Yet another reason to not over-rely on a few big tech companies for the majority of the planet's communication. Forget concerns about competition, monopolies and so on for now (as important as they are), what we want are many social networks, video conferencing apps, messenger apps. Every country should strive to build their own Google or FB, or certainly many more should. State-backed if needed. It's a question of resilience and security as much as anything.

koksik202 5y ago

my home connection with ISP is down Vodafone Ireland, so I guess they have such a big churn in Vodafone from FB BGP routes that it blew Vodafone network. Is it DNS or routing issue?

sabujp 5y ago

India runs on whatsapp. They'll have more backups now.

marchingvehicle 5y ago

Where could the physical data centers be that they need to access? How far away could it be?

chippy 5y ago
maybe, there are reports (i.e. unverified tweets) that employees cannot access sites due to the security systems also being down. I imagine email, and messaging for employees would also be down too. It may be very hard for employees to get to the physical boxes, and/or bypass any physical or software security systems.

MrYellowP 5y ago

I keep trying to submit to HN but I keep getting an error. What's wrong with the internet? FaceBook is down. My friend from Slovenia is having trouble with discord. It eats his messages. I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received. TrackMania players have talked about having input lag. ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this) ycombinator turned out to be giving only errors. Some sites I've found via google results seem to report that they are suffering from slow connections. Do you have anything to add to this?

user3939382 5y ago
Big ISP outages in NYC right now
saltyfamiliar 5y ago
I'm having issues with telegram as well. Images won't send and the app continuously says "updating" on the top status bar. How could facebook dns issues cause this?
- karlzt 5y ago
  Because everyone is using Telegram instead of Whatsapp due to WA being down.

MrYellowP 5y ago

I keep trying to submit to HN but I keep getting an error. What's wrong with the internet? FaceBook is down. My friend from Slovenia is having trouble with discord. It eats his messages. I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received. TrackMania players have talked about having input lag. ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this) ycombinator turned out to be giving only errors, but now seems to be working occasionally. I can not submit anything, though. Some sites I've found via google results seem to report that they are suffering from slow connections. Do you have anything to add to this?

tiluha 5y ago
Some of those can probably be explained by facebooks traffic being redistributed to other services, overloading them
agilob 5y ago
A few wordpress blogs crashed because addon facebook pixel is crashing. Very intensive lesson for the internet!
Animats 5y ago
What's wrong with the internet? With Facebook down, some large DNS servers seem to be struggling with the extra load of failing requests to look up "facebook.com". Cloudflare reports overload with their DNS server at 1.1.1.1, although that's working for me. Billions of things worldwide are trying to connect to Facebook. The lookup which normally returns the IP address for facebook.com on the first try now requires trying a.ns.facebook.com, b.ns.facebook.com, etc. several times each before giving up. Probably several times a minute for everyone who has a Facebook app in their phone turned on. That may be using a big fraction of world DNS resources. Vodaphone Ireland seems to be struggling with a DNS overload right now, per the Irish Independent. Also, their status page can't find "Dublin" as a city.

lazlee 5y ago

Hopefully forever.

pytlicek 5y ago

Who else sees their deleted messages on WhatsApp that shouldn't be there? https://news.ycombinator.com/item?id=28749652 https://news.ycombinator.com/item?id=28749652

freediver 5y ago

Terrible day for many people. Both working for Facebook and those depending on their services.

polynomial 5y ago

"It's always DNS."

asduoihfijnu 5y ago

ok mom i commented

htrp 5y ago

In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.

mkr-hn 5y ago
I'm genuinely not sure if the reports I heard of employees being locked out of the systems they need to fix it because their network is down are jokes or true.

the-dude 5y ago

It is about time to speculate about sabotage, a disgruntled employee or something more exotic. All this BGP talk is boring.

sheepybloke 5y ago
Like this being tweeted: https://twitter.com/YourAnonOne/status/1445082304393719818?s=20 https://twitter.com/YourAnonOne/status/1445082304393719818?s...

_yoqn 5y ago

It's about damn time. Hopefully they stay down. It will do the world some good (long term) to have some time away from this platform and platforms like it.

emmap21 5y ago

Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.

jy3 5y ago
No. All the ones you mentioned are up.
_fizz_buzz_ 5y ago
Youtube and google are definitely working for me without any problems (haven't tested telegram or zoom).

pytlicek 5y ago

Who else sees their deleted messages on WhatsApp that shouldn't be there? https://twitter.com/Pytlicek/status/1445072626729242637 https://twitter.com/Pytlicek/status/1445072626729242637

reilly3000 5y ago
That is wild and definitely newsworthy. Capture as many screenshots and data as you can. FWIW it seems possible that the messages remain cached locally on your device but deleted from their servers, and with their outage they aren’t being updated to delete?
- reilly3000 5y ago
  This is something to get in front of a tech journalist who covers Facebook. It’s a major breach of trust. Probably hit one of your favs with a tweet, but they also tend to list their contact info on author pages of the sites they publish for.

deleted 5y ago

[deleted]

cestith 5y ago

Out of band management is an important feature for the reliability of your network.

todd-davies 5y ago

Perhaps allowing Facebook, WhatsApp and Instagram to merge was efficient after all - now that they have synchronized outages, people finally have a chance to get on with their lives, free of clickbait news and misinformation.

mupuff1234 5y ago

Is there any site that tracks number of users for messaging apps? I'd be really curious to see if signal\telegram\etc are seeing a big bump.

lucidbee 5y ago

The timing of this is so rich in irony I can't help but wonder if there is an element of internal sabotage. How many FB employees hate FB right now? The latest expose of FB is both effective and truly awful. I can't imagine feeling good about a FB job. And it's gotten worse! Now they look like they can't even keep their websites up.

greeklish 5y ago
Can we really ever know? There are million of $ at stake!
ttobbaybbob 5y ago
Perhaps we'll find out. As fun as internal sabotage would be, schadenfreude-wise, i think it much more likely this will turn out to be a time when Hanlon's Razor applies

baalimago 5y ago

Aren't there places around poorer countries where Facebook is basically an ISP? What about them? They have literally 0 info. https://tcrn.ch/3kOHco1 https://tcrn.ch/3kOHco1

LuisMondragon 5y ago

Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057

baalimago 5y ago

Aren't there places around poorer countries where Facebook is basically an ISP? What about them? https://tcrn.ch/3kOHco1 https://tcrn.ch/3kOHco1 2Africa cable, as an example

codediesel 5y ago

still down, people going to signal to have a chat

dghughes 5y ago

Twitter seems to be a big buggy now too maybe just a coincidence. User comments under posts are not appearing.

jontro 5y ago

Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.

emmap21 5y ago

There is a global outage this morning starting from 8:00 AM. This list has Google, Tiktok, Zoom, Slack and of course FB products and services.

decrypt 5y ago

From this tweet: https://twitter.com/BlazejKrajnak/status/1445063232486531099 https://twitter.com/BlazejKrajnak/status/1445063232486531099 "Because of missing DNS records for http://Facebook.com http://Facebook.com, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."

jleyank 5y ago

I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I can never remember...

aenis 5y ago
most of what they do, they do with in house tools, and custom-everything, including hardware. as a consequence, for some classes of problems there are no experts - not at facebook, not anywhere. i feel for their netops people. uncharted territory with the whole world watching and, no doubt, a lot of morons from management trying to be "helpful" in getting this nice crisis resolved. for any crisis there is always a bunch of clowns with MBAs that consider it their golden opportunity to shine (nearly always at someone elses expense)

firstSpeaker 5y ago

Is there any place to see how the overall internet bandwidth usage has changed during this outage?

zanethomas 5y ago

hahaha, good riddance

38932ur98u 5y ago

This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.

jacke 5y ago

They made they own BGP tools and looks like it failed https://www.youtube.com/watch?v=wHfYUbKNEyc https://www.youtube.com/watch?v=wHfYUbKNEyc

typingmonkey 5y ago

If it is an DNS error, why is the .onion site also offline? - https://en.wikipedia.org/wiki/Facebook_onion_address https://en.wikipedia.org/wiki/Facebook_onion_address - facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion

detaro 5y ago
It's not just/primarily a DNS error.
ApolIllo 5y ago
My guess is that the FB backend also required DNS. The .Onion site isn't backed by a backend built on a onion native stack (is that a thing?)
lifthrasiir 5y ago
DNS outage is an outcome of faulty BGP updates. As such not only the Internet can't see the FB network, there is also no connectivity from the FB network to the Internet right now.

littlecranky67 5y ago

The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?

mdavidn 5y ago
Sounds like automation deployed a configuration update to most of Facebook's peering routers simultaneously. Something similar brought down Google in 2019.
- littlecranky67 5y ago
  If so, then it would simply be a BGP issue - no FB servers reachable, as all routes are down. But media+claims a combo of BGP/DNS. Hard to believe world-wide border routers, only responsible for networks containing DNS servers, are misconfigured. I am rely curious about that post-mortem :)
  - mdavidn 5y ago
    I think it was only a BGP issue. The DNS servers apparently shared the same peering routers as the rest of Facebook's infrastructure. Everyone focused on DNS because that's the first sign of failure to an end user.

vvpan 5y ago

Not an indepth technical comment here but: seeing a tech megacorp go offline for a day makes me very very happy.

MrStonedOne 5y ago

Rumormill is suggesting that facebook badge readers are also down causing issues with trying to get to the servers to manually fix them. https://twitter.com/sheeraf/status/1445099150316503057?s=21 https://twitter.com/sheeraf/status/1445099150316503057?s=21

mkaszkowiak 5y ago

This outage is huge. I'm waiting for the write-up, assuming they release one

TedShiller 5y ago

This is honestly the best feature Facebook has ever developed. I hope it's permanent. It has the following effects: you feel better about yourself, you can spend more time with your family, you are more productive.

mro_name 5y ago

frankly, who cares? Seriously. Those services are toxic for years now and everybody knows that. Who still uses them occasionally let alone relies on them can't be helped, can they?

deleted 5y ago

[deleted]

Animats 5y ago

Facebook outage is now the top story on CNN and Fox. Facebook stock down 5%. Facebook is not returning calls from Fox, or CNN.

someonehere 5y ago

When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt. I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.

tomohawk 5y ago

Many local governments use FB to get info out. Events like this show they should use multiple outlets instead of the big monopoly. Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.

muthdra 5y ago

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow and "having trouble serving requests".

Animats 5y ago

Outage is top story on CNN and Fox. Facebook is not returning their calls. Sheera Frenkel at the New York Times has been able to get a little more info, but not much. Now Twitter is starting to have problems with overload.

deleted 5y ago

[deleted]

muthdra 5y ago

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow.

babuskov 5y ago

Is HN hit by something as well? It's loading really slow for me.

runawaybottle 5y ago
I think the mods throttled logged in users to discourage over-discussion and thread creation. There’s no rhyme or reason for read-only to be snappy and logged-in to be crawling.

erjjones 5y ago

Does everyone just buy in that this is just a network change gone wrong? OR could they be mitigating a breach/hack? OR could it be some other theory?

polack 5y ago

After changing the screen resolution all operating systems will prompt the user if the applied settings where correct, otherwise it will time out and reset to last known good setting. Maybe time for the core internet infrastructure to implement something similar? :)

thealistra 5y ago
You can’t really make a system that will unboil an egg.

barbs 5y ago

I find myself a little bit happy that it's down. I use Facebook quite often, but mostly because everyone else I know uses it. If everyone is forced to find an alternative, that'd be fine by me.

DebtDeflation 5y ago

I assume all of the tertiary sites that use "Login with Facebook" are broken now too? So glad I never adopted that.

pmlittle 5y ago

This is all total left brain looping and complexity coming home to roost.

quattrofan 5y ago

And right now, the world is a slightly better place

EGreg 5y ago

Telegram seems down too, is it down for you?

jacobwilliamroy 5y ago

Should say "Facebook owned. Sites are down."

antocv 5y ago

Its alive! drill @1.1.1.1 www.facebook.com ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2172 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com. star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36

kblev 5y ago
"Sorry, something went wrong. Facebook © 2020"
- simonklitj 5y ago
  Yes, even Facebook falls prey to the wrong copyright year. Anyway, I got further now to a page that says "Account Temporarily Unavailable." and has the old Facebook layout. Would love a peek inside the Facebook codebase to see how this happens, hah!
  - ultrarunner 5y ago
    Definitely not recent, but you might find this interesting nonetheless: https://gist.github.com/nikcub/3833406 https://gist.github.com/nikcub/3833406
strenholme 5y ago
Kinda sorta. There are four DNS servers for Facebook: 129.134.30.12, 129.134.31.12, 185.89.218.12, and 185.89.219.12. Of those, only 185.89.219.12 is up right now (Edit All four DNS servers are now up). For people who want to add Facebook to hosts.txt, the A record (IP) I’m getting right now is 157.240.11.35 (it was 31.13.70.36)
daniellehmann 5y ago
See e.g. https://www.digwebinterface.com/?hostnames=facebook.com&ns=all https://www.digwebinterface.com/?hostnames=facebook.com&ns=a... for responses from different nameservers.

david_acm 5y ago

pings to a.ns.facebook.com are no longer timing out

shreezus 5y ago

It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections. I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.

gunshai 5y ago

Posting this comment will be like farting into a hurricane, but here goes. Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity. What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct. Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.

osrec 5y ago
Margin calls don't really exist as far as loans are concerned. Once you have agreed collateral, and an agreed schedule of payment, you only get in trouble if you miss a payment, regardless of how the collateral fluctuates in value.
- gunshai 5y ago
  Okay, but as the value of the collateral approaches 0 your lender asks you to increase your collateral correct?
  - odonnellryan 5y ago
    Yes. But banks won't loan 100% on equities. This scenario would just be basically impossible.
    - gunshai 5y ago
      I am not sure you understand my question / hypothetical. A bank is not the only form of a lender first off, second the reason there isn't a 100% loan on an equity is that it's understood that the value of the underlying collateral can fluctuate. These are called over collateralized loans.

cphoover 5y ago

now if only tiktok would fail

jader201 5y ago

FB seems to be finally loading for me, after nearly 6 hours. This will be a highly discussed topic for a bit.

korethr 5y ago

At 21:44 UTC, facebook.com resolves for me.

amediauk1 5y ago

tracert 129.134.30.12 Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254] 2 3 ms 3 ms 3 ms 172.16.14.63 3 * 5 ms 3 ms 213.121.98.145 4 5 ms 3 ms 4 ms 213.121.98.144 5 17 ms 8 ms 18 ms 87.237.20.142 6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149] 7 * * * Request timed out. 8 * * * Request timed out. 9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185] 10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38] 11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221] 12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173] 13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129] 14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102] 15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97] 17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143] 18 91 ms 93 ms 90 ms 157.240.36.97 19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12] Trace complete. this is what i got now

amediauk1 5y ago

tracert 129.134.30.12 Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms eehub.home [192.168.1.254] 2 3 ms 3 ms 3 ms 172.16.14.63 3 \* 5 ms 3 ms 213.121.98.145 4 5 ms 3 ms 4 ms 213.121.98.144 5 17 ms 8 ms 18 ms 87.237.20.142 6 8 ms 6 ms 7 ms lag-107.ear3.London2.Level3.net [212.187.166.149] 7 \* \* \* Request timed out. 8 \* \* \* Request timed out. 9 7 ms 7 ms 6 ms be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185] 10 70 ms 69 ms 70 ms be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38] 11 73 ms 73 ms 74 ms be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221] 12 84 ms 85 ms 84 ms be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173] 13 90 ms 90 ms 90 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129] 14 143 ms 142 ms 143 ms po111.asw02.sjc1.tfbnw.net [173.252.64.102] 15 114 ms 119 ms 114 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 16 125 ms 126 ms 124 ms be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97] 17 91 ms 92 ms 91 ms po734.psw03.ord2.tfbnw.net [129.134.35.143] 18 91 ms 93 ms 90 ms 157.240.36.97 19 74 ms 74 ms 73 ms a.ns.facebook.com [129.134.30.12] Trace complete.

ctur 5y ago

It's back as of approx 14:47 PST.

tcarn 5y ago
Agreed, website loading here, still no whatsapp though.

josalhor 5y ago

facebook.com resolves again! > ping facebook.com PING facebook.com (31.13.83.36) 56(84) bytes of data. 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=1 ttl=54 time=12.2 ms 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=2 ttl=54 time=12.1 ms 64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=3 ttl=54 time=11.7 ms Can't yet traceroute to a.ns.facebook.com tho

nazgulsenpai 5y ago

I just got the login page. It was fun while it lasted.

rstupek 5y ago

Looks like someone found the light switch and turned everything back on!

stonks 5y ago

Facebook and Messenger are working now. Instagram and WhatsApp - not yet.

sss111 5y ago
I was hoping it would stay down for longer haha

spicybright 5y ago

Still amazes me their infra team is supposedly the best in the world, and compensated as such, yet things like this happen. Personally I'm glad FB went down for a few hours, but it's hard to imagine how that would happen in the first place.

deleted 5y ago

[deleted]

spurgu 5y ago

They seem to be back now.

ExtraE 5y ago

Any way to read this without an account?

markus_zhang 5y ago

Just a sidenote I think outlook is down as well.

dnndev 5y ago

I am soooo happy I did not even notice! Don’t use any of these apps.

avodonosov 5y ago

When I click the HN link I am presented with facebook login page. I don't have an account so can not proceed. Is the article link just https://facebook.com/ https://facebook.com/ ?

lom 5y ago
No, it’s back up by now. The link was to show that facebook was down.

cemoktra 5y ago

Imagine this happening to AWS

infologics 5y ago

I want to to thank you for this good read!! I definitely enjoyed every little bit of it. I've got you book-marked to look at new stuff you post… http://infocampus.co.in/ui-development-training-in-bangalore.html http://infocampus.co.in/ui-development-training-in-bangalore... https://www.firstenquiry.com/ui-development-training-in-bangalore https://www.firstenquiry.com/ui-development-training-in-bang... http://infocampus.co.in/web-designing-training-in-bangalore.html http://infocampus.co.in/web-designing-training-in-bangalore.... http://infocampus.co.in/web-development-training-in-bangalore.html http://infocampus.co.in/web-development-training-in-bangalor... http://infocampus.co.in/react-js-training-in-bangalore http://infocampus.co.in/react-js-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.htm...

Nasreddin_Hodja 5y ago

Ha-ha!

segah 5y ago

My father (https://bit.ly/3acZAAI https://bit.ly/3acZAAI), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake. Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.