7 ms·

Facebook Down, Like Buttons Vanish, Internet Implodes

162 points by aresant 16y ago

ljf 16y ago
maybe its the 1 billion like buttons they serve a day that tipped them over the edge. ;)
- noodle 16y ago
  perhaps the Stuxnet virus was built to target facebook? ;)
  - byoung2 16y ago
    You don't get to 500 million friends without making a few enemies
    - judofyr 16y ago
      Wait, doesn't Facebook already have 500 million enemies?
aresant 16y ago
It's hard to overstate the damage this is causing for merchants that prominently display FB like on their pages - lots of big ugly blank spaces / errors out there. Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.
- j-g-faustus 16y ago
  Do better systems exist? A few hours downtime in several years is still better than most brick-and-mortar stores, I think.
- byoung2 16y ago
  Or people who rely on Facebook for shared login...that must be down as well.
  - aresant 16y ago
    Good point. We recently had a client set their "signup" to FB login by default and I cringed. Great, simple services but the macro effect that this is going to have on commerce today is crazy.
  - justinbach 16y ago
    You're telling me--of course this has to happen the day before client review of a major Facebook Connect site whose only authentication is via Facebook Accounts. We had discussed potentially having two types of accounts (Facebook- and email-based), but having two parallel authentication methods just seemed clunky. I guess this is the price we pay for trying to keep things simple...
  - seancron 16y ago
    This must really be hurting Quora. Luckily, they added Twitter authentication as well. But still, now a large portion of their site cannot login because of Facebook.
  - robryan 16y ago
    Seeing as Facebook login lets you easily collect email addresses these days you could always have a page that says, facebook is down click here to get an email confirmation link to login or something similar.
  - joey_bananas 16y ago
    Well frankly, If you rely on some other business for such critical functionality, you deserve to lose.
    - kgermino 16y ago
      Unfortunately its difficult to avoid relying on other businesses for crucial systems. Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa. Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.
- bl4k 16y ago
  for like buttons you should iframe them (same for anything from a third-party service) facebook uptime has been good overall, considering they doubled in size in 12 months to 550m users
  - sanj 16y ago
    Even iframed, you end up with an ugly "service unavailable" blot of text.
StavrosK 16y ago
I thought I had a crappy router, I didn't realise it was that crappy!
antidaily 16y ago
They closed shop after DHH's scathing criticism.
- alphadog 16y ago
  DHH's fantasies realized.
  - scorpion032 16y ago
    More like zuckerberg making a statement to detractors, how important they are.
- hrabago 16y ago
  They knew the gig was up and decided to cash in before everyone realized they really weren't worth $33B.
- borism 16y ago
  that and the movie. enough is enough!
aquarin 16y ago
But I can still harvest my cows ...
- deleted 16y ago
  [deleted]
  - bhjodokast 16y ago
    You're not a bowler, are you?
  - aquarin 16y ago
    Farmville. Or it is not available in China?
  - deleted 16y ago
    [deleted]
jacquesm 16y ago
Don't place 3rd party javascript on your pages if you don't really need it.
- natrius 16y ago
  Facebook is our biggest traffic source after Google. I'm inclined to forgive them for occasional outages.
  - jacquesm 16y ago
    I did say: > if you don't really need it. In your case, you obviously do.
    - natrius 16y ago
      I don't think we're uncommon. Most sites that I see plastered with Facebook embeds are publications, and I'm sure they all see results similar to ours. Facebook is RSS for the masses.
  - chopsueyar 16y ago
    PPC? CPM?
    - natrius 16y ago
      Not ads. We post links to people who like us on Facebook, and we also get traffic from people who share our articles with their friends.
KingOfB 16y ago
The one day I choose to update my FBConnect code!
smackfu 16y ago
I like how their API response time chart intentionally doesn't have a Y scale. But clearly it is busted: http://www.facebook.com/developers/chart.php?type=at_total_time http://www.facebook.com/developers/chart.php?type=at_total_t...
- someone_here 16y ago
  Oh dear, DNS error on that link.
  - alphabeat 16y ago
    As somebody pointed out, the www record was removed probably to stop the barrage of requests coming in.
bond 16y ago
Rarely use it so I didn't notice...
rwhitman 16y ago
And that, my friends, is why its risky to build your business on top of FB Graph API. In terms of reliability, FB is not twitter, but it certainly isn't google either...
- ryanwaggoner 16y ago
  I came here to post this same thing, but I just started thinking about it. When was the last time FB went down like this and took a bunch of core functionality down with it? Has it ever happened? And how long are they down...30 mins? Is that 30 mins of downtime every few years really more damaging than that the (potential) value that integration with the graph API or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe you have 100x as many users as you'd have otherwise. I think the real reason not to build your business on the FB API (or Twitter or whoever) is strategic, not because of potential downtime.
  - tonfa 16y ago
    They had a BGP-related screw-up a couple of weeks ago as well.
  - jbillingsley 16y ago
    I fully agree with you. If your app is heavily social it would be ridiculous not to leverage the worlds largest social network, even with very occasional down-time.
  - rwhitman 16y ago
    You're right, I should have phrased it "And that, my friends, is one of the many reasons its risky to build your entire business on top of FB Graph API." Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc. Not saying don't use FB ever, just that it's risky.
    - ryanwaggoner 16y ago
dansingerman 16y ago
As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems the term must be blacklisted in some way along with all the swears and other undesirable terms. Which seems a bit underhand by Twitter. Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged about this issue here: http://blog.dansingerman.com/post/1174729229/twitter-mostly-a-force-for-good http://blog.dansingerman.com/post/1174729229/twitter-mostly-... (if anyone cares)
- bl4k 16y ago
  during one of the twitter hacks the blacklist application was accessed and facebook was on the list (as was google and a bunch of other brand names)
  - rryyan 16y ago
    I'm curious about this and Googling failed me, do you have a link?
    - bl4k 16y ago
      I took a screenshot on my other machine - I will upload it when I am on there next There were hundreds of items on the list (Edit: here is this link for now: http://techcrunch.com/2009/07/15/another-security-tip-for-twitter-dont-use-password-as-your-password/ http://techcrunch.com/2009/07/15/another-security-tip-for-tw... I am sure that I saved the entire page of blacklisted words)
      - sssparkkk 16y ago
        That screenshot shows which of the current (more or less I guess) topics need to be blacklisted. Not which words are always blacklisted.
  - Astro9k 16y ago
    Wow. So twitter censors disparaging comments about companies? Wow.
orblivion 16y ago
I'm sure somebody out there told you so
yarapavan 16y ago
Current Status: API Latency Issues We are currently experiencing latency issues with the API, and we are actively investigating. We will provide an update when either the issue is resolved or we have an ETA for resolution.
deleted 16y ago
[deleted]
mtrn 16y ago
Who's next to go down? downrightnow.com?
- rryyan 16y ago
  downforeveryoneorjustme was throwing errors when I tried it: http://downforeveryoneorjustme.com/facebook.com http://downforeveryoneorjustme.com/facebook.com
bl4k 16y ago
Well somebody has screwed it up on their end. Only the www record is not resolving, a lot of the others are, for eg: http://developers.facebook.com/ http://developers.facebook.com/ $ nslookup facebook.com ns1.facebook.com Server: ns1.facebook.com Address: 204.74.66.132#53 Name: facebook.com Address: 69.63.189.11 Name: facebook.com Address: 69.63.181.12 Name: facebook.com Address: 69.63.189.16 $ nslookup www.facebook.com ns1.facebook.com Server: ns1.facebook.com Address: 204.74.66.132#53 Non-authoritative answer: *** Can't find www.facebook.com: No answer (YMMV based on local cache etc.)
- deleted 16y ago
  [deleted]
- jamii 16y ago
  Hilariously touch.facebook.com still seems to work.
- klochner 16y ago
  I imagine that would be a relatively easy fix, more likely they took the record down to debug internally without getting flooded with requests.
  - bl4k 16y ago
    It looks like you were right - they dropped that record to stop the requests coming in
doron 16y ago
IT Dept. around the world will now get washed with calls, on how the internets dont work.
- nkassis 16y ago
  the BOFH will be making a list of people wasting time on facebook for later blackmail.
  - doron 16y ago
    joking aside, this actually had this effect, as so many pages actually resolve some components to facebook.com, the slower response was felt on a wide variety of sites.
ax0n 16y ago
I [Like] this.
tfh 16y ago
It's a funny coincidence that this comes right after the recent facebook valuation news. http://en.wikipedia.org/wiki/Evil_eye http://en.wikipedia.org/wiki/Evil_eye
grandalf 16y ago
Is this a coordinated blackhat attack with DHH's PR attack intended to benefit some investor somewhere? (joking)
parbo 16y ago
First I just had an empty feed, then I got the feed back but posting links and comments timed out. Then it worked for a minute, and now I get the DNS error. Seems like this affects several of Facebook's applications.
chrisgoodrich 16y ago
Everyone should calm down and realize that Facebook being down for a few hours is not going to kill anyone's business. It may cause trouble and disruption, but not anything more intense than a daily commute in Los Angeles. That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.
- Terretta 16y ago
  > "If they truly want to build a completely social internet, this kind of downtime can't happen." Why not? Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue. Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else. The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on. "Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not. The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.
deleted 16y ago
[deleted]
dirktheman 16y ago
Nice to see the big boys experiencing mundane issues as well now and then...
- bhjodokast 16y ago
  I'm actually surprised they don't have weekly maintenance for a few hours...
9ec4c12949a4f3 16y ago
In A.D. 2010 War was Beginning. Internet: What happen? MarkZ: Somebody set up us the bomb. FB: We get signal MarkZ: What?! FB: Main screen turn on. MarkZ: It's you!! DDH: How are you gentlemen !! DDH: All your Information are belong to us. DDH: You are on the way to destruction.
daten 16y ago
The significance of the "like" button missing from other sites is lost on me. I've been using the Application Boundaries Enforcer feature of no-script in Firefox to block facebook content when I'm not visiting a facebook site directly. Site .facebook.com .fbcdn.net Accept from .facebook.com .fbcdn.net Deny This is motivated by privacy concerns of Facebook's ability to track which partner sites I visit.
bbuffone 16y ago
In researching web performance DNS configurations issues are common problem. It is usually related to www.foo.com and foo.com being configured differently. This seems to be at a high level the issue with facebook.com. if you go to "facebook.com" the DNS will resolve correctly, but it will redirect you to www.facebook.com. I guess they could stop the redirection and fix the problem immediately, or wait till the sorry.ak.facebook.com.edgesuite.net is fixed. =====www.facebook.com new-host-3:dpu2 robertbuffone$ dig -trace www.facebook.com ;; Warning, ignoring invalid type race ; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace www.facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19626 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 0 IN CNAME sorry.ak.facebook.com.edgesuite.net. sorry.ak.facebook.com.edgesuite.net. 0 IN CNAME a1030.g.akamai.net. a1030.g.akamai.net. 4 IN A 63.84.59.59 a1030.g.akamai.net. 4 IN A 63.84.59.10 ;; Query time: 10 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:29 2010 ;; MSG SIZE rcvd: 144 ========facebook.com new-host-3:dpu2 robertbuffone$ dig -trace facebook.com ;; Warning, ignoring invalid type race ; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19984 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; ANSWER SECTION: facebook.com. 352 IN A 69.63.181.12 facebook.com. 352 IN A 69.63.189.16 facebook.com. 352 IN A 69.63.189.11 ;; Query time: 13 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:44 2010 ;; MSG SIZE rcvd: 78
djcapelis 16y ago
Their SSL site still continues to simply not work: www.facebook.com uses an invalid security certificate. The certificate is only valid for the following names: a248.e.akamai.net , *.akamaihd.net This is the biggest problem with the SSL PKI. It is partially responsible for the lackluster deployment of SSL. We need to switch to a TOFU model supplemented by a more nimble version of the current PKI for first connect acceptance. You would not believe how often I run into problems just trying to use SSL on major sites. If you can't trust a large organizations like facebook and akamai to mess up SSL support during problems, you simply will never get it deployed well on all the rest of the sites. SSL is simply a poor user experience. Twitter's SSL site often goes down more often than the main site too. SSL users get a second class experience and that is the most damming thing I can think of for a technology where deployment is so critical.
- lanstein 16y ago
  Excellent points, but please be sure to simply never split infinitives. ;)
  - lukev 16y ago
    I shall continue to boldly split all the infinitives I please. We speak English, despite the best efforts of medieval Latin grammarians.
mjgoins 16y ago
This affects me in no direct way. I officially have a good life.
- ElbertF 16y ago
  Same here, I already have most of Facebook's domains in my hosts file. I haven't seen a Like button in ages.
akadien 16y ago
Thank goodness for Ping ...
spolsky 16y ago
cauliflower farms everywhere are withering for lack of water.
- moultano 16y ago
  The first 10 times I read this comment, I thought it was a pun on facebook's color scheme, until I realized that it's cornflower blue, not cauliflower blue.
  - bmunro 16y ago
    It's cornflour, not flower
    - whopa 16y ago
      Huh? It is cornflower: http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Cornflower_blue http://en.wikipedia.org/wiki/Cornflower_blue
      - al_james 16y ago
        Yes, because cornflour blue would be, well, white.
- danielnicollet 16y ago
  Please don't downvote me. I am not a native speaker. But I still don't get it. Someone care to explain ;-)
  - patrickmclaren 16y ago
    FarmVille.
    - wazoox 16y ago
      As a non facebook user, I didn't get it either... :)
whereareyou 16y ago
Aaaaaand, they're back. Down for about 15 mins (from Tucson, AZ).
joebasirico 16y ago
When I heard FB was down I went over to twitter to see how many tweets this was generating, I was surprised to see the first refresh of "XXXX more tweets since you started searching." I decided to start keeping track and I made a graph of the difference between refreshes. You can see my graph here. It seems like in the course of just a few minutes (I was probably conducting my little experiment for 10 min?) you can see the frenzy gaining momentum then it seems to die down until only hundreds of people are talking about it per 15 second refresh. A little bit about the graph, each tick mark on the bottom represents a refresh (15 seconds or so I think). The Y axis is the difference between refreshes. Graph: http://drp.ly/e3oH http://drp.ly/e3oH
Tichy 16y ago
At least Hacker News still works. All is well.
chewbranca 16y ago
heh... ironically facebook's error rate is at an all time low!! http://developers.facebook.com/live_status http://developers.facebook.com/live_status
chopsueyar 16y ago
I demand a refund.
stevefink 16y ago
DNS issues are the cause for so many outages by a myriad of prestigious web shops. It's incredible how often overlooked something so pivotal to your infrastructure goes without being properly monitored. I have seen entire businesses suffer hours of downtime because their monitoring systems would query an authoritative name server from RFC1918 IP space - but not the same from an externally visible address. An NS not responding to local queries can be just as detrimental internally as they are externally because of how web services are architected, obviously. In their defense, Facebook rarely incurs outages of this capacity, and I can bet it won't happen again and someone is getting seriously reamed. They can and will happen to the best of them, no matter how many MIT PhDs you have designing your systems.
zppx 16y ago
I experienced a facebook outage yesterday afternoon (-03:00 UTC) although the DNS resolved the facebook I could not ping their IP addresses. As you can read in facebook's twitter timeline.
code_duck 16y ago
It's odd, I've been hearing about this all day but I have had no problems at all contacting facebook.

162 points by aresant 16y ago

ljf 16y ago

maybe its the 1 billion like buttons they serve a day that tipped them over the edge. ;)

noodle 16y ago
perhaps the Stuxnet virus was built to target facebook? ;)
- byoung2 16y ago
  You don't get to 500 million friends without making a few enemies
  - judofyr 16y ago
    Wait, doesn't Facebook already have 500 million enemies?

aresant 16y ago

It's hard to overstate the damage this is causing for merchants that prominently display FB like on their pages - lots of big ugly blank spaces / errors out there. Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.

j-g-faustus 16y ago
Do better systems exist? A few hours downtime in several years is still better than most brick-and-mortar stores, I think.
byoung2 16y ago
Or people who rely on Facebook for shared login...that must be down as well.
- aresant 16y ago
  Good point. We recently had a client set their "signup" to FB login by default and I cringed. Great, simple services but the macro effect that this is going to have on commerce today is crazy.
- justinbach 16y ago
  You're telling me--of course this has to happen the day before client review of a major Facebook Connect site whose only authentication is via Facebook Accounts. We had discussed potentially having two types of accounts (Facebook- and email-based), but having two parallel authentication methods just seemed clunky. I guess this is the price we pay for trying to keep things simple...
- seancron 16y ago
  This must really be hurting Quora. Luckily, they added Twitter authentication as well. But still, now a large portion of their site cannot login because of Facebook.
- robryan 16y ago
  Seeing as Facebook login lets you easily collect email addresses these days you could always have a page that says, facebook is down click here to get an email confirmation link to login or something similar.
- joey_bananas 16y ago
  Well frankly, If you rely on some other business for such critical functionality, you deserve to lose.
  - kgermino 16y ago
    Unfortunately its difficult to avoid relying on other businesses for crucial systems. Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa. Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.
bl4k 16y ago
for like buttons you should iframe them (same for anything from a third-party service) facebook uptime has been good overall, considering they doubled in size in 12 months to 550m users
- sanj 16y ago
  Even iframed, you end up with an ugly "service unavailable" blot of text.

StavrosK 16y ago

I thought I had a crappy router, I didn't realise it was that crappy!

antidaily 16y ago

They closed shop after DHH's scathing criticism.

alphadog 16y ago
DHH's fantasies realized.
- scorpion032 16y ago
  More like zuckerberg making a statement to detractors, how important they are.
hrabago 16y ago
They knew the gig was up and decided to cash in before everyone realized they really weren't worth $33B.
borism 16y ago
that and the movie. enough is enough!

aquarin 16y ago

But I can still harvest my cows ...

deleted 16y ago
[deleted]
- bhjodokast 16y ago
  You're not a bowler, are you?
- aquarin 16y ago
  Farmville. Or it is not available in China?
- deleted 16y ago
  [deleted]

jacquesm 16y ago

Don't place 3rd party javascript on your pages if you don't really need it.

natrius 16y ago
Facebook is our biggest traffic source after Google. I'm inclined to forgive them for occasional outages.
- jacquesm 16y ago
  I did say: > if you don't really need it. In your case, you obviously do.
  - natrius 16y ago
    I don't think we're uncommon. Most sites that I see plastered with Facebook embeds are publications, and I'm sure they all see results similar to ours. Facebook is RSS for the masses.
- chopsueyar 16y ago
  PPC? CPM?
  - natrius 16y ago
    Not ads. We post links to people who like us on Facebook, and we also get traffic from people who share our articles with their friends.

KingOfB 16y ago

The one day I choose to update my FBConnect code!

smackfu 16y ago

I like how their API response time chart intentionally doesn't have a Y scale. But clearly it is busted: http://www.facebook.com/developers/chart.php?type=at_total_time http://www.facebook.com/developers/chart.php?type=at_total_t...

someone_here 16y ago
Oh dear, DNS error on that link.
- alphabeat 16y ago
  As somebody pointed out, the www record was removed probably to stop the barrage of requests coming in.

bond 16y ago

Rarely use it so I didn't notice...

rwhitman 16y ago

And that, my friends, is why its risky to build your business on top of FB Graph API. In terms of reliability, FB is not twitter, but it certainly isn't google either...

ryanwaggoner 16y ago
I came here to post this same thing, but I just started thinking about it. When was the last time FB went down like this and took a bunch of core functionality down with it? Has it ever happened? And how long are they down...30 mins? Is that 30 mins of downtime every few years really more damaging than that the (potential) value that integration with the graph API or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe you have 100x as many users as you'd have otherwise. I think the real reason not to build your business on the FB API (or Twitter or whoever) is strategic, not because of potential downtime.
- tonfa 16y ago
  They had a BGP-related screw-up a couple of weeks ago as well.
- jbillingsley 16y ago
  I fully agree with you. If your app is heavily social it would be ridiculous not to leverage the worlds largest social network, even with very occasional down-time.
- rwhitman 16y ago
  You're right, I should have phrased it "And that, my friends, is one of the many reasons its risky to build your entire business on top of FB Graph API." Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc. Not saying don't use FB ever, just that it's risky.
  - ryanwaggoner 16y ago

dansingerman 16y ago

As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems the term must be blacklisted in some way along with all the swears and other undesirable terms. Which seems a bit underhand by Twitter. Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged about this issue here: http://blog.dansingerman.com/post/1174729229/twitter-mostly-a-force-for-good http://blog.dansingerman.com/post/1174729229/twitter-mostly-... (if anyone cares)

bl4k 16y ago
during one of the twitter hacks the blacklist application was accessed and facebook was on the list (as was google and a bunch of other brand names)
- rryyan 16y ago
  I'm curious about this and Googling failed me, do you have a link?
  - bl4k 16y ago
    I took a screenshot on my other machine - I will upload it when I am on there next There were hundreds of items on the list (Edit: here is this link for now: http://techcrunch.com/2009/07/15/another-security-tip-for-twitter-dont-use-password-as-your-password/ http://techcrunch.com/2009/07/15/another-security-tip-for-tw... I am sure that I saved the entire page of blacklisted words)
    - sssparkkk 16y ago
      That screenshot shows which of the current (more or less I guess) topics need to be blacklisted. Not which words are always blacklisted.
- Astro9k 16y ago
  Wow. So twitter censors disparaging comments about companies? Wow.

orblivion 16y ago

I'm sure somebody out there told you so

yarapavan 16y ago

Current Status: API Latency Issues We are currently experiencing latency issues with the API, and we are actively investigating. We will provide an update when either the issue is resolved or we have an ETA for resolution.

deleted 16y ago

[deleted]

mtrn 16y ago

Who's next to go down? downrightnow.com?

rryyan 16y ago
downforeveryoneorjustme was throwing errors when I tried it: http://downforeveryoneorjustme.com/facebook.com http://downforeveryoneorjustme.com/facebook.com

bl4k 16y ago

Well somebody has screwed it up on their end. Only the www record is not resolving, a lot of the others are, for eg: http://developers.facebook.com/ http://developers.facebook.com/ $ nslookup facebook.com ns1.facebook.com Server: ns1.facebook.com Address: 204.74.66.132#53 Name: facebook.com Address: 69.63.189.11 Name: facebook.com Address: 69.63.181.12 Name: facebook.com Address: 69.63.189.16 $ nslookup www.facebook.com ns1.facebook.com Server: ns1.facebook.com Address: 204.74.66.132#53 Non-authoritative answer: *** Can't find www.facebook.com: No answer (YMMV based on local cache etc.)

deleted 16y ago
[deleted]
jamii 16y ago
Hilariously touch.facebook.com still seems to work.
klochner 16y ago
I imagine that would be a relatively easy fix, more likely they took the record down to debug internally without getting flooded with requests.
- bl4k 16y ago
  It looks like you were right - they dropped that record to stop the requests coming in

doron 16y ago

IT Dept. around the world will now get washed with calls, on how the internets dont work.

nkassis 16y ago
the BOFH will be making a list of people wasting time on facebook for later blackmail.
- doron 16y ago
  joking aside, this actually had this effect, as so many pages actually resolve some components to facebook.com, the slower response was felt on a wide variety of sites.

ax0n 16y ago

I [Like] this.

tfh 16y ago

It's a funny coincidence that this comes right after the recent facebook valuation news. http://en.wikipedia.org/wiki/Evil_eye http://en.wikipedia.org/wiki/Evil_eye

grandalf 16y ago

Is this a coordinated blackhat attack with DHH's PR attack intended to benefit some investor somewhere? (joking)

parbo 16y ago

First I just had an empty feed, then I got the feed back but posting links and comments timed out. Then it worked for a minute, and now I get the DNS error. Seems like this affects several of Facebook's applications.

chrisgoodrich 16y ago

Everyone should calm down and realize that Facebook being down for a few hours is not going to kill anyone's business. It may cause trouble and disruption, but not anything more intense than a daily commute in Los Angeles. That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.

Terretta 16y ago
> "If they truly want to build a completely social internet, this kind of downtime can't happen." Why not? Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue. Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else. The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on. "Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not. The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.

deleted 16y ago

[deleted]

dirktheman 16y ago

Nice to see the big boys experiencing mundane issues as well now and then...

bhjodokast 16y ago
I'm actually surprised they don't have weekly maintenance for a few hours...

9ec4c12949a4f3 16y ago

In A.D. 2010 War was Beginning. Internet: What happen? MarkZ: Somebody set up us the bomb. FB: We get signal MarkZ: What?! FB: Main screen turn on. MarkZ: It's you!! DDH: How are you gentlemen !! DDH: All your Information are belong to us. DDH: You are on the way to destruction.

daten 16y ago

The significance of the "like" button missing from other sites is lost on me. I've been using the Application Boundaries Enforcer feature of no-script in Firefox to block facebook content when I'm not visiting a facebook site directly. Site .facebook.com .fbcdn.net Accept from .facebook.com .fbcdn.net Deny This is motivated by privacy concerns of Facebook's ability to track which partner sites I visit.

bbuffone 16y ago

In researching web performance DNS configurations issues are common problem. It is usually related to www.foo.com and foo.com being configured differently. This seems to be at a high level the issue with facebook.com. if you go to "facebook.com" the DNS will resolve correctly, but it will redirect you to www.facebook.com. I guess they could stop the redirection and fix the problem immediately, or wait till the sorry.ak.facebook.com.edgesuite.net is fixed. =====www.facebook.com new-host-3:dpu2 robertbuffone$ dig -trace www.facebook.com ;; Warning, ignoring invalid type race ; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace www.facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19626 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 0 IN CNAME sorry.ak.facebook.com.edgesuite.net. sorry.ak.facebook.com.edgesuite.net. 0 IN CNAME a1030.g.akamai.net. a1030.g.akamai.net. 4 IN A 63.84.59.59 a1030.g.akamai.net. 4 IN A 63.84.59.10 ;; Query time: 10 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:29 2010 ;; MSG SIZE rcvd: 144 ========facebook.com new-host-3:dpu2 robertbuffone$ dig -trace facebook.com ;; Warning, ignoring invalid type race ; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19984 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; ANSWER SECTION: facebook.com. 352 IN A 69.63.181.12 facebook.com. 352 IN A 69.63.189.16 facebook.com. 352 IN A 69.63.189.11 ;; Query time: 13 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:44 2010 ;; MSG SIZE rcvd: 78

djcapelis 16y ago

Their SSL site still continues to simply not work: www.facebook.com uses an invalid security certificate. The certificate is only valid for the following names: a248.e.akamai.net , *.akamaihd.net This is the biggest problem with the SSL PKI. It is partially responsible for the lackluster deployment of SSL. We need to switch to a TOFU model supplemented by a more nimble version of the current PKI for first connect acceptance. You would not believe how often I run into problems just trying to use SSL on major sites. If you can't trust a large organizations like facebook and akamai to mess up SSL support during problems, you simply will never get it deployed well on all the rest of the sites. SSL is simply a poor user experience. Twitter's SSL site often goes down more often than the main site too. SSL users get a second class experience and that is the most damming thing I can think of for a technology where deployment is so critical.

lanstein 16y ago
Excellent points, but please be sure to simply never split infinitives. ;)
- lukev 16y ago
  I shall continue to boldly split all the infinitives I please. We speak English, despite the best efforts of medieval Latin grammarians.

mjgoins 16y ago

This affects me in no direct way. I officially have a good life.

ElbertF 16y ago
Same here, I already have most of Facebook's domains in my hosts file. I haven't seen a Like button in ages.

akadien 16y ago

Thank goodness for Ping ...

spolsky 16y ago

cauliflower farms everywhere are withering for lack of water.

moultano 16y ago
The first 10 times I read this comment, I thought it was a pun on facebook's color scheme, until I realized that it's cornflower blue, not cauliflower blue.
- bmunro 16y ago
  It's cornflour, not flower
  - whopa 16y ago
    Huh? It is cornflower: http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Cornflower_blue http://en.wikipedia.org/wiki/Cornflower_blue
    - al_james 16y ago
      Yes, because cornflour blue would be, well, white.
danielnicollet 16y ago
Please don't downvote me. I am not a native speaker. But I still don't get it. Someone care to explain ;-)
- patrickmclaren 16y ago
  FarmVille.
  - wazoox 16y ago
    As a non facebook user, I didn't get it either... :)

whereareyou 16y ago

Aaaaaand, they're back. Down for about 15 mins (from Tucson, AZ).

joebasirico 16y ago

When I heard FB was down I went over to twitter to see how many tweets this was generating, I was surprised to see the first refresh of "XXXX more tweets since you started searching." I decided to start keeping track and I made a graph of the difference between refreshes. You can see my graph here. It seems like in the course of just a few minutes (I was probably conducting my little experiment for 10 min?) you can see the frenzy gaining momentum then it seems to die down until only hundreds of people are talking about it per 15 second refresh. A little bit about the graph, each tick mark on the bottom represents a refresh (15 seconds or so I think). The Y axis is the difference between refreshes. Graph: http://drp.ly/e3oH http://drp.ly/e3oH

Tichy 16y ago

At least Hacker News still works. All is well.

chewbranca 16y ago

heh... ironically facebook's error rate is at an all time low!! http://developers.facebook.com/live_status http://developers.facebook.com/live_status

chopsueyar 16y ago

I demand a refund.

stevefink 16y ago

DNS issues are the cause for so many outages by a myriad of prestigious web shops. It's incredible how often overlooked something so pivotal to your infrastructure goes without being properly monitored. I have seen entire businesses suffer hours of downtime because their monitoring systems would query an authoritative name server from RFC1918 IP space - but not the same from an externally visible address. An NS not responding to local queries can be just as detrimental internally as they are externally because of how web services are architected, obviously. In their defense, Facebook rarely incurs outages of this capacity, and I can bet it won't happen again and someone is getting seriously reamed. They can and will happen to the best of them, no matter how many MIT PhDs you have designing your systems.

zppx 16y ago

I experienced a facebook outage yesterday afternoon (-03:00 UTC) although the DNS resolved the facebook I could not ping their IP addresses. As you can read in facebook's twitter timeline.

code_duck 16y ago

It's odd, I've been hearing about this all day but I have had no problems at all contacting facebook.