13 ms·

Facebook was down

325 points by sgloutnikov 12y ago

ubersync 12y ago
I can't wait to know the reason behind this.
- vladmiller 12y ago
  Looks like cdn or part of their network routers failed, because mobile apps for instagram and facebook work fine
pmalynin 12y ago
Hmm, interestingly enough Faceboook today "has complied with a Turkish court order demanding the blocking of a page it said offended the Prophet Muhammad." [0] [0] http://www.bbc.com/news/technology-30982556 http://www.bbc.com/news/technology-30982556
- deleted 12y ago
  [deleted]
taubi19 12y ago
I wonder how many people are calling their isp call centers right now ;)
vladmiller 12y ago
Oh no, I just got used to React.js/Flux
Hengjie 12y ago
Meanwhile world's productivity goes up substantially.
- Gigablah 12y ago
  Not with everyone constantly refreshing
fapjacks 12y ago
The post mortem is going to be very interesting. It's Facebook, not like a Freenode server or something.
noobermin 12y ago
I've always thought how strange it is when people notice and make such a big deal about facebook being down or accessible. It's almost as if fb has become a public utility or something. As others have remarked, one way one can interpret this is it is a testament to their ops' exceptional ability as engineers that such downtime is so noticeable.
- jhildings 12y ago
  >It's almost as if fb has become a public utility or something. Well thats what Zuckerberg answered a user to the question "Why Facebook wasn't cool anymore". He didn't want it to be cool , but more like a digital utility like water or electricity
  - bobbykjack 12y ago
    The trouble with that analogy is that water and electricity (just about) are absolutely essential to our day-to-day lives; a specific social media site isn't. Facebook needs to appeal to the trend to keep in business. Hugely popular social networking sites have failed in the past, precisely when they've fallen out if fashion.
mhr_online 12y ago
Confirmed, from Iran
zaporozhets 12y ago
Perhaps it's a good time to re-think our dependence on facebook for 'cheat' SSO.
- sliverstorm 12y ago
  ... because they haven't reached five nines? Because your own privately hosted authentication system will?
doaluke 12y ago
yes
borgchick 12y ago
yup
munirusman 12y ago
yes
wallzb 12y ago
yes
foxhedgehog 12y ago
http://www.downforeveryoneorjustme.com/facebook.com http://www.downforeveryoneorjustme.com/facebook.com It's not just you! http://facebook.com http://facebook.com looks down from here.
- kenrick95 12y ago
  There's a shorter link http://isup.me/facebook.com http://isup.me/facebook.com
  - gemma 12y ago
    It's about quality, not quantity. http://wheresitup.com/results/54c732d88c3dcf8416232c91 http://wheresitup.com/results/54c732d88c3dcf8416232c91
dgabriel 12y ago
No one has anything else to do tonight, I guess.
deleted 12y ago
[deleted]
webo 12y ago
It doesn't help to link to the website when it is already down.
- jtreminio 12y ago
  Traffic sent from HN is but a drop in a bucket the size of a dump truck for facebook.
evjan 12y ago
Yup
WillNotDownvote 12y ago
Instagram too, apparently. Are they sharing crucial infrastructure?
- philip1209 12y ago
  http://instagram-engineering.tumblr.com/post/89992572022/migrating-aws-fb http://instagram-engineering.tumblr.com/post/89992572022/mig...
evjan 12y ago
Instagram as well
- bhaai 12y ago
  Looks like they share some infrastructure with facebook. Whatsap web and app works though :)
hyperreality 12y ago
Yup. Down. http://www.downforeveryoneorjustme.com/facebook.com http://www.downforeveryoneorjustme.com/facebook.com
abm91 12y ago
Welp...that escalated quickly.
brunorsini 12y ago
yes. the fact that it feels so weird is testimony to their server eng team
- daeken 12y ago
  It really is. Facebook is far from bug-free, but I'm a near-constant user and this is the first time in a long time that I can recall it going down completely. I'll be really surprised if it lasts more than an hour, at most.
th0br0 12y ago
Even the fbcdn / akamai part seems to be down (an error occured while processing your request) ... funny
andreasen 12y ago
yes - Sydney, 5:59pm
vanadium 12y ago
According to Twitter, Facebook falling in the dead of night does in fact make (about a few million) sounds.
- jedrek 12y ago
  There is no dead of the night on a platform with users all over the world. Maybe when it's midday over the pacific, but not even then.
  - vanadium 12y ago
    Let's suspend disbelief in the name of a stretch joke.
wsidell 12y ago
Instagram is also down
dnohr 12y ago
Yep, it's down here in SE Asia too
deleted 12y ago
[deleted]
umair-a 12y ago
yep, even ping fails. even though historically it use to work. https://bitneer/facebook.com/ping/ https://bitneer/facebook.com/ping/
- SpaceInvader 12y ago
  where's .com? ;-)
  - umair-a 12y ago
    sorry about that :) https://bitneer.com/facebook.com/ping/ https://bitneer.com/facebook.com/ping/ working now of course.
flaie 12y ago
Akamai, the CDN used by Facebook went down before Facebook for me this morning: http://www.downforeveryoneorjustme.com/akamaihd.net http://www.downforeveryoneorjustme.com/akamaihd.net
- lewi 12y ago
  Seems to be effecting Instagram as well. I assume they both use Akamai then. Was using both when they became unresponsive, resetting routers all over the shop.
- clarkm 12y ago
  Looking through the customer list, most other sites seem to be working fine: http://www.akamai.com/html/customers/customer_list.html http://www.akamai.com/html/customers/customer_list.html I wonder what makes FB and Instagram different?
  - flaie 12y ago
    It was specifically fbstatic-b-a.akamaihd.net which was also not responsive, preventing css & js to load rendering a completely broken page on my browser.
  - fivedogit 12y ago
    Akamai has several different networks, one for streaming media, one for regular http, and one for https. They also have an S3-like storage system and a DNS service. Some customers are configured with several of these offerings at once, so it's hard to know which, if any, experienced problems and led to the outages. Honestly, I doubt it was an Akamai issue. If Akamai experienced network problems, dozens or hundreds of sites would be affected. If an Akamai config issue (ie human error) were to blame, then it would probably only affect one site, not several. Neither FB nor Akamai is dumb enough to push multiple site changes at once.
vaidik 12y ago
https://www.dropbox.com/s/zueam1oq3sdzv36/Screenshot%202015-01-27%2011.54.49.png?dl=0 https://www.dropbox.com/s/zueam1oq3sdzv36/Screenshot%202015-...
- clarkm 12y ago
  Charge your battery!
greenvaio 12y ago
DOWN for me too.
dnohr 12y ago
Yes, it's also down in SE Asia
greenvaio 12y ago
Lizard squard???
sidcool 12y ago
It's down for me as well (11:50 AM IST) in Mumbai, India.
deleted 12y ago
[deleted]
djacobs 12y ago
I restarted my modem and router two times before I realized that the network was fine and that it was just Facebook that was down. Props to their ops team for uptime that is so reliable.
- RandallBrown 12y ago
  I haven't seen Facebook go down in years. I wonder if this is going to be a big deal.
  - jmtame 12y ago
    I'm wondering how much revenue gets lost every minute that Facebook is down. It has to be in the tens of thousands. Edit: someone wrote this below in the comments, which came to $400 per second of downtime. Ouch.
    - sliverstorm 12y ago
      $400 per second somehow seems... smaller than I would have thought. Their opex on a per second basis has got to be orders of magnitude higher than that.
      - benhamner 12y ago
        $400 per second is about $13 billion per year
    - semi-extrinsic 12y ago
      For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion. (source for cost: Cormorant Alpha shutdown in 2013) Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.
      - 12y ago
deleted 12y ago
[deleted]
keyle 12y ago
This post might as well be a poll...
leowinterde 12y ago
yes
zht 12y ago
wonder if this has to do with the storm near IAD?
munirusman 12y ago
How much revenue are they losing every second?
- gburt 12y ago
  $7.87 billion in 2013 / 365 days / 24 hours / 60 minutes / 60 seconds is about $250 a second. Edit: using more up to date data: $3,300,000,000 in the 3rd quarter of 2014 / (365/4) / 24 / 60 / 60 = $418.58
  - munirusman 12y ago
    $1.5M an hour!!! But considering this is not the peak time in North America, it's should be little less.
    - fnayr 12y ago
      Yeah but I wonder if actually more people will net log in because everyone is talking about it being down.
    - together_us 12y ago
      But here in India and the rest of Asia, people wake up to Facebook. You can't ignore that.
      - q_no 12y ago
        In Asia Facebook isn't that big, they also have other social services
    - berkes 12y ago
      A site like Facebook has a different profile though. More like email. If you cannot reach your mail, you'll read it at a later point. When facebook comes up, people will catch up on "lost facebook time" most likely. Over the course of the week (or even day) it will most likely even out.
deleted 12y ago
[deleted]
phragg 12y ago
what's facebook?
willidiots 12y ago
Let's all have a moment of contemplation & relaxation on behalf of those poor souls on-call...
- joshmn 12y ago
  Our Facebook, who art in Datacenter...
  - rxaxm 12y ago
    hallowed be thy threads
    - JacobJans 12y ago
      thy pingdom come
      - davesque 12y ago
        thy browsing at work be done
        joshmn 12y ago
        on desktop as it is on mobile.
        bcantrill 12y ago
        Give us this day our daily selfies
        aaronsnoswell 12y ago
        And forgive us our sheep-throwing.
        marincounty 12y ago
goran216 12y ago
Yeah
goran216 12y ago
yeah
DigitalSea 12y ago
Yep. Down for me in Queensland, Australia as well (Facebook and Instagram). It is situations like this that give people a much needed break from social media (myself included). You don't realise just how much of a stranglehold (some more than others) that it has over you until you can't actually use it. Downtime for a site as big as Facebook is rare, but it happens, especially when your infrastructure isn't solely controlled by you (content distribution networks come to mind).
- deleted 12y ago
  [deleted]
Colargol 12y ago
yes (Central Europe)
pavelow 12y ago
Affirmative
milankragujevic 12y ago
Yep down for me too from Serbia.
Yuioup 12y ago
Something to do with the blizzard?
austinl 12y ago
It's interesting to see how Facebook downtime has a huge effect on Internet traffic – there's a Guardian report where they share their referral data while Facebook was down for 31 minutes (the longest outage in 4 years): http://www.theguardian.com/technology/2014/jun/19/facebook-unavailable-longest-outage-four-years http://www.theguardian.com/technology/2014/jun/19/facebook-u...
mhomde 12y ago
Somewhere in the world Jesse Eisenberg is throwing an epic fit
deleted 12y ago
[deleted]
vayarajesh 12y ago
Even instagram seems down
- grejdi 12y ago
  Makes sense. They share the same data centers. See: http://instagram-engineering.tumblr.com/post/89992572022/migrating-aws-fb http://instagram-engineering.tumblr.com/post/89992572022/mig...
datboitom 12y ago
Yes
gbachik 12y ago
OH MY GOD
vayarajesh 12y ago
with malaisian airlines getting hacked its making me think what if the ISIS got it to facebook?
- timpattinson 12y ago
  Very unlikely. The attack on Malaysia Airlines was a hijacking of the DNS, not an actual hack on the website. Plus with the reosurces fb dedicates to security, they have a much better chance of finding 0days than some terrorists.
arrowgunz 12y ago
It's down from here - Bay Area, California.
ujuc 12y ago
yes, @korea
ujuc 12y ago
yes @korea
gregthompsonjr 12y ago
Bet it has something to do with this damn snowstorm. I want this season to be over with!
- ptasci67 12y ago
  But http://aws.amazon.com/ http://aws.amazon.com/ and http://azure.microsoft.com/en-us/ http://azure.microsoft.com/en-us/ seem to be up so some devops work needs to be done here!
EtienneK 12y ago
South African here - Down for me too.
babo 12y ago
down in Norway
pirate21 12y ago
Here in Brazil is Down,Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:
- pirate21 12y ago
  http://check-host.net/check-http?host=facebook.com http://check-host.net/check-http?host=facebook.com
pirate21 12y ago
here in Brazil, Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:
ved_a 12y ago
Yes (from India)
bigphishy 12y ago
And with it millions of widgets and like buttons suddenly vanish from their webpages. Good riddance I say!
unimaxlin 12y ago
Its down in India
unimaxlin 12y ago
ISIS hacked into facebook network ?
- dudus 12y ago
  are you joking or do you actually have any basis for this suspicion?
t3ra 12y ago
Lizard squad associated account just tweeted this https://twitter.com/LizardMafia/status/559963134006292481 https://twitter.com/LizardMafia/status/559963134006292481
jaysonelliot 12y ago
I feel strangely free. Perhaps this is a sign that I should go back to blocking myself from Facebook with StayFocusd. Kind of sad that I needed this kick in the backside, but I'll take this as an opportunity to reclaim empty calories and wasted time.
- saraid216 12y ago
  So of course you're on Hacker News.
- code_duck 12y ago
  Might as well take a first step to ignoring facebook. I've spent more than my fair share of time on the site in the past few years. While it's been productive overall in terms of professional and personal networking, I could have achieved as much with 10% of the time if I focused solely on the productive aspects. Two weeks ago I decided I could do without endless pointless conversations with vague colleagues, being updated on largely irrelevant minute details and thoughts of people I don't actually know, and and endless stream of low quality news links. I deactivated my account for a week and then reactivated it but only visit once a day at most. My quality of life certainly has not decreased!
deleted 12y ago
[deleted]
MalcolmDiggs 12y ago
Quick, Google+ Engineers: take it down too! You don't want to fall behind.
- prostoalex 12y ago
  Gmail is 502'ing for me.
  - fuzzywalrus 12y ago
    Gmail seems to be in working order
- lmm 12y ago
  Google+ was actually down as well, but fortunately neither of their users tried to log in during the outage.
bombtrack 12y ago
Obviously anyone can "take credit" for things like this, but if this[1] is true, then jesus. Edit: Since it's topical, I enjoy listening to Chuck Rossi in interviews or presentations. Releng 2014 - Keynote 1: Chuck Rossi, Release Engineering, Facebook Inc. | Talks at Google [2] [1] https://twitter.com/lizardmafia/status/559963134006292481 https://twitter.com/lizardmafia/status/559963134006292481 [2] http://youtu.be/Nffzkkdq7GM?t=4m39s http://youtu.be/Nffzkkdq7GM?t=4m39s
- danso 12y ago
  A few years ago, who would've guessed that Twitter would be the major service still standing during such a mass attack?
  - deleted 12y ago
    [deleted]
    - deleted 12y ago
      [deleted]
  - SEJeff 12y ago
    Before they had Mesos, they wouldn't have.
- thrownaway2424 12y ago
  They had to leave twitter up so they could have somewhere to post?
  - Hexcles 12y ago
    lol. Might simply because people see white whales too often and that's not a surprise.
- joshhagler 12y ago
  I am not familiar with Lizard Mafia, do they have a history of famous hacks?
deleted 12y ago
[deleted]
quillo 12y ago
Nanog is asleep, but Ausnog might shed some light on it: http://lists.ausnog.net/pipermail/ausnog/2015-January/029463.html http://lists.ausnog.net/pipermail/ausnog/2015-January/029463...
D_Guidi 12y ago
yep
typon 12y ago
Is the API down as well? I imagine thousands of websites and apps rely on Facebook being up.
- kevan 12y ago
  The main app and Messenger aren't connecting at all for me, Facebook's OAuth is also inaccessible.
- webo 12y ago
  Yes, can't use log-in with Facebook.
- markthethomas 12y ago
  Seems to be --> https://developers.facebook.com/status/ https://developers.facebook.com/status/
jacko0 12y ago
Yea, it could be Lizard Squad. https://twitter.com/LizardMafia https://twitter.com/LizardMafia. Facebook, Instagram, Tinder, AIM, Hipchat #offline #LizardSquad
- yaeger 12y ago
  Or it could be a power issue brought on by this little bigger than average blizzard that will hit the Boston area the hardest where, incidentally, Akamai is located. But sure, couple of script kiddies might be responsible as well. It's usually one or the other.
ohmygeek 12y ago
Instagram is down as well!
stevewilhelm 12y ago
Over the last fifteen years, when ever I have experienced intermittent Web access, I try www.cisco.com. It has never been down. Ever.
anant90 12y ago
http://youtu.be/qpNNa_Mokok?t=10s http://youtu.be/qpNNa_Mokok?t=10s
arasmussen 12y ago
Back up for me, showing an error page now though.
arunabh010 12y ago
down in Singapore
TheAceOfHearts 12y ago
Seems Facebook isn't very webscale!
mirceagoia 12y ago
Down for me too.
overworkedasian 12y ago
This doesnt look right: whois facebook.com @8.8.8.8 Whois Server Version 2.0 Domain names in the .com and .net domains can now be registered with many different competing registrars. Go to http://www.internic.net http://www.internic.net for detailed information. FACEBOOK.COM.DISABLE.YOUR.TIMELINE.NOW.WITH.THE.ORIGINAL.TIMELINE-REMOVE.NET FACEBOOK.COM.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM FACEBOOK.COM.LOVED.BY.WWW.SHQIPHOST.COM FACEBOOK.COM.MORE.INFO.AT.WWW.BEYONDWHOIS.COM FACEBOOK.COM.ZZZZZ.GET.LAID.AT.WWW.SWINGINGCOMMUNITY.COM FACEBOOK.COM To single out one record, look it up with "xxx", where xxx is one of the records displayed above. If the records are the same, look them up with "=xxx" to receive a full display for each record. >>> Last update of whois database: Tue, 27 Jan 2015 06:56:04 GMT <<<
- zaporozhets 12y ago
  "Facebook finally figures out that sex /still/ sells better than advertising"
- anthony_franco 12y ago
  It's just the way whois lookups work. Do a whois lookup of any big site (google.com, microsoft.com, etc) and you'll get a similar response.
- jeffreyrogers 12y ago
  It's been that way for a while now and happens with any large site. Try google.com for example and you'll see the same thing.
- markthethomas 12y ago
  I've run WHOIS queries on FB before and gotten the same(ish) thing more than a few times -- assumed it was temporary tampering before
- axaxs 12y ago
  Appears to return hosts with name starting with search term, which doesn't make a lot of sense but has likely always been the case. Also @ flag is a dig thing, not whois.
- 12y ago
nurettin 12y ago
down here (in Turkey) and surprisingly, it isn't the government this time!
deleted 12y ago
[deleted]
toephu2 12y ago
instagram and facebook down as of this posting
phpfour 12y ago
Down from Bangladesh !
Hexcles 12y ago
Facebook did go down before, but not like this time. I tried to ping many IPs of Facebook, all even failed to reach. Can be a sign of DDoS. Anyway, if that is true, people have to take it really seriously this time.
thangh9 12y ago
FYI https://twitter.com/hashtag/facebookdown?src=tren https://twitter.com/hashtag/facebookdown?src=tren
logotype 12y ago
The only subdomain working is: http://newsroom.fb.com http://newsroom.fb.com and hosted by wordpress.com
justindocanto 12y ago
All the 'is it down' websites are slowly going down with people searching for it.
alecsx6 12y ago
It's down in Eastern Europe too.
deleted 12y ago
[deleted]
rdlecler1 12y ago
One of our vendors crashed which hung out site. Damn counterparty risk!
alexivanovs 12y ago
If they guy is posting pics of himself on Twitter asking others to "dox him", it could mean that he's pretty emotionally stable not to fall for the common pitfalls when it comes to online hacking. In that sense, it's very possible that he has access to some major database dumps that the public is unaware of, and given that he is also claiming the glory behind the Malaysia Airlines hack; it's clear that he does more than DDoS, no matter what the more educated hackers are saying. A long way to go for a secure internet, that's for sure.
rbosinger 12y ago
I blame global warming. Them servers were too hot I betcha.
joshhagler 12y ago
Just tweeted "@LizardMafia: More to come soon. Side note: We're still organizing the @MAS email dump, stay tuned for that."
ookblah 12y ago
one of the comments in thenextweb article is going nuts on the upvote/like
egil 12y ago
Up again here from Norway.
fredkelly 12y ago
It's back!
sekasi 12y ago
Back up now. Total downtime around 55 minutes that I could see.
philip1209 12y ago
Expensify is having issues too: http://status.expensify.com/ http://status.expensify.com/ I feel like this is more the norm rather than an exception for them at this point, though.
philip1209 12y ago
Facebook back online in SF as of 7:09 GMT
jpgvm 12y ago
Back up now. Will be good to read the postmortem on this one.
andydev 12y ago
Facebook seems to be back online for me.
avnerner 12y ago
Seems facebook is back. a total of 55 min or so.
vaidik 12y ago
Seems like they are back up now. 100% HA is a myth, for even those who can afford it.
tannerc 12y ago
As of 11:12 PM PST Facebook and Instagram are back online.
rbosinger 12y ago
Ok they're back for me in Alberta! What a wild ride. I pooped everywhere.
crgt 12y ago
Back up for me again
markthethomas 12y ago
I can get back on; able to see posts from hours ago, noting within 5 hours-ish. Anyone with an ops background able to give a JS eng who has no ops experience like myself a glimpse into what bringing services back online like that looks like? No experience with ddos mitigation techniques
deleted 12y ago
[deleted]
Shrugs 12y ago
Looks like a classic Stack Snowverflow error to me.
ioseph 12y ago
Annd it's back up again
plumeria 12y ago
Down here (Central America)
sandworm 12y ago
I mean, like, WOW. If they have done this we all must give them props for ambition. Forget Sony. Nobody outside of hollywood bigwigs were impacted by that. But Facebook? The US government is going to go nuts. I hope lizardsquad are driving drone-proof cars. Facebook is now back online for me on my home connection (BC Canada) and every Tor node I try. Either the attack is over or FB has fought back.
ankit-mehta 12y ago
It's Up and Running. All the websites are accessible now.
thienchu 12y ago
Looks like they're back online
deleted 12y ago
[deleted]
bigstueyc 12y ago
back up
sandworm 12y ago
Update from Facebook: "Current State: Fix Pushed Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226... Might not be a hack! Might just be weather in Boston: "Akamai (provider for FB, Instagram and so on) claims to be down due to power outage. #LizardSquad claims to have hacked them. Get popcorn." https://twitter.com/petterkarlsson/status/559975340526014464 https://twitter.com/petterkarlsson/status/559975340526014464
pskittle 12y ago
It's working for me
deleted 12y ago
[deleted]
binoyxj 12y ago
Official update: "Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226...
- kevan 12y ago
  "The issue was resolved at 23:10 PST and the site stabilized shortly afterwards. Our internal and external monitoring shows that API requests are being served with normal latency and error rates, in all geographic regions. We are sorry for any inconvenience this may have caused you and the users of your apps." No hints there, guess we'll just have to wait for a full post-mortem to come out.
  - deleted 12y ago
    [deleted]
- bombtrack 12y ago
  "This was not the result of a third-party attack but instead occurred after we introduced a change that affected our configuration systems." http://www.bbc.com/news/technology-30996928 http://www.bbc.com/news/technology-30996928
niubility 12y ago
Was completely dead in Australia. Down on both my local ISP and VPN to the National Research network. It was not a DNS issue as the IP Address is the same as it was during the outage. "Host unreachable", maybe they took out the load-balancer? - traceroute to star.c10r.facebook.com (2a03:2880:f00c:6:face:b00c:0:2) 1 redacted 2 ge-1-0-0.bb1.a.syd.aarnet.net.au (2001:388:1:5001::1) 107.649 ms 91.569 ms 91.814 ms 3 ae9.pe2.brwy.nsw.aarnet.net.au (2001:388:1:88::1) 91.981 ms 92.822 ms 103.957 ms 4 ae5.pe1.brwy.nsw.aarnet.net.au (2001:388:1:87::1) 92.447 ms 93.29 ms 101.182 ms 5 et-1-1-0.pe1.rsby.nsw.aarnet.net.au (2001:388:1:66::1) 94.606 ms 109.654 ms 91.987 ms 6 et-0-3-0.nsw-msct-bdr1.aarnet.net.au (2001:388:1:a3::2) 96.119 ms 92.032 ms 92.57 ms 7 6453.syd.equinix.com (2001:de8:6::6453:1) 99.204 ms 111.52 ms 190.499 ms 8 if-xe-0-3-1.3.thar1.1MH-Sydney.ipv6.as6453.net (2405:2000:ffd0::a) 90.392 ms 90.862 ms 92.174 ms 9 if-3-0-0.2.core1.PV4-Piti.ipv6.as6453.net (2405:2000:ffd0::1a) 232.07 ms 210.376 ms 163.293 ms 10 if-xe-3-1-1.10.tcore1.TV2-Tokyo.ipv6.as6453.net (2405:2000:ffb::22) 191.668 ms 200.129 ms 226.642 ms 11 if-ae2.2.tcore2.TV2-Tokyo.ipv6.as6453.net (2001:5a0:2200:300::2) 192.905 ms 191.953 ms 201.681 ms 12 if-ae6.2.tcore1.SVW-Singapore.ipv6.as6453.net (2405:2000:ffa0:100::49) 288.696 ms 279.671 ms 269.912 ms 13 if-ae11.2.thar1.SVQ-Singapore.ipv6.as6453.net (2405:2000:300:100::d) 264.938 ms 264.522 ms 266.912 ms 14 2405:2000:300:100::16 (2405:2000:300:100::16) 354.279 ms 349.7 ms 389.75 ms 15 ae2.bb02.sin1.tfbnw.net (2620:0:1cff:dead:beef::84c) 349.649 ms 348.205 ms 347.904 ms 16 ae0.bb01.hkg1.tfbnw.net (2620:0:1cff:dead:beef::1bdc) 349.552 ms 356.726 ms 348.74 ms 17 be6.bb01.pdx1.tfbnw.net (2620:0:1cff:dead:beef::5dd) 366.212 ms 368.99 ms 368.89 ms 18 be9.bb01.prn2.tfbnw.net (2620:0:1cff:dead:beef::f5) 365.81 ms 366.327 ms 366.598 ms 19 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 377.387 ms 365.613 ms 365.392 ms 23 * * * 24 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 387.156 ms !H 383.877 ms !H 374.672 ms !H
fugyk 12y ago
Facebook status[1] on this seems dry. Hoping to see a tech blog post soon on this issue. [1] https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226...
alexbate 12y ago
BBC[1] are reporting it was a "configuration problem", no hack [1] http://www.bbc.co.uk/news/technology-30996928 http://www.bbc.co.uk/news/technology-30996928
- alexgartrell 12y ago
  This should be at the top.
- danepowell 12y ago
  "Never attribute to malice that which is adequately explained by a configuration problem."
foofoobar 12y ago
"This was not the result of a third party attack but instead occurred after we introduced a change that affected our configuration systems. We moved quickly to fix the problem, and both services are back to 100% for everyone." - Facebook spokesperson [1] [1] https://twitter.com/blakeyblogs/status/559995252485140480 https://twitter.com/blakeyblogs/status/559995252485140480
sidcool 12y ago
WhatsApp wasn't affected. It still runs on its own infrastructure it seems. And this might be common knowledge, but not to me.
feroz1 12y ago
Still shows Facebook is pretty popular given the widespread news its outage has caused!
tempodox 12y ago
Funny how an outage of something as pointless as facebook generates this humongous comment count. But then, some source material for http://www.lamebook.com http://www.lamebook.com may have been lost...
- vacri 12y ago
  Speculating on outages at web juggernauts is squarely within the HN remit of business and web tech.
- neotek 12y ago
  I don't get how anyone could think that Facebook, a platform that has literally defined a paradigm shift in the way people communicate across the planet, could be genuinely considered pointless. It's okay not to like Facebook, and there are many good reasons to be sceptical about them as a company, but to call them pointless is so utterly myopic that it beggars belief.
  - tempodox 12y ago
    I was calling fb pointless from my own point of view. Obviously, lots of people disagree, otherwise it wouldn't have gotten that big. And I’m sure that every single user has their (valid) opinion on what “the point” of fb is for them. But as far as I'm concerned, I'll still call it pointless — no offence.
mironathetin 12y ago
"Facebook was down" So what?
pkorzeniewski 12y ago
That's why I can't comprehend companies using SaaS solutions like Slack or HipChat for something as crucial as internal communication, that requires a connection to an external server to work. What's even more scary is the data leakage risk - with DDoS you know when something bad happens, because the service simply doesn't work, but with data leakage how can you tell? Every single conversation stored on a 3rd party server. It's ridiculous. Yes, they're well protected, but it's also much more inviting for hackers - once you breach it, you gain a massive amount of sensitive data about thousands of companies... Well, it seems nice UI and 3rd app integrations are more important than security and reliability these days I guess.
- randomsearch 12y ago
  If you are, say, a startup with sensitive information being discussed in an IRC server that you run, do you really think your data would be more secure then? Keep in mind that you can't afford a security expert (or your own time) to admin your system if you're a small business. As for reliability, I've been using Slack for months without a single outage. If it went down for a day, the impact would be minimal -- I'd just use email and communicate less for that one day. Or I'd switch to another provider. Big deal.
  - pkorzeniewski 12y ago
    How about P2P connections, so nothing leave your company's internal network, because why should it? It's the most secure and reliable solution, but even running an IRC server on a local machine is so easy that you don't need an admin.
    - randomsearch 12y ago
      Easy, yes. But very secure? No.
- Hates_ 12y ago
  What about solutions like Dropbox or Gmail? Or even plain old email for that matter. Even if you look after those yourself, how can you be 100% sure that you can offer a service that is more secure and reliable than a third party?
rurban 12y ago
Looks more like a "SW update gown wrong" scenario. People reported problems a few days before already, empty timelines and such. They roll out incremently. Those kind of problems quickly affect the backends.
deleted 12y ago
[deleted]
AJNDL 12y ago
That'll teach you to not rely on FB to sign in to Spotify.
vayarajesh 12y ago
They are saying it was human error by facebook and hence the outage.. but how come other apps like hipchat and tinder were also down? can't be co-incidence.
siranen 12y ago
True fact is that in each country it was randomly shut down

Facebook was down

325 points by sgloutnikov 12y ago

ubersync 12y ago

I can't wait to know the reason behind this.

vladmiller 12y ago
Looks like cdn or part of their network routers failed, because mobile apps for instagram and facebook work fine

pmalynin 12y ago

Hmm, interestingly enough Faceboook today "has complied with a Turkish court order demanding the blocking of a page it said offended the Prophet Muhammad." [0] [0] http://www.bbc.com/news/technology-30982556 http://www.bbc.com/news/technology-30982556

deleted 12y ago
[deleted]

taubi19 12y ago

I wonder how many people are calling their isp call centers right now ;)

vladmiller 12y ago

Oh no, I just got used to React.js/Flux

Hengjie 12y ago

Meanwhile world's productivity goes up substantially.

Gigablah 12y ago
Not with everyone constantly refreshing

fapjacks 12y ago

The post mortem is going to be very interesting. It's Facebook, not like a Freenode server or something.

noobermin 12y ago

I've always thought how strange it is when people notice and make such a big deal about facebook being down or accessible. It's almost as if fb has become a public utility or something. As others have remarked, one way one can interpret this is it is a testament to their ops' exceptional ability as engineers that such downtime is so noticeable.

jhildings 12y ago
>It's almost as if fb has become a public utility or something. Well thats what Zuckerberg answered a user to the question "Why Facebook wasn't cool anymore". He didn't want it to be cool , but more like a digital utility like water or electricity
- bobbykjack 12y ago
  The trouble with that analogy is that water and electricity (just about) are absolutely essential to our day-to-day lives; a specific social media site isn't. Facebook needs to appeal to the trend to keep in business. Hugely popular social networking sites have failed in the past, precisely when they've fallen out if fashion.

mhr_online 12y ago

Confirmed, from Iran

zaporozhets 12y ago

Perhaps it's a good time to re-think our dependence on facebook for 'cheat' SSO.

sliverstorm 12y ago
... because they haven't reached five nines? Because your own privately hosted authentication system will?

doaluke 12y ago

yes

borgchick 12y ago

yup

munirusman 12y ago

yes

wallzb 12y ago

yes

foxhedgehog 12y ago

http://www.downforeveryoneorjustme.com/facebook.com http://www.downforeveryoneorjustme.com/facebook.com It's not just you! http://facebook.com http://facebook.com looks down from here.

kenrick95 12y ago
There's a shorter link http://isup.me/facebook.com http://isup.me/facebook.com
- gemma 12y ago
  It's about quality, not quantity. http://wheresitup.com/results/54c732d88c3dcf8416232c91 http://wheresitup.com/results/54c732d88c3dcf8416232c91

dgabriel 12y ago

No one has anything else to do tonight, I guess.

deleted 12y ago

[deleted]

webo 12y ago

It doesn't help to link to the website when it is already down.

jtreminio 12y ago
Traffic sent from HN is but a drop in a bucket the size of a dump truck for facebook.

evjan 12y ago

Yup

WillNotDownvote 12y ago

Instagram too, apparently. Are they sharing crucial infrastructure?

philip1209 12y ago
http://instagram-engineering.tumblr.com/post/89992572022/migrating-aws-fb http://instagram-engineering.tumblr.com/post/89992572022/mig...

evjan 12y ago

Instagram as well

bhaai 12y ago
Looks like they share some infrastructure with facebook. Whatsap web and app works though :)

hyperreality 12y ago

Yup. Down. http://www.downforeveryoneorjustme.com/facebook.com http://www.downforeveryoneorjustme.com/facebook.com

abm91 12y ago

Welp...that escalated quickly.

brunorsini 12y ago

yes. the fact that it feels so weird is testimony to their server eng team

daeken 12y ago
It really is. Facebook is far from bug-free, but I'm a near-constant user and this is the first time in a long time that I can recall it going down completely. I'll be really surprised if it lasts more than an hour, at most.

th0br0 12y ago

Even the fbcdn / akamai part seems to be down (an error occured while processing your request) ... funny

andreasen 12y ago

yes - Sydney, 5:59pm

vanadium 12y ago

According to Twitter, Facebook falling in the dead of night does in fact make (about a few million) sounds.

jedrek 12y ago
There is no dead of the night on a platform with users all over the world. Maybe when it's midday over the pacific, but not even then.
- vanadium 12y ago
  Let's suspend disbelief in the name of a stretch joke.

wsidell 12y ago

Instagram is also down

dnohr 12y ago

Yep, it's down here in SE Asia too

deleted 12y ago

[deleted]

umair-a 12y ago

yep, even ping fails. even though historically it use to work. https://bitneer/facebook.com/ping/ https://bitneer/facebook.com/ping/

SpaceInvader 12y ago
where's .com? ;-)
- umair-a 12y ago
  sorry about that :) https://bitneer.com/facebook.com/ping/ https://bitneer.com/facebook.com/ping/ working now of course.

flaie 12y ago

Akamai, the CDN used by Facebook went down before Facebook for me this morning: http://www.downforeveryoneorjustme.com/akamaihd.net http://www.downforeveryoneorjustme.com/akamaihd.net

lewi 12y ago
Seems to be effecting Instagram as well. I assume they both use Akamai then. Was using both when they became unresponsive, resetting routers all over the shop.
clarkm 12y ago
Looking through the customer list, most other sites seem to be working fine: http://www.akamai.com/html/customers/customer_list.html http://www.akamai.com/html/customers/customer_list.html I wonder what makes FB and Instagram different?
- flaie 12y ago
  It was specifically fbstatic-b-a.akamaihd.net which was also not responsive, preventing css & js to load rendering a completely broken page on my browser.
- fivedogit 12y ago
  Akamai has several different networks, one for streaming media, one for regular http, and one for https. They also have an S3-like storage system and a DNS service. Some customers are configured with several of these offerings at once, so it's hard to know which, if any, experienced problems and led to the outages. Honestly, I doubt it was an Akamai issue. If Akamai experienced network problems, dozens or hundreds of sites would be affected. If an Akamai config issue (ie human error) were to blame, then it would probably only affect one site, not several. Neither FB nor Akamai is dumb enough to push multiple site changes at once.

vaidik 12y ago

https://www.dropbox.com/s/zueam1oq3sdzv36/Screenshot%202015-01-27%2011.54.49.png?dl=0 https://www.dropbox.com/s/zueam1oq3sdzv36/Screenshot%202015-...

clarkm 12y ago
Charge your battery!

greenvaio 12y ago

DOWN for me too.

dnohr 12y ago

Yes, it's also down in SE Asia

greenvaio 12y ago

Lizard squard???

sidcool 12y ago

It's down for me as well (11:50 AM IST) in Mumbai, India.

deleted 12y ago

[deleted]

djacobs 12y ago

I restarted my modem and router two times before I realized that the network was fine and that it was just Facebook that was down. Props to their ops team for uptime that is so reliable.

RandallBrown 12y ago
I haven't seen Facebook go down in years. I wonder if this is going to be a big deal.
- jmtame 12y ago
  I'm wondering how much revenue gets lost every minute that Facebook is down. It has to be in the tens of thousands. Edit: someone wrote this below in the comments, which came to $400 per second of downtime. Ouch.
  - sliverstorm 12y ago
    $400 per second somehow seems... smaller than I would have thought. Their opex on a per second basis has got to be orders of magnitude higher than that.
    - benhamner 12y ago
      $400 per second is about $13 billion per year
  - semi-extrinsic 12y ago
    For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion. (source for cost: Cormorant Alpha shutdown in 2013) Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.
    - 12y ago

deleted 12y ago

[deleted]

keyle 12y ago

This post might as well be a poll...

leowinterde 12y ago

yes

zht 12y ago

wonder if this has to do with the storm near IAD?

munirusman 12y ago

How much revenue are they losing every second?

gburt 12y ago
$7.87 billion in 2013 / 365 days / 24 hours / 60 minutes / 60 seconds is about $250 a second. Edit: using more up to date data: $3,300,000,000 in the 3rd quarter of 2014 / (365/4) / 24 / 60 / 60 = $418.58
- munirusman 12y ago
  $1.5M an hour!!! But considering this is not the peak time in North America, it's should be little less.
  - fnayr 12y ago
    Yeah but I wonder if actually more people will net log in because everyone is talking about it being down.
  - together_us 12y ago
    But here in India and the rest of Asia, people wake up to Facebook. You can't ignore that.
    - q_no 12y ago
      In Asia Facebook isn't that big, they also have other social services
  - berkes 12y ago
    A site like Facebook has a different profile though. More like email. If you cannot reach your mail, you'll read it at a later point. When facebook comes up, people will catch up on "lost facebook time" most likely. Over the course of the week (or even day) it will most likely even out.

deleted 12y ago

[deleted]

phragg 12y ago

what's facebook?

willidiots 12y ago

Let's all have a moment of contemplation & relaxation on behalf of those poor souls on-call...

joshmn 12y ago
Our Facebook, who art in Datacenter...
- rxaxm 12y ago
  hallowed be thy threads
  - JacobJans 12y ago
    thy pingdom come
    - davesque 12y ago
      thy browsing at work be done
      - joshmn 12y ago
        on desktop as it is on mobile.
        bcantrill 12y ago
        Give us this day our daily selfies
        aaronsnoswell 12y ago
        And forgive us our sheep-throwing.
        marincounty 12y ago

goran216 12y ago

Yeah

goran216 12y ago

yeah

DigitalSea 12y ago

Yep. Down for me in Queensland, Australia as well (Facebook and Instagram). It is situations like this that give people a much needed break from social media (myself included). You don't realise just how much of a stranglehold (some more than others) that it has over you until you can't actually use it. Downtime for a site as big as Facebook is rare, but it happens, especially when your infrastructure isn't solely controlled by you (content distribution networks come to mind).

deleted 12y ago
[deleted]

Colargol 12y ago

yes (Central Europe)

pavelow 12y ago

Affirmative

milankragujevic 12y ago

Yep down for me too from Serbia.

Yuioup 12y ago

Something to do with the blizzard?

austinl 12y ago

It's interesting to see how Facebook downtime has a huge effect on Internet traffic – there's a Guardian report where they share their referral data while Facebook was down for 31 minutes (the longest outage in 4 years): http://www.theguardian.com/technology/2014/jun/19/facebook-unavailable-longest-outage-four-years http://www.theguardian.com/technology/2014/jun/19/facebook-u...

mhomde 12y ago

Somewhere in the world Jesse Eisenberg is throwing an epic fit

deleted 12y ago

[deleted]

vayarajesh 12y ago

Even instagram seems down

grejdi 12y ago
Makes sense. They share the same data centers. See: http://instagram-engineering.tumblr.com/post/89992572022/migrating-aws-fb http://instagram-engineering.tumblr.com/post/89992572022/mig...

datboitom 12y ago

Yes

gbachik 12y ago

OH MY GOD

vayarajesh 12y ago

with malaisian airlines getting hacked its making me think what if the ISIS got it to facebook?

timpattinson 12y ago
Very unlikely. The attack on Malaysia Airlines was a hijacking of the DNS, not an actual hack on the website. Plus with the reosurces fb dedicates to security, they have a much better chance of finding 0days than some terrorists.

arrowgunz 12y ago

It's down from here - Bay Area, California.

ujuc 12y ago

yes, @korea

ujuc 12y ago

yes @korea

gregthompsonjr 12y ago

Bet it has something to do with this damn snowstorm. I want this season to be over with!

ptasci67 12y ago
But http://aws.amazon.com/ http://aws.amazon.com/ and http://azure.microsoft.com/en-us/ http://azure.microsoft.com/en-us/ seem to be up so some devops work needs to be done here!

EtienneK 12y ago

South African here - Down for me too.

babo 12y ago

down in Norway

pirate21 12y ago

Here in Brazil is Down,Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:

pirate21 12y ago
http://check-host.net/check-http?host=facebook.com http://check-host.net/check-http?host=facebook.com

pirate21 12y ago

here in Brazil, Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:

ved_a 12y ago

Yes (from India)

bigphishy 12y ago

And with it millions of widgets and like buttons suddenly vanish from their webpages. Good riddance I say!

unimaxlin 12y ago

Its down in India

unimaxlin 12y ago

ISIS hacked into facebook network ?

dudus 12y ago
are you joking or do you actually have any basis for this suspicion?

t3ra 12y ago

Lizard squad associated account just tweeted this https://twitter.com/LizardMafia/status/559963134006292481 https://twitter.com/LizardMafia/status/559963134006292481

jaysonelliot 12y ago

I feel strangely free. Perhaps this is a sign that I should go back to blocking myself from Facebook with StayFocusd. Kind of sad that I needed this kick in the backside, but I'll take this as an opportunity to reclaim empty calories and wasted time.

saraid216 12y ago
So of course you're on Hacker News.
code_duck 12y ago
Might as well take a first step to ignoring facebook. I've spent more than my fair share of time on the site in the past few years. While it's been productive overall in terms of professional and personal networking, I could have achieved as much with 10% of the time if I focused solely on the productive aspects. Two weeks ago I decided I could do without endless pointless conversations with vague colleagues, being updated on largely irrelevant minute details and thoughts of people I don't actually know, and and endless stream of low quality news links. I deactivated my account for a week and then reactivated it but only visit once a day at most. My quality of life certainly has not decreased!

deleted 12y ago

[deleted]

MalcolmDiggs 12y ago

Quick, Google+ Engineers: take it down too! You don't want to fall behind.

prostoalex 12y ago
Gmail is 502'ing for me.
- fuzzywalrus 12y ago
  Gmail seems to be in working order
lmm 12y ago
Google+ was actually down as well, but fortunately neither of their users tried to log in during the outage.

bombtrack 12y ago

Obviously anyone can "take credit" for things like this, but if this[1] is true, then jesus. Edit: Since it's topical, I enjoy listening to Chuck Rossi in interviews or presentations. Releng 2014 - Keynote 1: Chuck Rossi, Release Engineering, Facebook Inc. | Talks at Google [2] [1] https://twitter.com/lizardmafia/status/559963134006292481 https://twitter.com/lizardmafia/status/559963134006292481 [2] http://youtu.be/Nffzkkdq7GM?t=4m39s http://youtu.be/Nffzkkdq7GM?t=4m39s

danso 12y ago
A few years ago, who would've guessed that Twitter would be the major service still standing during such a mass attack?
- deleted 12y ago
  [deleted]
  - deleted 12y ago
    [deleted]
- SEJeff 12y ago
  Before they had Mesos, they wouldn't have.
thrownaway2424 12y ago
They had to leave twitter up so they could have somewhere to post?
- Hexcles 12y ago
  lol. Might simply because people see white whales too often and that's not a surprise.
joshhagler 12y ago
I am not familiar with Lizard Mafia, do they have a history of famous hacks?

deleted 12y ago

[deleted]

quillo 12y ago

Nanog is asleep, but Ausnog might shed some light on it: http://lists.ausnog.net/pipermail/ausnog/2015-January/029463.html http://lists.ausnog.net/pipermail/ausnog/2015-January/029463...

D_Guidi 12y ago

yep

typon 12y ago

Is the API down as well? I imagine thousands of websites and apps rely on Facebook being up.

kevan 12y ago
The main app and Messenger aren't connecting at all for me, Facebook's OAuth is also inaccessible.
webo 12y ago
Yes, can't use log-in with Facebook.
markthethomas 12y ago
Seems to be --> https://developers.facebook.com/status/ https://developers.facebook.com/status/

jacko0 12y ago

Yea, it could be Lizard Squad. https://twitter.com/LizardMafia https://twitter.com/LizardMafia. Facebook, Instagram, Tinder, AIM, Hipchat #offline #LizardSquad

yaeger 12y ago
Or it could be a power issue brought on by this little bigger than average blizzard that will hit the Boston area the hardest where, incidentally, Akamai is located. But sure, couple of script kiddies might be responsible as well. It's usually one or the other.

ohmygeek 12y ago

Instagram is down as well!

stevewilhelm 12y ago

Over the last fifteen years, when ever I have experienced intermittent Web access, I try www.cisco.com. It has never been down. Ever.

anant90 12y ago

http://youtu.be/qpNNa_Mokok?t=10s http://youtu.be/qpNNa_Mokok?t=10s

arasmussen 12y ago

Back up for me, showing an error page now though.

arunabh010 12y ago

down in Singapore

TheAceOfHearts 12y ago

Seems Facebook isn't very webscale!

mirceagoia 12y ago

Down for me too.

overworkedasian 12y ago

This doesnt look right: whois facebook.com @8.8.8.8 Whois Server Version 2.0 Domain names in the .com and .net domains can now be registered with many different competing registrars. Go to http://www.internic.net http://www.internic.net for detailed information. FACEBOOK.COM.DISABLE.YOUR.TIMELINE.NOW.WITH.THE.ORIGINAL.TIMELINE-REMOVE.NET FACEBOOK.COM.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM FACEBOOK.COM.LOVED.BY.WWW.SHQIPHOST.COM FACEBOOK.COM.MORE.INFO.AT.WWW.BEYONDWHOIS.COM FACEBOOK.COM.ZZZZZ.GET.LAID.AT.WWW.SWINGINGCOMMUNITY.COM FACEBOOK.COM To single out one record, look it up with "xxx", where xxx is one of the records displayed above. If the records are the same, look them up with "=xxx" to receive a full display for each record. >>> Last update of whois database: Tue, 27 Jan 2015 06:56:04 GMT <<<

zaporozhets 12y ago
"Facebook finally figures out that sex /still/ sells better than advertising"
anthony_franco 12y ago
It's just the way whois lookups work. Do a whois lookup of any big site (google.com, microsoft.com, etc) and you'll get a similar response.
jeffreyrogers 12y ago
It's been that way for a while now and happens with any large site. Try google.com for example and you'll see the same thing.
markthethomas 12y ago
I've run WHOIS queries on FB before and gotten the same(ish) thing more than a few times -- assumed it was temporary tampering before
axaxs 12y ago
Appears to return hosts with name starting with search term, which doesn't make a lot of sense but has likely always been the case. Also @ flag is a dig thing, not whois.
12y ago

nurettin 12y ago

down here (in Turkey) and surprisingly, it isn't the government this time!

deleted 12y ago

[deleted]

toephu2 12y ago

instagram and facebook down as of this posting

phpfour 12y ago

Down from Bangladesh !

Hexcles 12y ago

Facebook did go down before, but not like this time. I tried to ping many IPs of Facebook, all even failed to reach. Can be a sign of DDoS. Anyway, if that is true, people have to take it really seriously this time.

thangh9 12y ago

FYI https://twitter.com/hashtag/facebookdown?src=tren https://twitter.com/hashtag/facebookdown?src=tren

logotype 12y ago

The only subdomain working is: http://newsroom.fb.com http://newsroom.fb.com and hosted by wordpress.com

justindocanto 12y ago

All the 'is it down' websites are slowly going down with people searching for it.

alecsx6 12y ago

It's down in Eastern Europe too.

deleted 12y ago

[deleted]

rdlecler1 12y ago

One of our vendors crashed which hung out site. Damn counterparty risk!

alexivanovs 12y ago

If they guy is posting pics of himself on Twitter asking others to "dox him", it could mean that he's pretty emotionally stable not to fall for the common pitfalls when it comes to online hacking. In that sense, it's very possible that he has access to some major database dumps that the public is unaware of, and given that he is also claiming the glory behind the Malaysia Airlines hack; it's clear that he does more than DDoS, no matter what the more educated hackers are saying. A long way to go for a secure internet, that's for sure.

rbosinger 12y ago

I blame global warming. Them servers were too hot I betcha.

joshhagler 12y ago

Just tweeted "@LizardMafia: More to come soon. Side note: We're still organizing the @MAS email dump, stay tuned for that."

ookblah 12y ago

one of the comments in thenextweb article is going nuts on the upvote/like

egil 12y ago

Up again here from Norway.

fredkelly 12y ago

It's back!

sekasi 12y ago

Back up now. Total downtime around 55 minutes that I could see.

philip1209 12y ago

Expensify is having issues too: http://status.expensify.com/ http://status.expensify.com/ I feel like this is more the norm rather than an exception for them at this point, though.

philip1209 12y ago

Facebook back online in SF as of 7:09 GMT

jpgvm 12y ago

Back up now. Will be good to read the postmortem on this one.

andydev 12y ago

Facebook seems to be back online for me.

avnerner 12y ago

Seems facebook is back. a total of 55 min or so.

vaidik 12y ago

Seems like they are back up now. 100% HA is a myth, for even those who can afford it.

tannerc 12y ago

As of 11:12 PM PST Facebook and Instagram are back online.

rbosinger 12y ago

Ok they're back for me in Alberta! What a wild ride. I pooped everywhere.

crgt 12y ago

Back up for me again

markthethomas 12y ago

I can get back on; able to see posts from hours ago, noting within 5 hours-ish. Anyone with an ops background able to give a JS eng who has no ops experience like myself a glimpse into what bringing services back online like that looks like? No experience with ddos mitigation techniques

deleted 12y ago

[deleted]

Shrugs 12y ago

Looks like a classic Stack Snowverflow error to me.

ioseph 12y ago

Annd it's back up again

plumeria 12y ago

Down here (Central America)

sandworm 12y ago

I mean, like, WOW. If they have done this we all must give them props for ambition. Forget Sony. Nobody outside of hollywood bigwigs were impacted by that. But Facebook? The US government is going to go nuts. I hope lizardsquad are driving drone-proof cars. Facebook is now back online for me on my home connection (BC Canada) and every Tor node I try. Either the attack is over or FB has fought back.

ankit-mehta 12y ago

It's Up and Running. All the websites are accessible now.

thienchu 12y ago

Looks like they're back online

deleted 12y ago

[deleted]

bigstueyc 12y ago

back up

sandworm 12y ago

Update from Facebook: "Current State: Fix Pushed Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226... Might not be a hack! Might just be weather in Boston: "Akamai (provider for FB, Instagram and so on) claims to be down due to power outage. #LizardSquad claims to have hacked them. Get popcorn." https://twitter.com/petterkarlsson/status/559975340526014464 https://twitter.com/petterkarlsson/status/559975340526014464

pskittle 12y ago

It's working for me

deleted 12y ago

[deleted]

binoyxj 12y ago

Official update: "Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226...

kevan 12y ago
"The issue was resolved at 23:10 PST and the site stabilized shortly afterwards. Our internal and external monitoring shows that API requests are being served with normal latency and error rates, in all geographic regions. We are sorry for any inconvenience this may have caused you and the users of your apps." No hints there, guess we'll just have to wait for a full post-mortem to come out.
- deleted 12y ago
  [deleted]
bombtrack 12y ago
"This was not the result of a third-party attack but instead occurred after we introduced a change that affected our configuration systems." http://www.bbc.com/news/technology-30996928 http://www.bbc.com/news/technology-30996928

niubility 12y ago

Was completely dead in Australia. Down on both my local ISP and VPN to the National Research network. It was not a DNS issue as the IP Address is the same as it was during the outage. "Host unreachable", maybe they took out the load-balancer? - traceroute to star.c10r.facebook.com (2a03:2880:f00c:6:face:b00c:0:2) 1 redacted 2 ge-1-0-0.bb1.a.syd.aarnet.net.au (2001:388:1:5001::1) 107.649 ms 91.569 ms 91.814 ms 3 ae9.pe2.brwy.nsw.aarnet.net.au (2001:388:1:88::1) 91.981 ms 92.822 ms 103.957 ms 4 ae5.pe1.brwy.nsw.aarnet.net.au (2001:388:1:87::1) 92.447 ms 93.29 ms 101.182 ms 5 et-1-1-0.pe1.rsby.nsw.aarnet.net.au (2001:388:1:66::1) 94.606 ms 109.654 ms 91.987 ms 6 et-0-3-0.nsw-msct-bdr1.aarnet.net.au (2001:388:1:a3::2) 96.119 ms 92.032 ms 92.57 ms 7 6453.syd.equinix.com (2001:de8:6::6453:1) 99.204 ms 111.52 ms 190.499 ms 8 if-xe-0-3-1.3.thar1.1MH-Sydney.ipv6.as6453.net (2405:2000:ffd0::a) 90.392 ms 90.862 ms 92.174 ms 9 if-3-0-0.2.core1.PV4-Piti.ipv6.as6453.net (2405:2000:ffd0::1a) 232.07 ms 210.376 ms 163.293 ms 10 if-xe-3-1-1.10.tcore1.TV2-Tokyo.ipv6.as6453.net (2405:2000:ffb::22) 191.668 ms 200.129 ms 226.642 ms 11 if-ae2.2.tcore2.TV2-Tokyo.ipv6.as6453.net (2001:5a0:2200:300::2) 192.905 ms 191.953 ms 201.681 ms 12 if-ae6.2.tcore1.SVW-Singapore.ipv6.as6453.net (2405:2000:ffa0:100::49) 288.696 ms 279.671 ms 269.912 ms 13 if-ae11.2.thar1.SVQ-Singapore.ipv6.as6453.net (2405:2000:300:100::d) 264.938 ms 264.522 ms 266.912 ms 14 2405:2000:300:100::16 (2405:2000:300:100::16) 354.279 ms 349.7 ms 389.75 ms 15 ae2.bb02.sin1.tfbnw.net (2620:0:1cff:dead:beef::84c) 349.649 ms 348.205 ms 347.904 ms 16 ae0.bb01.hkg1.tfbnw.net (2620:0:1cff:dead:beef::1bdc) 349.552 ms 356.726 ms 348.74 ms 17 be6.bb01.pdx1.tfbnw.net (2620:0:1cff:dead:beef::5dd) 366.212 ms 368.99 ms 368.89 ms 18 be9.bb01.prn2.tfbnw.net (2620:0:1cff:dead:beef::f5) 365.81 ms 366.327 ms 366.598 ms 19 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 377.387 ms 365.613 ms 365.392 ms 23 * * * 24 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 387.156 ms !H 383.877 ms !H 374.672 ms !H

fugyk 12y ago

Facebook status[1] on this seems dry. Hoping to see a tech blog post soon on this issue. [1] https://developers.facebook.com/status/issues/393998364112264/ https://developers.facebook.com/status/issues/39399836411226...

alexbate 12y ago

BBC[1] are reporting it was a "configuration problem", no hack [1] http://www.bbc.co.uk/news/technology-30996928 http://www.bbc.co.uk/news/technology-30996928

alexgartrell 12y ago
This should be at the top.
danepowell 12y ago
"Never attribute to malice that which is adequately explained by a configuration problem."

foofoobar 12y ago

"This was not the result of a third party attack but instead occurred after we introduced a change that affected our configuration systems. We moved quickly to fix the problem, and both services are back to 100% for everyone." - Facebook spokesperson [1] [1] https://twitter.com/blakeyblogs/status/559995252485140480 https://twitter.com/blakeyblogs/status/559995252485140480

sidcool 12y ago

WhatsApp wasn't affected. It still runs on its own infrastructure it seems. And this might be common knowledge, but not to me.

feroz1 12y ago

Still shows Facebook is pretty popular given the widespread news its outage has caused!

tempodox 12y ago

Funny how an outage of something as pointless as facebook generates this humongous comment count. But then, some source material for http://www.lamebook.com http://www.lamebook.com may have been lost...

vacri 12y ago
Speculating on outages at web juggernauts is squarely within the HN remit of business and web tech.
neotek 12y ago
I don't get how anyone could think that Facebook, a platform that has literally defined a paradigm shift in the way people communicate across the planet, could be genuinely considered pointless. It's okay not to like Facebook, and there are many good reasons to be sceptical about them as a company, but to call them pointless is so utterly myopic that it beggars belief.
- tempodox 12y ago
  I was calling fb pointless from my own point of view. Obviously, lots of people disagree, otherwise it wouldn't have gotten that big. And I’m sure that every single user has their (valid) opinion on what “the point” of fb is for them. But as far as I'm concerned, I'll still call it pointless — no offence.

mironathetin 12y ago

"Facebook was down" So what?

pkorzeniewski 12y ago

That's why I can't comprehend companies using SaaS solutions like Slack or HipChat for something as crucial as internal communication, that requires a connection to an external server to work. What's even more scary is the data leakage risk - with DDoS you know when something bad happens, because the service simply doesn't work, but with data leakage how can you tell? Every single conversation stored on a 3rd party server. It's ridiculous. Yes, they're well protected, but it's also much more inviting for hackers - once you breach it, you gain a massive amount of sensitive data about thousands of companies... Well, it seems nice UI and 3rd app integrations are more important than security and reliability these days I guess.

randomsearch 12y ago
If you are, say, a startup with sensitive information being discussed in an IRC server that you run, do you really think your data would be more secure then? Keep in mind that you can't afford a security expert (or your own time) to admin your system if you're a small business. As for reliability, I've been using Slack for months without a single outage. If it went down for a day, the impact would be minimal -- I'd just use email and communicate less for that one day. Or I'd switch to another provider. Big deal.
- pkorzeniewski 12y ago
  How about P2P connections, so nothing leave your company's internal network, because why should it? It's the most secure and reliable solution, but even running an IRC server on a local machine is so easy that you don't need an admin.
  - randomsearch 12y ago
    Easy, yes. But very secure? No.
Hates_ 12y ago
What about solutions like Dropbox or Gmail? Or even plain old email for that matter. Even if you look after those yourself, how can you be 100% sure that you can offer a service that is more secure and reliable than a third party?

rurban 12y ago

Looks more like a "SW update gown wrong" scenario. People reported problems a few days before already, empty timelines and such. They roll out incremently. Those kind of problems quickly affect the backends.

deleted 12y ago

[deleted]

AJNDL 12y ago

That'll teach you to not rely on FB to sign in to Spotify.

vayarajesh 12y ago

They are saying it was human error by facebook and hence the outage.. but how come other apps like hipchat and tinder were also down? can't be co-incidence.

siranen 12y ago

True fact is that in each country it was randomly shut down