13 ms·

Facebook, Instagram and Messenger were down

101 points by mtsx 7y ago

dvduval 7y ago
Down here in LA. Happy Thanksgiving.
- thekyle 7y ago
  Down in Orlando, Happy Thanksgiving to you too.
- xenospn 7y ago
  Both work for me here in LA!
samwillis 7y ago
There is a major outage they are aware of effecting Facebook Login, Instagram, Messenger and the Marketing API: https://developers.facebook.com/status/issues/2287293591515186/ https://developers.facebook.com/status/issues/22872935915151... https://developers.facebook.com/support/bugs/555889181863481/ https://developers.facebook.com/support/bugs/555889181863481... Edit: I think there is a bug with the status page, the outage hasn't been going for the 18 hours it is showing me here: https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/
- pmlnr 7y ago
  "you must log in to continue" Hang on... to check the status of login you need to login? What a wonderful circular dependency that is!
thekyle 7y ago
I wonder if WhatsApp is also affected. I know they've been trying to merge its infrastructure with that of Messenger.
- olcor 7y ago
  Nope. Working fine in India. Edit: Even FB is fine here.
  - wprapido 7y ago
    Thailand: FB, Messenger didn't work till a wee while ago. No WhatsApp service interruption though.
- ksec 7y ago
  WhatsApp not affected.The power of FreeBSD and Erlang. ( Not sure if they are still on that stack )
  - notacoward 7y ago
    Nope, nothing to do with FreeBSD or Erlang. Can't say what it is, but there are hundreds of other differences between which pieces of FB infrastructure WhatsApp uses vs. what other products use. This is in one of those hundreds. ;)
    - rescbr 7y ago
      What always amazed me was the scale of this difference not being larger. Really fine engineering. ;)
drcongo 7y ago
Perfect opportunity for everyone to wean themselves off this cesspool.
- dang 7y ago
  Maybe so, but please don't post unsubstantive comments to Hacker News.
celticninja 7y ago
Good
- dang 7y ago
  Could you please stop posting unsubstantive comments to Hacker News?
djanogo 7y ago
Move fast and break things during major holiday sales. I wonder if some new holiday sales feature that was supposed to go live caused outage.
- Benjammer 7y ago
  Move fast and break stable infra
bonniemuffin 7y ago
About 2 days ago, I noticed that Facebook added multi-level reply threads for a small percent of posters, but as of this morning it seemed totally broken for me on the latest ios build. Seems unlikely to be related, but I wonder if it's possible they broke the site trying to hotfix it. Regardless of the root cause, I'm feeling for the oncall folks whose Thanksgivings are getting ruined right about now.
- ceejayoz 7y ago
  I work pretty heavily with the Facebook Graph API and things have been pretty messed up ever since Cambridge Analytica and the scramble afterwards. I've got a ticket from August about a basic piece of functionality - deleting a post via its ID - that's completely broken, and all I get is an every-few-weeks "should be fixed", "nope, it's clearly not", "ok stand by" exchange with them. Basic stuff like "show me the user's Facebook pages" breaks for everyone for hours at a time. When we attended F8 they talked heavily about their intensive automated testing, but I don't see that much evidence of it. I suspect a lot of things just got turned off in the mad "shut down EVERYTHING" scramble and they're still working out the impacts.
smaili 7y ago
Is it just a crazy coincidence this happened right after Trump’s signing of the new Hong Kong bill?
- maest 7y ago
  probably
- ryanmercer 7y ago
  Given most of America, except those unlucky people like me that are at work for their 14th Thanksgiving in a row, are hanging out with family not starting at the internet either watching parades or football instead of tracking political happenings that the vast majority of Americans are clueless about because we have no realistic way as an individual to do anything about even stuff at a town/county level, I'm guessing it has nothing to do with that whatsoever.
chanw 7y ago
https://status.fb.com/ads/ https://status.fb.com/ads/
hprotagonist 7y ago
Ahh. happy thanksgiving!
throwaway40324 7y ago
It must be just a coincidence that PHP 7.4 was released today, and surely FB doesnt rely on PHP version updates to latest, right? Just speculating.. Edit: that twitter thread is quite entertaining
- cranekam 7y ago
  Facebook doesn't run vanilla PHP. It runs on Hack, a language developed from PHP that runs on HHVM, a JIT VM it developed: https://engineering.fb.com/developer-tools/hack-a-new-programming-language-for-hhvm/ https://engineering.fb.com/developer-tools/hack-a-new-progra...
  - throwaway40324 7y ago
    Right, but up until after Hackv3.3, PHP code was supported, which support ended 9 days ago. "HHVM v3.30 will be the last release series where HHVM aims to support PHP. The key dates are... 2019-11-19: expected end of support for v3.30" https://hhvm.com/blog/2018/09/12/end-of-php-support-future-of-hack.html https://hhvm.com/blog/2018/09/12/end-of-php-support-future-o... Note: I really dont believe my above speculation is related, just thought it was an interesting coincidence, at best.
rinchik 7y ago
the scale FB operates on is staggering! Human factor, while still present, is very low. Wonder what could have happened there. They don't usually release postmortems, right? (since the general service they provide to the masses is "free", meaning minimum obligating to reporting. Would be curious to read the postmortem, analysis of what happened.
jessaustin 7y ago
I guess GitLab are no longer the most notable outage of the day...
g051051 7y ago
If I hadn't been looking here, I might never have known. Yes, I'm bragging.
rinchik 7y ago
Also curious if FB outages affect global internet traffic? Are there any relative stats? (E.g. if google to go down, it would have an economic cost comparable to a natural disaster)
- pdpi 7y ago
  There's several anecdotes internally at Facebook that get told to new starters about some instances of when facebook.com was down. One involves masses of people calling the police to report that facebook is offline (wat, why?), another involves so many people flocking to twitter to complain that fb is down that that crowd then knocked twitter offline ("How do you know if facebook is down? Check whether twitter is online" sort of thing). I don't feel comfortable sharing the exact figures, but during my time there the cost of a few minutes of downtime on an ads-adjacent message queue cluster directly turned into millions in lost revenue, so it's safe to say that losing the capability to _serve_ ads (as opposed to charging for serving them) would have a pretty damn high economic cost to those advertisers. As Workplace gains traction, downtime there will creep into the "cataclysmic" side of the economic costs scale.
  - jfim 7y ago
    Did you count the lost revenue as revenue per minute times the number of minutes the site was unavailable? If so, that's actually incorrect when calculating revenue loss. There will be actually be an uptick in traffic after the outage, either through people postponing their intended visit or checking if the site is down for them after it comes back. I know that at LinkedIn we broke part of the advertisement campaign creation flow from downtime in our service for a short time, and the impact was negligible since ads were still being served, and new ad creation would be just be retried later by advertisers. Now if you just lost the capacity to serve ads, there might be an impact, but since a lot of advertisements are display ads (to raise brand awareness over an immediate call to action), it's not clear whether or not it's a simple minutes down times impact per minute calculation.
    - jacquesm 7y ago
      Facebook being down is only lost revenue for Facebook. Every other company on the planet just gained in terms of productivity and revenues.
chadlavi 7y ago
It's a thanksgiving miracle
rossdavidh 7y ago
Man, what a wasted opportunity. If it had happened on a workday, we could have seen a massive productivity surge to boost the economy.
- AznHisoka 7y ago
  At least families will be forced to talk to each other instead of being on FB...
100016054749004 7y ago
https://m.facebook.com/profile.php?id=100016054749004&refsrc=https%3A%2F%2Fm.facebook.com%2Fprofile.php&_rdr https://m.facebook.com/profile.php?id=100016054749004&refsrc...
100016054749004 7y ago
100016054749004

101 points by mtsx 7y ago

dvduval 7y ago

Down here in LA. Happy Thanksgiving.

thekyle 7y ago
Down in Orlando, Happy Thanksgiving to you too.
xenospn 7y ago
Both work for me here in LA!

samwillis 7y ago

There is a major outage they are aware of effecting Facebook Login, Instagram, Messenger and the Marketing API: https://developers.facebook.com/status/issues/2287293591515186/ https://developers.facebook.com/status/issues/22872935915151... https://developers.facebook.com/support/bugs/555889181863481/ https://developers.facebook.com/support/bugs/555889181863481... Edit: I think there is a bug with the status page, the outage hasn't been going for the 18 hours it is showing me here: https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/

pmlnr 7y ago
"you must log in to continue" Hang on... to check the status of login you need to login? What a wonderful circular dependency that is!

thekyle 7y ago

I wonder if WhatsApp is also affected. I know they've been trying to merge its infrastructure with that of Messenger.

olcor 7y ago
Nope. Working fine in India. Edit: Even FB is fine here.
- wprapido 7y ago
  Thailand: FB, Messenger didn't work till a wee while ago. No WhatsApp service interruption though.
ksec 7y ago
WhatsApp not affected.The power of FreeBSD and Erlang. ( Not sure if they are still on that stack )
- notacoward 7y ago
  Nope, nothing to do with FreeBSD or Erlang. Can't say what it is, but there are hundreds of other differences between which pieces of FB infrastructure WhatsApp uses vs. what other products use. This is in one of those hundreds. ;)
  - rescbr 7y ago
    What always amazed me was the scale of this difference not being larger. Really fine engineering. ;)

drcongo 7y ago

Perfect opportunity for everyone to wean themselves off this cesspool.

dang 7y ago
Maybe so, but please don't post unsubstantive comments to Hacker News.

celticninja 7y ago

Good

dang 7y ago
Could you please stop posting unsubstantive comments to Hacker News?

djanogo 7y ago

Move fast and break things during major holiday sales. I wonder if some new holiday sales feature that was supposed to go live caused outage.

Benjammer 7y ago
Move fast and break stable infra

bonniemuffin 7y ago

About 2 days ago, I noticed that Facebook added multi-level reply threads for a small percent of posters, but as of this morning it seemed totally broken for me on the latest ios build. Seems unlikely to be related, but I wonder if it's possible they broke the site trying to hotfix it. Regardless of the root cause, I'm feeling for the oncall folks whose Thanksgivings are getting ruined right about now.

ceejayoz 7y ago
I work pretty heavily with the Facebook Graph API and things have been pretty messed up ever since Cambridge Analytica and the scramble afterwards. I've got a ticket from August about a basic piece of functionality - deleting a post via its ID - that's completely broken, and all I get is an every-few-weeks "should be fixed", "nope, it's clearly not", "ok stand by" exchange with them. Basic stuff like "show me the user's Facebook pages" breaks for everyone for hours at a time. When we attended F8 they talked heavily about their intensive automated testing, but I don't see that much evidence of it. I suspect a lot of things just got turned off in the mad "shut down EVERYTHING" scramble and they're still working out the impacts.

smaili 7y ago

Is it just a crazy coincidence this happened right after Trump’s signing of the new Hong Kong bill?

maest 7y ago
probably
ryanmercer 7y ago
Given most of America, except those unlucky people like me that are at work for their 14th Thanksgiving in a row, are hanging out with family not starting at the internet either watching parades or football instead of tracking political happenings that the vast majority of Americans are clueless about because we have no realistic way as an individual to do anything about even stuff at a town/county level, I'm guessing it has nothing to do with that whatsoever.

chanw 7y ago

https://status.fb.com/ads/ https://status.fb.com/ads/

hprotagonist 7y ago

Ahh. happy thanksgiving!

throwaway40324 7y ago

It must be just a coincidence that PHP 7.4 was released today, and surely FB doesnt rely on PHP version updates to latest, right? Just speculating.. Edit: that twitter thread is quite entertaining

cranekam 7y ago
Facebook doesn't run vanilla PHP. It runs on Hack, a language developed from PHP that runs on HHVM, a JIT VM it developed: https://engineering.fb.com/developer-tools/hack-a-new-programming-language-for-hhvm/ https://engineering.fb.com/developer-tools/hack-a-new-progra...
- throwaway40324 7y ago
  Right, but up until after Hackv3.3, PHP code was supported, which support ended 9 days ago. "HHVM v3.30 will be the last release series where HHVM aims to support PHP. The key dates are... 2019-11-19: expected end of support for v3.30" https://hhvm.com/blog/2018/09/12/end-of-php-support-future-of-hack.html https://hhvm.com/blog/2018/09/12/end-of-php-support-future-o... Note: I really dont believe my above speculation is related, just thought it was an interesting coincidence, at best.

rinchik 7y ago

the scale FB operates on is staggering! Human factor, while still present, is very low. Wonder what could have happened there. They don't usually release postmortems, right? (since the general service they provide to the masses is "free", meaning minimum obligating to reporting. Would be curious to read the postmortem, analysis of what happened.

jessaustin 7y ago

I guess GitLab are no longer the most notable outage of the day...

g051051 7y ago

If I hadn't been looking here, I might never have known. Yes, I'm bragging.

rinchik 7y ago

Also curious if FB outages affect global internet traffic? Are there any relative stats? (E.g. if google to go down, it would have an economic cost comparable to a natural disaster)

pdpi 7y ago
There's several anecdotes internally at Facebook that get told to new starters about some instances of when facebook.com was down. One involves masses of people calling the police to report that facebook is offline (wat, why?), another involves so many people flocking to twitter to complain that fb is down that that crowd then knocked twitter offline ("How do you know if facebook is down? Check whether twitter is online" sort of thing). I don't feel comfortable sharing the exact figures, but during my time there the cost of a few minutes of downtime on an ads-adjacent message queue cluster directly turned into millions in lost revenue, so it's safe to say that losing the capability to _serve_ ads (as opposed to charging for serving them) would have a pretty damn high economic cost to those advertisers. As Workplace gains traction, downtime there will creep into the "cataclysmic" side of the economic costs scale.
- jfim 7y ago
  Did you count the lost revenue as revenue per minute times the number of minutes the site was unavailable? If so, that's actually incorrect when calculating revenue loss. There will be actually be an uptick in traffic after the outage, either through people postponing their intended visit or checking if the site is down for them after it comes back. I know that at LinkedIn we broke part of the advertisement campaign creation flow from downtime in our service for a short time, and the impact was negligible since ads were still being served, and new ad creation would be just be retried later by advertisers. Now if you just lost the capacity to serve ads, there might be an impact, but since a lot of advertisements are display ads (to raise brand awareness over an immediate call to action), it's not clear whether or not it's a simple minutes down times impact per minute calculation.
  - jacquesm 7y ago
    Facebook being down is only lost revenue for Facebook. Every other company on the planet just gained in terms of productivity and revenues.

chadlavi 7y ago

It's a thanksgiving miracle

rossdavidh 7y ago

Man, what a wasted opportunity. If it had happened on a workday, we could have seen a massive productivity surge to boost the economy.

AznHisoka 7y ago
At least families will be forced to talk to each other instead of being on FB...

100016054749004 7y ago

https://m.facebook.com/profile.php?id=100016054749004&refsrc=https%3A%2F%2Fm.facebook.com%2Fprofile.php&_rdr https://m.facebook.com/profile.php?id=100016054749004&refsrc...

100016054749004 7y ago

100016054749004