4 ms·

Facebook Platform was down

74 points by haney 8y ago

A check of https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps. EDIT: Commenters are reporting that this appears to only impact users that are hitting the US data center[1], and that it may only impact users through specific ISPs 1.https://news.ycombinator.com/item?id=18434457 EDIT: Appears to be back up

em3rgent0rdr 8y ago
me too.
garymoon 8y ago
same
si1entstill 8y ago
Moving fast and breaking things!
chuckcode 8y ago
same. downdetector reporting 1200 plus reports of facebook down https://downdetector.com/status/facebook https://downdetector.com/status/facebook
bbcoimbra 8y ago
Is down here
CJKinni 8y ago
So much for the pivot to "move fast with stable infrastructure."
- seattle_spring 8y ago
  They've been down for, what, 20 minutes in the past year?
  - greglindahl 8y ago
    Most of their outages appear to be regional, and only part of their site. This one also appears to have been regional. So it's hard for us outsiders to know the statistics. These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.
deleted 8y ago
[deleted]
champagnepapi 8y ago
same as well as facebook.com
mkong1 8y ago
broken image from static.facebook.com for the logo, too.
faitswulff 8y ago
This should be an interesting postmortem, if they have one. I think Netflix never released a postmortem of their latest downtime.
- dgritsko 8y ago
  Unrelated to OP, but did Google ever release a postmortem for the YouTube outage a few weeks back? Was interested to hear the cause of that one.
  - deleted 8y ago
    [deleted]
- garysahota93 8y ago
  I love reading the postmortems. I've read some from a few different tech sites and it's fascinating to hear just how people are doing things and how they fixed it.
  - gadders 8y ago
    My semi-dormant website: https://www.outagereports.net/ https://www.outagereports.net/ I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.
danieldiazas 8y ago
Down on latin america too
Simulacra 8y ago
Yep but only through the Verizon connection. Phone works, Tor connection to FB works, so it's probably localized.
- noarchy 8y ago
  I can confirm what you say about Tor. I get the login screen when hitting FB via Tor, but not otherwise.
cronix 8y ago
Strange, I just logged into the website fine from Portland, OR via Comcast
yeldarb 8y ago
Interestingly I VPN'd into a node in Dublin and it is still working there. Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.
LinuxBender 8y ago
Looks like they had a little blip [1] [1] - https://downdetector.com/ https://downdetector.com/
hakanito 8y ago
works in EU
- shashanoid 8y ago
  But does it?
bploetz 8y ago
https://www.youtube.com/watch?v=Rf2tz7_-2NI&feature=youtu.be&t=4 https://www.youtube.com/watch?v=Rf2tz7_-2NI&feature=youtu.be...
skizm 8y ago
Using a completely oversimplified estimation method and 2017 revenue numbers, they're losing about $1300 every second their website is down.
- mattnewton 8y ago
  Not Facebook.com, the platform that Facebook apps are built on I think.
  - txcwpalpha 8y ago
    Facebook.com was definitely down for me for several minutes at least.
  - cronix 8y ago
    It's down for a lot. Seems to primarily be NE US: https://downdetector.com/status/facebook/map/ https://downdetector.com/status/facebook/map/
nabla9 8y ago
From EU: Facebook Platform is Healthy Since October 23 at 4:10 PM
- a13n 8y ago
  Different data center, looks like the outage is US-based.
alphabettsy 8y ago
Regular login down too.
Mc_Big_G 8y ago
Ideally it would never come back up.
a13n 8y ago
Back up for me. Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks. Pretty staggering to think about. [1]: https://ycharts.com/companies/FB/revenues https://ycharts.com/companies/FB/revenues
- abkumar 8y ago
  > this outage probably cost them just over a million bucks I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage. For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.
  - derefr 8y ago
    That works with e-commerce, but not necessarily with advertising. If you were going to browse Facebook during lunch in the office (and be shown ads), but Facebook is down, then you won't necessarily just "browse Facebook later in the day instead." You weren't on Facebook to catch up; you were on Facebook because you had nothing better to do.
    - greglindahl 8y ago
      This is definitely true, I expect that users will spend a bit less time on Facebook today. However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.
  - a13n 8y ago
    For e-commerce that argument makes sense, but not for advertising. They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.
pknopf 8y ago
It's working now.

74 points by haney 8y ago

em3rgent0rdr 8y ago

me too.

garymoon 8y ago

same

si1entstill 8y ago

Moving fast and breaking things!

chuckcode 8y ago

same. downdetector reporting 1200 plus reports of facebook down https://downdetector.com/status/facebook https://downdetector.com/status/facebook

bbcoimbra 8y ago

Is down here

CJKinni 8y ago

So much for the pivot to "move fast with stable infrastructure."

seattle_spring 8y ago
They've been down for, what, 20 minutes in the past year?
- greglindahl 8y ago
  Most of their outages appear to be regional, and only part of their site. This one also appears to have been regional. So it's hard for us outsiders to know the statistics. These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.

deleted 8y ago

[deleted]

champagnepapi 8y ago

same as well as facebook.com

mkong1 8y ago

broken image from static.facebook.com for the logo, too.

faitswulff 8y ago

This should be an interesting postmortem, if they have one. I think Netflix never released a postmortem of their latest downtime.

dgritsko 8y ago
Unrelated to OP, but did Google ever release a postmortem for the YouTube outage a few weeks back? Was interested to hear the cause of that one.
- deleted 8y ago
  [deleted]
garysahota93 8y ago
I love reading the postmortems. I've read some from a few different tech sites and it's fascinating to hear just how people are doing things and how they fixed it.
- gadders 8y ago
  My semi-dormant website: https://www.outagereports.net/ https://www.outagereports.net/ I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.

danieldiazas 8y ago

Down on latin america too

Simulacra 8y ago

Yep but only through the Verizon connection. Phone works, Tor connection to FB works, so it's probably localized.

noarchy 8y ago
I can confirm what you say about Tor. I get the login screen when hitting FB via Tor, but not otherwise.

cronix 8y ago

Strange, I just logged into the website fine from Portland, OR via Comcast

yeldarb 8y ago

Interestingly I VPN'd into a node in Dublin and it is still working there. Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.

LinuxBender 8y ago

Looks like they had a little blip [1] [1] - https://downdetector.com/ https://downdetector.com/

hakanito 8y ago

works in EU

shashanoid 8y ago
But does it?

bploetz 8y ago

https://www.youtube.com/watch?v=Rf2tz7_-2NI&feature=youtu.be&t=4 https://www.youtube.com/watch?v=Rf2tz7_-2NI&feature=youtu.be...

skizm 8y ago

Using a completely oversimplified estimation method and 2017 revenue numbers, they're losing about $1300 every second their website is down.

mattnewton 8y ago
Not Facebook.com, the platform that Facebook apps are built on I think.
- txcwpalpha 8y ago
  Facebook.com was definitely down for me for several minutes at least.
- cronix 8y ago
  It's down for a lot. Seems to primarily be NE US: https://downdetector.com/status/facebook/map/ https://downdetector.com/status/facebook/map/

nabla9 8y ago

From EU: Facebook Platform is Healthy Since October 23 at 4:10 PM

a13n 8y ago
Different data center, looks like the outage is US-based.

alphabettsy 8y ago

Regular login down too.

Mc_Big_G 8y ago

Ideally it would never come back up.

a13n 8y ago

Back up for me. Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks. Pretty staggering to think about. [1]: https://ycharts.com/companies/FB/revenues https://ycharts.com/companies/FB/revenues

abkumar 8y ago
> this outage probably cost them just over a million bucks I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage. For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.
- derefr 8y ago
  That works with e-commerce, but not necessarily with advertising. If you were going to browse Facebook during lunch in the office (and be shown ads), but Facebook is down, then you won't necessarily just "browse Facebook later in the day instead." You weren't on Facebook to catch up; you were on Facebook because you had nothing better to do.
  - greglindahl 8y ago
    This is definitely true, I expect that users will spend a bit less time on Facebook today. However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.
- a13n 8y ago
  For e-commerce that argument makes sense, but not for advertising. They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.

pknopf 8y ago

It's working now.