8 ms·

Love the status page right now. https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/ PS. Instagram app/site, wha

Love the status page right now. https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/ PS. Instagram app/site, whatsapp.com, messenger.com, oculus.com also down, anything else? UPDATE: Everything back up! (Even the status page)

threatripper 5y ago
Do they share infrastructure?
- fastball 5y ago
  Yeah, that's what I was wondering. Strange that there is a SPOF for Facebook/Insta/Whatsapp.
  - threatripper 5y ago
    I had the impression that they are still run like mostly independent companies who only share some data on the backend.
    - fastball 5y ago
      Yeah that's what I was thinking too, so this is probably DNS related.
    - thrusong 5y ago
      AFAIK, I remember reading something about Instagram moving into Facebook datacentres some time ago. I believe they were on AWS before the acquisition.
  - godelmachine 5y ago
    Might I ask what’s SPOF?
    - fastball 5y ago
      Single Point of Failure.
      - godelmachine 5y ago
        Thanks
    - octo_t 5y ago
      Single Point Of Failure
      - godelmachine 5y ago
        Ta
- lavp 5y ago
  Given the fact that it's also down, I'd say yes
  - threatripper 5y ago
    It could be some rouge admin who actually deleted Facebook.
    - viraptor 5y ago
      At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
      - londons_explore 5y ago
        I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue. For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined... I guess they have enough pending stock to deter them...
        tracyhenry 5y ago
        How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.
        londons_explore 5y ago
        It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly). Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line. Besides, the vast majority of code reviews are 10 minutes or less.
DavidPeiffer 5y ago
Anecdotally, whatsapp worked for me (message was sent and received successfully) though Facebook gave an error. I was already logged into both services.
- eroded 5y ago
  whatsapp.com as in the web-based client, which is down.
aarongolliver 5y ago
Finally a status page that updates the moment an issue starts
alexcnwy 5y ago
Even the status page is down
cure 5y ago
Seriously, they host the status page on their own infrastructure? That's ... not smart.
- kickling 5y ago
  Seems like Giphy is up at least, also facebook owned.
- gh-throw 5y ago
  Every company's actual status page (and news feed) is on Twitter, no matter what else they say. I don't know where Twitter hosts their real status page. Probably Facebook. Notably, Facebook hasn't updated theirs, though.
  - btown 5y ago
    > every company's actual status page is on Twitter Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage. EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
    - spondyl 5y ago
      During the recent Microsoft Azure AD outage, I attempted to use the chat widget. Selecting "Technical support" or "Billing Support" both seemed to return 500 status codes while the "Sales Support" routed somewhere else and connected me to a person, perfectly fine. Presumably the sales team don't use Azure for anything.
setpatchaddress 5y ago
for posterity: <?xml version="1.0"?> <!DOCTYPE html> <html lang="en" id="facebook"> <head> <title>Error</title> <meta charset="utf-8"/> <meta http-equiv="Cache-Control" content="no-cache"/> <meta name="robots" content="noindex,nofollow"/> <style> html, body { color: #333; font-family: 'Lucida Grande', 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: 0; text-align: center;} #header { height: 30px; padding-bottom: 10px; padding-top: 10px; text-align: center;} #icon { width: 30px;} .core { margin: auto; padding: 1em 0; text-align: left; width: 904px;} h1 { font-size: 18px;} p { font-size: 13px;} .footer { border-top: 1px solid #ddd; color: #777; float: left; font-size: 11px; padding: 5px 8px 6px 0; width: 904px;} </style> </head> <body> <div id="header"> <a href="//www.facebook.com/"> <img id="icon" src="//static.facebook.com/images/logos/facebook_2x.png"/> </a> </div> <div class="core"> <h1>Sorry, something went wrong.</h1> <p>We're working on getting this fixed as soon as we can.</p> <p> <a id="back" href="//www.facebook.com/">Go Back</a> </p> <div class="footer"> Facebook © 2021 · <a href="//www.facebook.com/help/">Help</a></div> </div> <script> document.getElementById("back").onclick = function() { if (history.length > 1) { history.back(); return false; } }; </script> </body> </html> And the png does not load.
- unpythonic 5y ago
  And here are the headers: # curl -i https://www.facebook.com/ HTTP/1.1 500 Internal Server Error X-Frame-Options: DENY X-XSS-Protection: 0 X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=15552000; preload Set-Cookie: ... Expires: Sat, 01 Jan 2000 00:00:00 GMT Cache-Control: private, no-cache, no-store, must-revalidate Vary: Accept-Encoding Pragma: no-cache x-fb-rlafr: 0 Content-Type: text/html; charset="utf-8" X-FB-Debug: ... Date: Thu, 08 Apr 2021 21:41:49 GMT Priority: u=3,i Transfer-Encoding: chunked Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600 Connection: keep-alive <!DOCTYPE html><html lang="en" id="facebook"><head><title>Error</title>...
  - 5y ago
css 5y ago
It loads fine for me and shows no downtime or incidents. It even says "Facebook Platform is Healthy"
- kickling 5y ago
  Yup, everything just came back online.
  - css 5y ago
    Now I am unable to connect at all: ~# wget https://developers.facebook.com/status/dashboard/ Connecting to developers.facebook.com (157.240.206.16:443) wget: can't connect to remote host (157.240.206.16): Operation timed out
- chris_wot 5y ago
  Try posting
- reaperducer 5y ago
  Not here. Status page still does not load at all.
  - liquidify 5y ago
    same, and can't visit the site at all
grensley 5y ago
I like that it doesn't know that anything went wrong.

threatripper 5y ago

Do they share infrastructure?

fastball 5y ago
Yeah, that's what I was wondering. Strange that there is a SPOF for Facebook/Insta/Whatsapp.
- threatripper 5y ago
  I had the impression that they are still run like mostly independent companies who only share some data on the backend.
  - fastball 5y ago
    Yeah that's what I was thinking too, so this is probably DNS related.
  - thrusong 5y ago
    AFAIK, I remember reading something about Instagram moving into Facebook datacentres some time ago. I believe they were on AWS before the acquisition.
- godelmachine 5y ago
  Might I ask what’s SPOF?
  - fastball 5y ago
    Single Point of Failure.
    - godelmachine 5y ago
      Thanks
  - octo_t 5y ago
    Single Point Of Failure
    - godelmachine 5y ago
      Ta
lavp 5y ago
Given the fact that it's also down, I'd say yes
- threatripper 5y ago
  It could be some rouge admin who actually deleted Facebook.
  - viraptor 5y ago
    At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
    - londons_explore 5y ago
      I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue. For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined... I guess they have enough pending stock to deter them...
      - tracyhenry 5y ago
        How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.
        londons_explore 5y ago
        It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly). Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line. Besides, the vast majority of code reviews are 10 minutes or less.

DavidPeiffer 5y ago

Anecdotally, whatsapp worked for me (message was sent and received successfully) though Facebook gave an error. I was already logged into both services.

eroded 5y ago
whatsapp.com as in the web-based client, which is down.

aarongolliver 5y ago

Finally a status page that updates the moment an issue starts

alexcnwy 5y ago

Even the status page is down

cure 5y ago

Seriously, they host the status page on their own infrastructure? That's ... not smart.

kickling 5y ago
Seems like Giphy is up at least, also facebook owned.
gh-throw 5y ago
Every company's actual status page (and news feed) is on Twitter, no matter what else they say. I don't know where Twitter hosts their real status page. Probably Facebook. Notably, Facebook hasn't updated theirs, though.
- btown 5y ago
  > every company's actual status page is on Twitter Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage. EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
  - spondyl 5y ago
    During the recent Microsoft Azure AD outage, I attempted to use the chat widget. Selecting "Technical support" or "Billing Support" both seemed to return 500 status codes while the "Sales Support" routed somewhere else and connected me to a person, perfectly fine. Presumably the sales team don't use Azure for anything.

setpatchaddress 5y ago

for posterity: <?xml version="1.0"?> <!DOCTYPE html> <html lang="en" id="facebook"> <head> <title>Error</title> <meta charset="utf-8"/> <meta http-equiv="Cache-Control" content="no-cache"/> <meta name="robots" content="noindex,nofollow"/> <style> html, body { color: #333; font-family: 'Lucida Grande', 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: 0; text-align: center;} #header { height: 30px; padding-bottom: 10px; padding-top: 10px; text-align: center;} #icon { width: 30px;} .core { margin: auto; padding: 1em 0; text-align: left; width: 904px;} h1 { font-size: 18px;} p { font-size: 13px;} .footer { border-top: 1px solid #ddd; color: #777; float: left; font-size: 11px; padding: 5px 8px 6px 0; width: 904px;} </style> </head> <body> <div id="header"> <a href="//www.facebook.com/"> <img id="icon" src="//static.facebook.com/images/logos/facebook_2x.png"/> </a> </div> <div class="core"> <h1>Sorry, something went wrong.</h1> <p>We're working on getting this fixed as soon as we can.</p> <p> <a id="back" href="//www.facebook.com/">Go Back</a> </p> <div class="footer"> Facebook © 2021 · <a href="//www.facebook.com/help/">Help</a></div> </div> <script> document.getElementById("back").onclick = function() { if (history.length > 1) { history.back(); return false; } }; </script> </body> </html> And the png does not load.

unpythonic 5y ago
And here are the headers: # curl -i https://www.facebook.com/ HTTP/1.1 500 Internal Server Error X-Frame-Options: DENY X-XSS-Protection: 0 X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=15552000; preload Set-Cookie: ... Expires: Sat, 01 Jan 2000 00:00:00 GMT Cache-Control: private, no-cache, no-store, must-revalidate Vary: Accept-Encoding Pragma: no-cache x-fb-rlafr: 0 Content-Type: text/html; charset="utf-8" X-FB-Debug: ... Date: Thu, 08 Apr 2021 21:41:49 GMT Priority: u=3,i Transfer-Encoding: chunked Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600 Connection: keep-alive <!DOCTYPE html><html lang="en" id="facebook"><head><title>Error</title>...
- 5y ago

css 5y ago

It loads fine for me and shows no downtime or incidents. It even says "Facebook Platform is Healthy"

kickling 5y ago
Yup, everything just came back online.
- css 5y ago
  Now I am unable to connect at all: ~# wget https://developers.facebook.com/status/dashboard/ Connecting to developers.facebook.com (157.240.206.16:443) wget: can't connect to remote host (157.240.206.16): Operation timed out
chris_wot 5y ago
Try posting
reaperducer 5y ago
Not here. Status page still does not load at all.
- liquidify 5y ago
  same, and can't visit the site at all

grensley 5y ago

I like that it doesn't know that anything went wrong.