5 ms·

Can't wait for the post mortem technical post

avaika 5y ago
There's some explanation by CloudFlare available [0]. But for sure that'd be interesting to read facebooks version. [0]: https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
type_Ben_struct 5y ago
If past outages are anything to go by there’s no chance of this seeing the light of day.
- infini8 5y ago
  This one in particular seems interesting. Felt like one of the longer outages with soo many different sites effected.
- paxys 5y ago
  This one will definitely see the light of day considering how big a deal it is. Shareholders could easily sue over it.
  - drusepth 5y ago
    What makes you think shareholders could sue over this outage without knowing the cause?
    - paxys 5y ago
      They could sue over Facebook not telling them the cause
      - otterley 5y ago
        Shareholders aren't entitled to know every detail of a company's operations, including why outages occur.
- weeeeelp 5y ago
  Here's hoping to a whistleblower...
devmunchies 5y ago
mark accidentally logged out. first time he'd done it since it started.
- throwdecro 5y ago
  Who's "mark"?
  - powersurge360 5y ago
    Refers to Mark Zuckerberg, founder of Facebook
    - alexilliamson 5y ago
      And Founder of Our Feast!
andrewxdiamond 5y ago
Facebook, unlike other big tech companies, has no financial incentive to share postmortems. Others are bound by SLA agreements with business customers.
- itslennysfault 5y ago
  Well, as someone that has an app that had degraded performance due to their SDK failing to talk to Facebook servers. I FOR ONE would really like an explanation.
  - andrewxdiamond 5y ago
    Then collaborate with your peers to get enough leverage over Facebook to demand this. It’s not like FB’s market position let’s them dictate terms in an unfair way that would require a certain governing force to step in and rebalance the tables…… /s
  - callalex 5y ago
    I assume that if your business relies on this service you have an account representative you can contact for help with technical and billing issues?
- renaudg 5y ago
  FB does have business customers with (at least) Workplace. Hence the name of this dashboard.
itomato 5y ago
Zuck is writing it now.
this_user 5y ago
Allegedly, this was a BGP update gone wrong that locked out their remote access. And the people with physical access didn't have the necessary privileges to fix the issue.
canada_dry 5y ago
I've seen a couple predominant tin-foil-hat theories are making the rounds: - a disgruntled NOC engineer - the company is scrubbing any/all data related to the leak that 60 Minutes reported It seems pretty astonishing that all BGP routing just vanished. I'm guessing the only way we'll find out is if an insider spills the beans.

There's some explanation by CloudFlare available [0]. But for sure that'd be interesting to read facebooks version. [0]: https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/

type_Ben_struct 5y ago

If past outages are anything to go by there’s no chance of this seeing the light of day.

infini8 5y ago
This one in particular seems interesting. Felt like one of the longer outages with soo many different sites effected.
paxys 5y ago
This one will definitely see the light of day considering how big a deal it is. Shareholders could easily sue over it.
- drusepth 5y ago
  What makes you think shareholders could sue over this outage without knowing the cause?
  - paxys 5y ago
    They could sue over Facebook not telling them the cause
    - otterley 5y ago
      Shareholders aren't entitled to know every detail of a company's operations, including why outages occur.
weeeeelp 5y ago
Here's hoping to a whistleblower...

devmunchies 5y ago

mark accidentally logged out. first time he'd done it since it started.

throwdecro 5y ago
Who's "mark"?
- powersurge360 5y ago
  Refers to Mark Zuckerberg, founder of Facebook
  - alexilliamson 5y ago
    And Founder of Our Feast!

andrewxdiamond 5y ago

Facebook, unlike other big tech companies, has no financial incentive to share postmortems. Others are bound by SLA agreements with business customers.

itslennysfault 5y ago
Well, as someone that has an app that had degraded performance due to their SDK failing to talk to Facebook servers. I FOR ONE would really like an explanation.
- andrewxdiamond 5y ago
  Then collaborate with your peers to get enough leverage over Facebook to demand this. It’s not like FB’s market position let’s them dictate terms in an unfair way that would require a certain governing force to step in and rebalance the tables…… /s
- callalex 5y ago
  I assume that if your business relies on this service you have an account representative you can contact for help with technical and billing issues?
renaudg 5y ago
FB does have business customers with (at least) Workplace. Hence the name of this dashboard.

itomato 5y ago

Zuck is writing it now.

this_user 5y ago

Allegedly, this was a BGP update gone wrong that locked out their remote access. And the people with physical access didn't have the necessary privileges to fix the issue.

canada_dry 5y ago

I've seen a couple predominant tin-foil-hat theories are making the rounds: - a disgruntled NOC engineer - the company is scrubbing any/all data related to the leak that 60 Minutes reported It seems pretty astonishing that all BGP routing just vanished. I'm guessing the only way we'll find out is if an insider spills the beans.