5 ms·

Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.

johnduhart 5y ago
I think you need to re-adjust your expectations, it's not reasonable to have a fully fleshed out RCA blog post available within hours of incident resolution. Most other cloud providers take a few days for theirs.
- padolsey 5y ago
  I mean, not an RCA per se, but info more akin to cloudflare's blog post would be v welcome IMHO: https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
  - bawolff 5y ago
    Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.
    - gizmo385 5y ago
      The Cloudflare post includes graphs depicting how the issue looked downstream and a rough initial timeline for the incident. The Facebook post says basically nothing more than that there was a networking mistake. I hope we get something more substantial and informative than that over the next couple of weeks, but it doesn't seem (at least from my searching) that Facebook is in the business of publicly posting in-depth post-mortems for their outages, which I personally find unfortunate.
      - ignoramous 5y ago
        Cloudflare can scratch the surface of the issue it wouldn't matter, it is a content marketing piece after all. Facebook, otoh, needs to be thorough.
      - brianwawok 5y ago
        How tech savvy are people who pay Cloudflare money? vs How tech savvy are people that Facebook profits from? Gotta target your audience, every communication is PR...
- tinus_hn 5y ago
  It’s not reasonable to demand any details at all, it’s nice of them to notify people of what went wrong but it really is none of our business.
  - dTal 5y ago
    On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet. Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.
    - ikt 5y ago
      I don't want an explanation nor do I care, Facebook could disappear tomorrow like all the other networks before it and it wouldn't make a dent in my day.
      - oefnak 5y ago
        Doe you honestly believe that Facebook and its subsidiaries don't have a major impact on the world?
      - dTal 5y ago
        Okay? Why are you commenting here then?
      - unethical_ban 5y ago
        Whether we like it or not, all three platforms are relied upon by hundreds of millions of people and businesses everyday for communication. I'm sure the world would quickly adapt by re-adopting these things called "websites" and "email" but in the meantime, it's highly self-centered to think this "didn't matter".
    - tinus_hn 5y ago
      Or else? You will angrily stamp your foot? Start an e-petition?
paxys 5y ago
RCAs take time. It's best to issue vague statements during and right after an incident rather than make guesses.
- KronisLV 5y ago
  > It's best to issue vague statements during and right after an incident rather than make guesses. Why? Why couldn't you just post that the RCA is still ongoing and that proper updates will follow? Otherwise all you get is meaningless fluff.
  - jve 5y ago
    Well, because you can fail on initial assumptions. And may spread officially-false information. This is not meaningless fluff. It may not provide info to technical persons, but valid info to other persons, as others noted (was hacked: yes/no). Getting down to root cause takes time. It is usually multiple-things-at-once that caused X to happen. And then they must also make a decision on how to prevent X to happen again. All that must be written into RCA. It takes days not hours. An example from my life: Service has intermittent disruptions. Antivirus activity correlated 100% with disruptions. Upon further investigation turns out that AV was just doing its job when there was less load. (And before anyone points out why on earth there is AV on such service, well, because it deals with user uploaded files) So what should I have called out - AV is the guilty one? And then say: oh, no, false info.
vmception 5y ago
Its important for many stakeholders to understand it wasn’t a hack/exploit or malicious third party or malicious insider Its much better for some random committee in Congress to debate antitrust forever, instead of bigger committees and agencies debating national security threats
ajkjk 5y ago
Disagree -- it's here to establish something that a lot of people have been speculating about, which is whether it's hacking-related. It doesn't say much because its purpose is to deliver a single bit of information: { hacking: boolean }
bawolff 5y ago
It clearly is a PR piece for investors and customers. And that's ok, not everything is an eng blog.
- retSava 5y ago
  Pointing out that this is published under _engineering_.fb.com.
rplnt 5y ago
Not worth clicking really, everything is in the url.
yuliyp 5y ago
It's less vague than you realize. It points out that the problem was within Facebook's network between its datacenters. This not only suggests it's related to express backbone, but also suggests that the DNS BGP withdrawal which Cloudflare observed was not the primary issue. It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.
- EricE 5y ago
  A point of distinction - there is no "DNS BGP withdrawl". DNS is related to BGP only that without the right BGP routes in the routers, no packets can get to the facebook networks and thus the facbook DNS servers. That their DNS servers were taken out was a side affect of the root issue - they withdrew all the routes to their networks from the rest of the Internet. Not picking on you - but there has been a lot of confusion around DNS that is mostly a red herring and people should just drop it from the conversation. Everything on facebook networks disappeared, not just DNS. The main issue is they effectively took a pair of scissors to every one of their internet connections - d'oh!
  - yuliyp 5y ago
    FB did not totally disconnect from the internet as far as what I observed. They broke their backbone (from the blog post, as well as seeing that their edge was returning 503s during the incident), and FB withdrew specific BGP announcements which covered their DNS servers are what I saw. I think this is consistent with what Cloudflare actually said, although not what has been often repeated since then.
  - matsur 5y ago
    This is not what we observed. Less specific routes were still present in routing tables, but the more specifics that routed DNS traffic disappeared.
deleted 5y ago
[deleted]

I think you need to re-adjust your expectations, it's not reasonable to have a fully fleshed out RCA blog post available within hours of incident resolution. Most other cloud providers take a few days for theirs.

padolsey 5y ago
I mean, not an RCA per se, but info more akin to cloudflare's blog post would be v welcome IMHO: https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
- bawolff 5y ago
  Both posts have essentially the same info - the fb one just didn't include an explainer on how the internet works.
  - gizmo385 5y ago
    The Cloudflare post includes graphs depicting how the issue looked downstream and a rough initial timeline for the incident. The Facebook post says basically nothing more than that there was a networking mistake. I hope we get something more substantial and informative than that over the next couple of weeks, but it doesn't seem (at least from my searching) that Facebook is in the business of publicly posting in-depth post-mortems for their outages, which I personally find unfortunate.
    - ignoramous 5y ago
      Cloudflare can scratch the surface of the issue it wouldn't matter, it is a content marketing piece after all. Facebook, otoh, needs to be thorough.
    - brianwawok 5y ago
      How tech savvy are people who pay Cloudflare money? vs How tech savvy are people that Facebook profits from? Gotta target your audience, every communication is PR...
tinus_hn 5y ago
It’s not reasonable to demand any details at all, it’s nice of them to notify people of what went wrong but it really is none of our business.
- dTal 5y ago
  On the off-chance this isn't sarcasm, Facebook's routing shenanigans slowed down the entire internet. Not to mention that they're a publicly traded company, and one which has gone out of its way to assume an infrastructure role. They don't have a right to privacy here, and we are all owed an explanation.
  - ikt 5y ago
    I don't want an explanation nor do I care, Facebook could disappear tomorrow like all the other networks before it and it wouldn't make a dent in my day.
    - oefnak 5y ago
      Doe you honestly believe that Facebook and its subsidiaries don't have a major impact on the world?
    - dTal 5y ago
      Okay? Why are you commenting here then?
    - unethical_ban 5y ago
      Whether we like it or not, all three platforms are relied upon by hundreds of millions of people and businesses everyday for communication. I'm sure the world would quickly adapt by re-adopting these things called "websites" and "email" but in the meantime, it's highly self-centered to think this "didn't matter".
  - tinus_hn 5y ago
    Or else? You will angrily stamp your foot? Start an e-petition?

paxys 5y ago

RCAs take time. It's best to issue vague statements during and right after an incident rather than make guesses.

KronisLV 5y ago
> It's best to issue vague statements during and right after an incident rather than make guesses. Why? Why couldn't you just post that the RCA is still ongoing and that proper updates will follow? Otherwise all you get is meaningless fluff.
- jve 5y ago
  Well, because you can fail on initial assumptions. And may spread officially-false information. This is not meaningless fluff. It may not provide info to technical persons, but valid info to other persons, as others noted (was hacked: yes/no). Getting down to root cause takes time. It is usually multiple-things-at-once that caused X to happen. And then they must also make a decision on how to prevent X to happen again. All that must be written into RCA. It takes days not hours. An example from my life: Service has intermittent disruptions. Antivirus activity correlated 100% with disruptions. Upon further investigation turns out that AV was just doing its job when there was less load. (And before anyone points out why on earth there is AV on such service, well, because it deals with user uploaded files) So what should I have called out - AV is the guilty one? And then say: oh, no, false info.

vmception 5y ago

Its important for many stakeholders to understand it wasn’t a hack/exploit or malicious third party or malicious insider Its much better for some random committee in Congress to debate antitrust forever, instead of bigger committees and agencies debating national security threats

ajkjk 5y ago

Disagree -- it's here to establish something that a lot of people have been speculating about, which is whether it's hacking-related. It doesn't say much because its purpose is to deliver a single bit of information: { hacking: boolean }

bawolff 5y ago

It clearly is a PR piece for investors and customers. And that's ok, not everything is an eng blog.

retSava 5y ago
Pointing out that this is published under _engineering_.fb.com.

rplnt 5y ago

Not worth clicking really, everything is in the url.

yuliyp 5y ago

It's less vague than you realize. It points out that the problem was within Facebook's network between its datacenters. This not only suggests it's related to express backbone, but also suggests that the DNS BGP withdrawal which Cloudflare observed was not the primary issue. It's not a full root cause analysis, to be sure, and leaves many open questions, but I definitely wouldn't describe it as painfully vague.

EricE 5y ago
A point of distinction - there is no "DNS BGP withdrawl". DNS is related to BGP only that without the right BGP routes in the routers, no packets can get to the facebook networks and thus the facbook DNS servers. That their DNS servers were taken out was a side affect of the root issue - they withdrew all the routes to their networks from the rest of the Internet. Not picking on you - but there has been a lot of confusion around DNS that is mostly a red herring and people should just drop it from the conversation. Everything on facebook networks disappeared, not just DNS. The main issue is they effectively took a pair of scissors to every one of their internet connections - d'oh!
- yuliyp 5y ago
  FB did not totally disconnect from the internet as far as what I observed. They broke their backbone (from the blog post, as well as seeing that their edge was returning 503s during the incident), and FB withdrew specific BGP announcements which covered their DNS servers are what I saw. I think this is consistent with what Cloudflare actually said, although not what has been often repeated since then.
- matsur 5y ago
  This is not what we observed. Less specific routes were still present in routing tables, but the more specifics that routed DNS traffic disappeared.

deleted 5y ago

[deleted]