6 ms·

Yeah the patch to fix BGP to reach the DNS is sent by email to @facebook.com. Ooops no DNS to resolve the MX records to send the patch to fix the BGP routers.

yoelo 5y ago
Seriously? Is that how it works?
- john37386 5y ago
  I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
  - boshomi 5y ago
    Sheera Frenkel: Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
  - _joel 5y ago
    You'd think they'd have worked that into their DR plans for a complete P1 outage of the domain/DNS, but perhaps not, or at least they didn't add removal of BGP announcements to the mix.
- weisk 5y ago
  No, the backbone of the internet is not maintained with patches sent in emails.
  - chiluk 5y ago
    https://lkml.org/ https://lkml.org/
  - chiluk 5y ago
    You are very wrong about that https://lkml.org/ https://lkml.org/
    - cbarrick 5y ago
      Clearly you and the person you replied to are talking about very different things.
  - chiluk 5y ago
    You are very wrong about that ;) https://lkml.org/ https://lkml.org/
  - _joel 5y ago
    I think the sub-comment is confusing the linux kernel with BGP.
    - nacs 5y ago
      In a way, the Linux kernel does power the "backbones of the internet".
      - _joel 5y ago
        There are a hell of a lot of non-linux OS's running on core routers, but yes, in a way. However BGP isn't via email.
        dsr_ 5y ago
        On the other hand, I and my office mate at the time negotiated the setup of a ridiculous number of BGP sessions over email, including sending configs. That was 20 years ago.
      - NexRebular 5y ago
        luckily not... would be absolutely terrible to have the backbone only on linux
        dsr_ 5y ago
        Interoperability and a thriving ecosystem are necessities for resiliency. Note that resiliency and efficiency are often working against each other.
- deleted 5y ago
  [deleted]
- cranekam 5y ago
  No. A network like Facebook's is vast and complicated and managed by higher-level configuration systems, not people emailing patches around. If this issue is even to do with BGP it's much more likely the root of the problem is somewhere in this configuration system and that fixing it is compounded by some other issues that nobody foresaw. Huge events like this are always a perfect storm of several factors, any one or two of which would be a total noop alone.
  - KuiN 5y ago
    The Swiss cheese model of accidents. Occasionally the holes all align. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
  - soneil 5y ago
    The fun part of BGP is they apparently make a lot of use of it within their network, not just advertising routes externally. https://engineering.fb.com/2021/05/13/data-center-engineering/bgp/ https://engineering.fb.com/2021/05/13/data-center-engineerin... (and yes, fb.com resolves)

yoelo 5y ago

Seriously? Is that how it works?

john37386 5y ago
I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
- boshomi 5y ago
  Sheera Frenkel: Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
- _joel 5y ago
  You'd think they'd have worked that into their DR plans for a complete P1 outage of the domain/DNS, but perhaps not, or at least they didn't add removal of BGP announcements to the mix.
weisk 5y ago
No, the backbone of the internet is not maintained with patches sent in emails.
- chiluk 5y ago
  https://lkml.org/ https://lkml.org/
- chiluk 5y ago
  You are very wrong about that https://lkml.org/ https://lkml.org/
  - cbarrick 5y ago
    Clearly you and the person you replied to are talking about very different things.
- chiluk 5y ago
  You are very wrong about that ;) https://lkml.org/ https://lkml.org/
- _joel 5y ago
  I think the sub-comment is confusing the linux kernel with BGP.
  - nacs 5y ago
    In a way, the Linux kernel does power the "backbones of the internet".
    - _joel 5y ago
      There are a hell of a lot of non-linux OS's running on core routers, but yes, in a way. However BGP isn't via email.
      - dsr_ 5y ago
        On the other hand, I and my office mate at the time negotiated the setup of a ridiculous number of BGP sessions over email, including sending configs. That was 20 years ago.
    - NexRebular 5y ago
      luckily not... would be absolutely terrible to have the backbone only on linux
      - dsr_ 5y ago
        Interoperability and a thriving ecosystem are necessities for resiliency. Note that resiliency and efficiency are often working against each other.
deleted 5y ago
[deleted]
cranekam 5y ago
No. A network like Facebook's is vast and complicated and managed by higher-level configuration systems, not people emailing patches around. If this issue is even to do with BGP it's much more likely the root of the problem is somewhere in this configuration system and that fixing it is compounded by some other issues that nobody foresaw. Huge events like this are always a perfect storm of several factors, any one or two of which would be a total noop alone.
- KuiN 5y ago
  The Swiss cheese model of accidents. Occasionally the holes all align. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- soneil 5y ago
  The fun part of BGP is they apparently make a lot of use of it within their network, not just advertising routes externally. https://engineering.fb.com/2021/05/13/data-center-engineering/bgp/ https://engineering.fb.com/2021/05/13/data-center-engineerin... (and yes, fb.com resolves)