4 ms·

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data cen

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

dr_orpheus 5y ago
Yeah, I think that is true. If you look at the Update near the end of the Cloudflare article there is a huge spike in the BGP activity (I assume re-announcing all of the routes). So that part of it was relatively instantaneous after they got all of their ducks in a row actually getting to the routers and locating the BGP from some earlier version before it went offline this morning that they could use.
rifkiamil 5y ago
We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management https://en.wikipedia.org/wiki/Out-of-band_management
- ampdepolymerase 5y ago
  HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.
  - jrochkind1 5y ago
    Who the hell put HN engineers are in charge of facebook, and why haven't we gotten more out of it than a temporary outage??
    - ampdepolymerase 5y ago
      Previously they had physical access to the data centers and weren't locked out.
- alfalfasprout 5y ago
  This. And if you're really worried about it you can go crazy with security with individually issued hardware security tokens and one time use access tokens. It's pretty inexcusable that FB wasn't able to use OOB management.
- allarm 5y ago
  It’s likely that their oobm systems are depending on dns in some way - aaa for instance. It’s not possible that they don’t have oobm at all, really.
huuuz 5y ago
Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.
- OJFord 5y ago
  Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.
  - wyldfire 5y ago
    I read that on-site badge readers denied access to employees! (Not necessarily the critical team to resolving the outage, but still...)
  - notbuying1 5y ago
    Not sure Im buying this, facebook have 131 data centers according to one post yesterday, I think they still had some form of OOB

dr_orpheus 5y ago

Yeah, I think that is true. If you look at the Update near the end of the Cloudflare article there is a huge spike in the BGP activity (I assume re-announcing all of the routes). So that part of it was relatively instantaneous after they got all of their ducks in a row actually getting to the routers and locating the BGP from some earlier version before it went offline this morning that they could use.

rifkiamil 5y ago

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management https://en.wikipedia.org/wiki/Out-of-band_management

ampdepolymerase 5y ago
HN engineers believe out of band admin control planes to be surveillance and backdoor firmware so they are disabled for privacy reasons.
- jrochkind1 5y ago
  Who the hell put HN engineers are in charge of facebook, and why haven't we gotten more out of it than a temporary outage??
  - ampdepolymerase 5y ago
    Previously they had physical access to the data centers and weren't locked out.
alfalfasprout 5y ago
This. And if you're really worried about it you can go crazy with security with individually issued hardware security tokens and one time use access tokens. It's pretty inexcusable that FB wasn't able to use OOB management.
allarm 5y ago
It’s likely that their oobm systems are depending on dns in some way - aaa for instance. It’s not possible that they don’t have oobm at all, really.

huuuz 5y ago

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

OJFord 5y ago
Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.
- wyldfire 5y ago
  I read that on-site badge readers denied access to employees! (Not necessarily the critical team to resolving the outage, but still...)
- notbuying1 5y ago
  Not sure Im buying this, facebook have 131 data centers according to one post yesterday, I think they still had some form of OOB