3 ms·

...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and t

...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)

notyourday 5y ago
I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
- packetslave 5y ago
  And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
  - deeblering4 5y ago
    weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
  - notyourday 5y ago
    > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.

notyourday 5y ago

I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.

packetslave 5y ago
And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
- deeblering4 5y ago
  weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
- notyourday 5y ago
  > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.