5 ms·

The first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points co

The first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points connected to IP over LTE or completely dedicated network of a different provider. We have been doing this since 1996

packetslave 5y ago
...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)
- notyourday 5y ago
  I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
  - packetslave 5y ago
    And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
    - deeblering4 5y ago
      weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
    - notyourday 5y ago
      > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.
ImBanned 5y ago
I thought the first rule of netops was, "Don't talk about netops." But this is a strong number 2.
- no-s 5y ago
  The first rule of systems administration is don’t do anything you can’t back out of. This is just a derivative rule that happens when risk management goes wrong, eg if nobody noticed, it didn’t happen. Sort of a weak invariant principle…
deckard1 5y ago
I think we've all known this since 1968. When HAL refused to open the pod bay doors for Dave.

packetslave 5y ago

...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)

notyourday 5y ago
I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
- packetslave 5y ago
  And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
  - deeblering4 5y ago
    weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
  - notyourday 5y ago
    > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.

ImBanned 5y ago

I thought the first rule of netops was, "Don't talk about netops." But this is a strong number 2.

no-s 5y ago
The first rule of systems administration is don’t do anything you can’t back out of. This is just a derivative rule that happens when risk management goes wrong, eg if nobody noticed, it didn’t happen. Sort of a weak invariant principle…

deckard1 5y ago

I think we've all known this since 1968. When HAL refused to open the pod bay doors for Dave.