5 ms·

To reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyon

To reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyone in due to network outage.

notyourday 5y ago
The first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points connected to IP over LTE or completely dedicated network of a different provider. We have been doing this since 1996
- packetslave 5y ago
  ...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)
  - notyourday 5y ago
    I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
    - packetslave 5y ago
      And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
      - deeblering4 5y ago
        weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
      - notyourday 5y ago
        > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.
- ImBanned 5y ago
  I thought the first rule of netops was, "Don't talk about netops." But this is a strong number 2.
  - no-s 5y ago
    The first rule of systems administration is don’t do anything you can’t back out of. This is just a derivative rule that happens when risk management goes wrong, eg if nobody noticed, it didn’t happen. Sort of a weak invariant principle…
- deckard1 5y ago
  I think we've all known this since 1968. When HAL refused to open the pod bay doors for Dave.
easton 5y ago
Isn’t the answer here if you can’t get someone with keys in a reasonable amount of time to break a window? Most doors have windows adjacent to them, and the company can just call the security people and tell them it’s part of DR.
- ashtonkem 5y ago
  Kind of! Any secured area isn’t going to have easily broken windows, but there’s also no such thing as an intrusion proof building. Ultimately the main point of any security system is to slow down an attacker before people with guns show up and pointedly ask you to stop doing that. If it’s your building, you can use a blowtorch to get in if you need to, it just won’t be as fast as say, breaking into a residential home.
  - no-s 5y ago
    AKA always carry a wrecking bar and a 5 pound sledgehammer in the data center quick reaction kit.
- jws 5y ago
  Rent a gasoline powered demolition saw with a diamond blade for $55 at Home Depot. Solve problem. Explain later.
- isbvhodnvemrwvn 5y ago
  My company is not Facebook, but in our data center there are no windows and the doors and walls in specific areas are designed to hold off physical attacks.
- perihelions 5y ago
  I think you're supposed to try HVAC first. https://xkcd.com/705/ https://xkcd.com/705/
- sectumsempra 5y ago
  a break-glass policy if you will
jedberg 5y ago
I worked at eBay many years ago. At the time, the process to get into the datacenter involved multiple ID checks by both humans and computers and multiple authentication factors. The process took about 10 minutes from walking in the front door to actually being in the cage. One day we had a major outage. The first few engineers to arrive were started on the 10 minute process. As more engineers arrived, a line started to form out the door. Then the VP showed up. He told the poor security guard that if they didn't open all the doors right now, eBay would find a new datacenter (we were probably more than 1/2 the space at the time). They used the emergency fire release to just unlock all the doors at once, and then called the fire department and told them to ignore the alarm. So now all the doors were open and if there were a fire no one would come. That day, if you happen to know where the eBay datacenter was, you could just walk right in. The only check against it was that most of use who had DC access knew everyone else who did and could call out imposters, but you'd have to stop what you were doing and think about it first. The point is, there is always a way to avoid security when needed.
- Johnny555 5y ago
  I wonder how many of the customers colocated in the other half of the datacenter decided to leave or seek compensation when they found that that the datacenter didn't have the physical security that they were promised. I'm pretty surprised that anyone was on-site that was authorized to make a snap decision that affects contracts with every customer colocated there.
  - jedberg 5y ago
    Well the front desk security guard probably didn't have the authority but got bullied into using the emergency fire system (which everyone has access to for safety reasons) to open all the doors. I don't remember what happened with the other customers. They were all in a single shared cage, so it's possible they just locked that cage back up. Or they just didn't know. Usually the datacenter was pretty empty -- even then you didn't really go all that often.
    - Johnny555 5y ago

To reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyone in due to network outage.

notyourday 5y ago

The first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points connected to IP over LTE or completely dedicated network of a different provider. We have been doing this since 1996

packetslave 5y ago
...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)
- notyourday 5y ago
  I'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
  - packetslave 5y ago
    And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
    - deeblering4 5y ago
      weekly? just maintain an inventory of deployed systems and test out of band connectivity every few minutes with standard black box monitoring
    - notyourday 5y ago
      > And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.
ImBanned 5y ago
I thought the first rule of netops was, "Don't talk about netops." But this is a strong number 2.
- no-s 5y ago
  The first rule of systems administration is don’t do anything you can’t back out of. This is just a derivative rule that happens when risk management goes wrong, eg if nobody noticed, it didn’t happen. Sort of a weak invariant principle…
deckard1 5y ago
I think we've all known this since 1968. When HAL refused to open the pod bay doors for Dave.

easton 5y ago

Isn’t the answer here if you can’t get someone with keys in a reasonable amount of time to break a window? Most doors have windows adjacent to them, and the company can just call the security people and tell them it’s part of DR.

ashtonkem 5y ago
Kind of! Any secured area isn’t going to have easily broken windows, but there’s also no such thing as an intrusion proof building. Ultimately the main point of any security system is to slow down an attacker before people with guns show up and pointedly ask you to stop doing that. If it’s your building, you can use a blowtorch to get in if you need to, it just won’t be as fast as say, breaking into a residential home.
- no-s 5y ago
  AKA always carry a wrecking bar and a 5 pound sledgehammer in the data center quick reaction kit.
jws 5y ago
Rent a gasoline powered demolition saw with a diamond blade for $55 at Home Depot. Solve problem. Explain later.
isbvhodnvemrwvn 5y ago
My company is not Facebook, but in our data center there are no windows and the doors and walls in specific areas are designed to hold off physical attacks.
perihelions 5y ago
I think you're supposed to try HVAC first. https://xkcd.com/705/ https://xkcd.com/705/
sectumsempra 5y ago
a break-glass policy if you will

jedberg 5y ago

I worked at eBay many years ago. At the time, the process to get into the datacenter involved multiple ID checks by both humans and computers and multiple authentication factors. The process took about 10 minutes from walking in the front door to actually being in the cage. One day we had a major outage. The first few engineers to arrive were started on the 10 minute process. As more engineers arrived, a line started to form out the door. Then the VP showed up. He told the poor security guard that if they didn't open all the doors right now, eBay would find a new datacenter (we were probably more than 1/2 the space at the time). They used the emergency fire release to just unlock all the doors at once, and then called the fire department and told them to ignore the alarm. So now all the doors were open and if there were a fire no one would come. That day, if you happen to know where the eBay datacenter was, you could just walk right in. The only check against it was that most of use who had DC access knew everyone else who did and could call out imposters, but you'd have to stop what you were doing and think about it first. The point is, there is always a way to avoid security when needed.

Johnny555 5y ago
I wonder how many of the customers colocated in the other half of the datacenter decided to leave or seek compensation when they found that that the datacenter didn't have the physical security that they were promised. I'm pretty surprised that anyone was on-site that was authorized to make a snap decision that affects contracts with every customer colocated there.
- jedberg 5y ago
  Well the front desk security guard probably didn't have the authority but got bullied into using the emergency fire system (which everyone has access to for safety reasons) to open all the doors. I don't remember what happened with the other customers. They were all in a single shared cage, so it's possible they just locked that cage back up. Or they just didn't know. Usually the datacenter was pretty empty -- even then you didn't really go all that often.
  - Johnny555 5y ago