4 ms·

Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ https://blog.cloudf

Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...

alex-mohr 4y ago
It does seem like network configuration remains rather manual compared to other large scale systems that include more automation. In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead: - "We have blocked highly impactful commands from getting executed on the devices (Completed)" - "We will require all command execution on the devices to follow safe change guidelines (Estimated completion: February 2023)" Requiring commands to follow guidelines sounds suspiciously like they're requiring network ops not to break things.
- iso1631 4y ago
  That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.
  - jsz0 4y ago
    One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible documentation to bolster their support contract profits. I was just the guy typing in the commands and getting all the blame.
    - spookthesunset 4y ago
      I remember the first time I got access to an employers production Cisco router. It’s pretty scary how easy it is to majorly fuck something up. There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live. To counter this we’d write all the commands we planned on executing and peer review it. Nothing was to be done “on the fly” (at least in theory) In short, coming from a developer perspective with ample version controls and gated releases… networking is a very wild ride.
      - meltyness 4y ago
        Interesting. There's also some stuff in Cisco that can't be done both atomically and remotely, so you may have to push a change as a file to the router and then source the file into the running config with some permutation of `copy`.
      - simoncion 4y ago
        > There isn’t a concept of a transaction or a rollback. Yeah, Cisco gear is bonkers. Mikrotik has "Safe Mode", which undoes all commands since you entered "Safe Mode" if the connection that created the shell gets interrupted. It has saved my bacon on several occasions, but there are several obvious situations in which you can get yourself locked out. Juniper gear has "commit confirmed $NUMBER_OF_MINUTES", which will roll back everything since your last commit if you don't do a "commit" within $NUMBER_OF_MINUTES. It will also, apply all of the changes you've staged all at once (and do configuration sanity checking before it performs the commit). I do have no idea how Juniper's rollback works when multiple users are doing simultaneous config editing... maybe don't do that?
        iso1631 4y ago
        > I do have no idea how Juniper's rollback works when multiple users are doing simultaneous config editing... maybe don't do that? You get a warning Users currently editing the configuration: bob termainal p0...." But the failure here is actually sshing to a network switch in the first place. Some cisco kit has restconf which is better for automation, but it's buggy.
  - throw0101a 4y ago
    > That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo. Juniper JunOS has auto-rollback if you don't confirm the change after "x" minutes: * https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html#id-activating-a-junos-os-configuration-but-requiring-confirmation https://www.juniper.net/documentation/us/en/software/junos/c... So if you did something that causes breakage and disconnection from the router, you (ideally) don't have to do anything but wait it out.
    - AdamJacobMuller 4y ago
      commit confirmed is such a life-saver. I ran a production network which spanned multiple continents and even though I probably only ever actually needed commit confirmed a single digit number of times, the fact that it was there made every change I did 99% less stressful. I knew that even if I made a mistake, all I had to do was wait 5-10 minutes and it would all revert. Compare this to my cisco/foundry/other experience where I would delay changes until I was in the office (physically colocated with main routers) or calling people to be onsite for what was 99% of the time an innocuous change. The stress of it led to me deferring changes or just skipping them entirely which led to more issues/stress/etc. I'm really not sure there is a single software feature which improved my life as much as "commit confirmed"
      - iso1631 4y ago
        So instead of one ripple across your BGP network, you have two as it rollsback the change? The problem is that the state in routing tables isn't stored in a single location, it's dynamically built over time. Breaking a single router in the wrong way can break the state, and there's no rollback of that state
        AdamJacobMuller 4y ago
libraryatnight 4y ago
Holy shit I have been there and it sucks. I wasn't the guy who made the change, but I was on the long call that followed.
- int0x2e 4y ago
  Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE
  - libraryatnight 4y ago
    This was an awesome lunch listen, thank you for sharing!
latchkey 4y ago
How about describing how you implement systems that prevent this? You kind of talk about what was 'fixed', but not how. CI/CD is pretty hard to do for global networking changes. I'm sure whatever CF has done in this area is a lot of magic sauce and it would be super interesting to learn more about it, even at a high level.
newah1 4y ago
I remember this happening. The 20 some sites we ran went down as they were supported by cloudflare. I spent a panicked 30 minutes trying to figure out what I had done wrong, to eventually find out it was on CF's end. I remember voicing at our team meeting "boy, they must be panicking at CloudFlare." Cloudflare works so spectacularly we just wrote it off as a one time thing.
- jgrahamc 4y ago
  There was no panic but there was a lot of VUF (Very Urgent Focus)!
bogomipz 4y ago
Your CEO sure doesn't seem to have much empathy when it's someone else though: https://twitter.com/eastdakota/status/1143182575680143361 https://twitter.com/eastdakota/status/1143182575680143361

alex-mohr 4y ago

It does seem like network configuration remains rather manual compared to other large scale systems that include more automation. In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead: - "We have blocked highly impactful commands from getting executed on the devices (Completed)" - "We will require all command execution on the devices to follow safe change guidelines (Estimated completion: February 2023)" Requiring commands to follow guidelines sounds suspiciously like they're requiring network ops not to break things.

iso1631 4y ago
That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.
- jsz0 4y ago
  One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible documentation to bolster their support contract profits. I was just the guy typing in the commands and getting all the blame.
  - spookthesunset 4y ago
    I remember the first time I got access to an employers production Cisco router. It’s pretty scary how easy it is to majorly fuck something up. There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live. To counter this we’d write all the commands we planned on executing and peer review it. Nothing was to be done “on the fly” (at least in theory) In short, coming from a developer perspective with ample version controls and gated releases… networking is a very wild ride.
    - meltyness 4y ago
      Interesting. There's also some stuff in Cisco that can't be done both atomically and remotely, so you may have to push a change as a file to the router and then source the file into the running config with some permutation of `copy`.
    - simoncion 4y ago
      > There isn’t a concept of a transaction or a rollback. Yeah, Cisco gear is bonkers. Mikrotik has "Safe Mode", which undoes all commands since you entered "Safe Mode" if the connection that created the shell gets interrupted. It has saved my bacon on several occasions, but there are several obvious situations in which you can get yourself locked out. Juniper gear has "commit confirmed $NUMBER_OF_MINUTES", which will roll back everything since your last commit if you don't do a "commit" within $NUMBER_OF_MINUTES. It will also, apply all of the changes you've staged all at once (and do configuration sanity checking before it performs the commit). I do have no idea how Juniper's rollback works when multiple users are doing simultaneous config editing... maybe don't do that?
      - iso1631 4y ago
        > I do have no idea how Juniper's rollback works when multiple users are doing simultaneous config editing... maybe don't do that? You get a warning Users currently editing the configuration: bob termainal p0...." But the failure here is actually sshing to a network switch in the first place. Some cisco kit has restconf which is better for automation, but it's buggy.
- throw0101a 4y ago
  > That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo. Juniper JunOS has auto-rollback if you don't confirm the change after "x" minutes: * https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html#id-activating-a-junos-os-configuration-but-requiring-confirmation https://www.juniper.net/documentation/us/en/software/junos/c... So if you did something that causes breakage and disconnection from the router, you (ideally) don't have to do anything but wait it out.
  - AdamJacobMuller 4y ago
    commit confirmed is such a life-saver. I ran a production network which spanned multiple continents and even though I probably only ever actually needed commit confirmed a single digit number of times, the fact that it was there made every change I did 99% less stressful. I knew that even if I made a mistake, all I had to do was wait 5-10 minutes and it would all revert. Compare this to my cisco/foundry/other experience where I would delay changes until I was in the office (physically colocated with main routers) or calling people to be onsite for what was 99% of the time an innocuous change. The stress of it led to me deferring changes or just skipping them entirely which led to more issues/stress/etc. I'm really not sure there is a single software feature which improved my life as much as "commit confirmed"
    - iso1631 4y ago
      So instead of one ripple across your BGP network, you have two as it rollsback the change? The problem is that the state in routing tables isn't stored in a single location, it's dynamically built over time. Breaking a single router in the wrong way can break the state, and there's no rollback of that state
      - AdamJacobMuller 4y ago

libraryatnight 4y ago

Holy shit I have been there and it sucks. I wasn't the guy who made the change, but I was on the long call that followed.

int0x2e 4y ago
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE
- libraryatnight 4y ago
  This was an awesome lunch listen, thank you for sharing!

latchkey 4y ago

How about describing how you implement systems that prevent this? You kind of talk about what was 'fixed', but not how. CI/CD is pretty hard to do for global networking changes. I'm sure whatever CF has done in this area is a lot of magic sauce and it would be super interesting to learn more about it, even at a high level.

newah1 4y ago

I remember this happening. The 20 some sites we ran went down as they were supported by cloudflare. I spent a panicked 30 minutes trying to figure out what I had done wrong, to eventually find out it was on CF's end. I remember voicing at our team meeting "boy, they must be panicking at CloudFlare." Cloudflare works so spectacularly we just wrote it off as a one time thing.

jgrahamc 4y ago
There was no panic but there was a lot of VUF (Very Urgent Focus)!

bogomipz 4y ago

Your CEO sure doesn't seem to have much empathy when it's someone else though: https://twitter.com/eastdakota/status/1143182575680143361 https://twitter.com/eastdakota/status/1143182575680143361