2 ms·
I guess it's easy to become complacent when you're a networking expert at Cloudflare and likely making several of these ad-hoc, on-the-fly config changes every
by jlgaddis 6y ago
I guess it's easy to become complacent when you're a networking expert at Cloudflare and likely making several of these ad-hoc, on-the-fly config changes every single day, but it's always good to remember why Juniper introduced the automatic rollback feature.
Of course, this particular outage would not have been prevented even if they had used
# commit confirmed
as it can't stop you from screwing up but it almost certainly would have limited the duration of the outage to ~10 minutes (plus a minute or two, perhaps, for the network to recover after the rollback) -- and it could have been shorter than that had they used "commit confirmed 3", for example.
Even as a lowly network engineer working on networks much, much smaller than Cloudflare's, for pretty much my entire professional career, it's been my standard practice to start off pretty much ANY change -- no matter how trivial -- with
# commit confirmed 3
or
# reload in 3
or similar, depending on what type of gear I was working on (and, of course, assuming the vendor supported such a feature).
This applies even when making a changes that are so "simple" that they just "can't" go wrong or have any unexpected or unintended effects.
In fact, it applies ESPECIALLY in those case! It's when you let your guard down that you'll get hit.
---
Fortunately, all that was necessary (I assume) to recover in this case was to
# rollback
to the previous configuration. Then, the correct configuration could be made. That still had to be done manually, however, and resulted in a 27 minute outage instead of what could have been a 5 or 10 minute outage.
I would hope that Cloudflare has full out-of-band access to all of their gear and are able to easily recover from mistakes like this. If they had lost access to the Atlanta router and weren't able to log in and revert the configuration manually, this outage could have lasted much, much longer.