4 ms·

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen when you make a globally impacting config change.

arter45 9mo ago
I’m not sure this would be a good idea in this kind of change. Flapping is bad in the networking world. Flapping BGP routes, specifically, is bad because it can stress all BGP routers involved to the point where they can “go crazy”. Routes are explicitly advertised, so if you keep changing the routes, you are tasking the router CPU to process new stuff, discard it and process new stuff. In fact, BGP route flaps are specifically the focus of an entire RFC: https://datatracker.ietf.org/doc/html/rfc2439 https://datatracker.ietf.org/doc/html/rfc2439 More in general, a flapping link (on/off/on/off) can really mess with TCP. Flapping in the networking world is not something you want to do intentionally.
- vlovich123 9mo ago
  Ok. You can flap it slower and less frequently. The RFC you mentioned talks about timers on the order of a minute or so. So I would say advertise the new route for 1 minute and unconditionally restore for 10. Then only after that advertise for 2 minutes and restore for 10. There’s clearly some interval of on/off that isn’t a problem and that’s an effective way to evaluate the impact you of a deployed route change gradually over time rather than fucking up the internet for 25 minutes until someone figures out what’s going on. And obviously you don’t do this on every individual route change - you batch them so it’s a release train. If you think there’s better techniques other than “don’t break things” I’m all for it.
  - arter45 9mo ago
    This specific outage is the equivalent of this scenario: You have an if/then statement with N conditions in AND. You remove one condition, leaving the rest of the statement unchanged. What happens? The answer is that if you remove one condition, your input (in this case routes) is more likely to match N-1 conditions than N, so more input is going to be processed according to the “then” clause. The impact of course depends on the fact that these were BGP routes, advertised to the Internet,… but the problem itself is generic. What can you do? 1) check this kind of if/then statements with special care, in order to analyze under which condition the input is processed by the “then” clause. This is exactly one of their followups [1] 2) consider adding “global”, catch-all policies acting as an additional safety net (if applicable) 3) test your changes not just syntactically. Set up a test environment with multiple routers, apply the configuration and see what happens. [1] Adding automatic routing policy evaluation into our CI/CD pipelines that looks specifically for empty or erroneous policy terms
    - vlovich123 9mo ago
      Yeah, but all of those basically boil down to “the next outage will look different from before” which fine but isn’t an actual solution IMO. My point is you want to do that and gradual rollouts that you don’t make permanent until you’ve observed the real world behavior if you want to prevent all future outages. This specific temporary rollout and automatic rollback also has the side effect that even if you don’t do any of the “hardening steps” outlined, your system will still prevent any kind of mistake you’ve made from rolling out and becoming more permanent. Like I said, the “flapping” parameters can be tuned however you want and you can aggregate updates into an automated “release train”. If you want you can do so with automated health metrics although it can be hard to implement automating validation that behavior before and after the route is “correct” (maybe trained ML models would be helpful here). This is btw in many ways how Google releases code into production - they bundle a bunch of PRs into a giant “publish” step - if CI fails or anything in production fails, they automatically rollback the entire set of changes since they can’t know which part of the release went bad. It’s a huge hammer to solve any issue they didn’t account for.
PunchyHamster 9mo ago
nice way to 100% the router CPUs for all your peers
eqvinox 9mo ago
> It gives you a cheap way to preflight what will happen when you make a globally impacting config change. Your "1-minute flap" can propagate and trigger load on every single DFZ BGP router on the planet. That's not cheap. And 1 minute is too short to even propagate across carriers. There are all kinds of timers working to reduce previous point; your update can still be propagating half an hour later. It can also change state for when you do it for real. And worst of all, BGP routes can get stuck. It's rare, but a real problem.
- vlovich123 9mo ago
  Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes (again - you’d batch the new route advertisements in one cohort rather than updating each individual route back and forth). And stuck routes are a problem but not one this would make worse since those routes would get stuck from normal changes anyway. The propagation problem isn’t real because clearly most route advertisements that handle most of the traffic actually happen quickly. You shouldn’t care about the long tail - you want to minimize the risk of your new route. The old route being present isn’t a problem and the new route disappearing back to the old also shouldn’t be a problem UNLESS the new route was buggy in which case you wanted to rollback anyway. TLDR: these don’t feel like risks unique to advertising and then undoing it given the route publishing already has to be handled anyway AND cloudflare is a major Tier 1 ISP and handles a good chunk of the entire internet’s traffic. This isn’t about a strategy for some random tier 2/3 ISP.
  - eqvinox 9mo ago
    > This isn’t about a strategy for some random tier 2/3 ISP. That's not a constraint you mentioned in your original post. > Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes […] I see you haven't had to deal with the operational reality of devices handling things they weren't quite designed for, and/or have been overdue for replacement, and/or were just designed to the limit to begin with. Good for you. But your solution would affect the entire internet. If you're serious, you could try posting your suggestion to the NANOG or RIPE mailing lists. At the very least you'll probably learn a whole new set of expletives and curses… but I'd recommend against it.

arter45 9mo ago

I’m not sure this would be a good idea in this kind of change. Flapping is bad in the networking world. Flapping BGP routes, specifically, is bad because it can stress all BGP routers involved to the point where they can “go crazy”. Routes are explicitly advertised, so if you keep changing the routes, you are tasking the router CPU to process new stuff, discard it and process new stuff. In fact, BGP route flaps are specifically the focus of an entire RFC: https://datatracker.ietf.org/doc/html/rfc2439 https://datatracker.ietf.org/doc/html/rfc2439 More in general, a flapping link (on/off/on/off) can really mess with TCP. Flapping in the networking world is not something you want to do intentionally.

vlovich123 9mo ago
Ok. You can flap it slower and less frequently. The RFC you mentioned talks about timers on the order of a minute or so. So I would say advertise the new route for 1 minute and unconditionally restore for 10. Then only after that advertise for 2 minutes and restore for 10. There’s clearly some interval of on/off that isn’t a problem and that’s an effective way to evaluate the impact you of a deployed route change gradually over time rather than fucking up the internet for 25 minutes until someone figures out what’s going on. And obviously you don’t do this on every individual route change - you batch them so it’s a release train. If you think there’s better techniques other than “don’t break things” I’m all for it.
- arter45 9mo ago
  This specific outage is the equivalent of this scenario: You have an if/then statement with N conditions in AND. You remove one condition, leaving the rest of the statement unchanged. What happens? The answer is that if you remove one condition, your input (in this case routes) is more likely to match N-1 conditions than N, so more input is going to be processed according to the “then” clause. The impact of course depends on the fact that these were BGP routes, advertised to the Internet,… but the problem itself is generic. What can you do? 1) check this kind of if/then statements with special care, in order to analyze under which condition the input is processed by the “then” clause. This is exactly one of their followups [1] 2) consider adding “global”, catch-all policies acting as an additional safety net (if applicable) 3) test your changes not just syntactically. Set up a test environment with multiple routers, apply the configuration and see what happens. [1] Adding automatic routing policy evaluation into our CI/CD pipelines that looks specifically for empty or erroneous policy terms
  - vlovich123 9mo ago
    Yeah, but all of those basically boil down to “the next outage will look different from before” which fine but isn’t an actual solution IMO. My point is you want to do that and gradual rollouts that you don’t make permanent until you’ve observed the real world behavior if you want to prevent all future outages. This specific temporary rollout and automatic rollback also has the side effect that even if you don’t do any of the “hardening steps” outlined, your system will still prevent any kind of mistake you’ve made from rolling out and becoming more permanent. Like I said, the “flapping” parameters can be tuned however you want and you can aggregate updates into an automated “release train”. If you want you can do so with automated health metrics although it can be hard to implement automating validation that behavior before and after the route is “correct” (maybe trained ML models would be helpful here). This is btw in many ways how Google releases code into production - they bundle a bunch of PRs into a giant “publish” step - if CI fails or anything in production fails, they automatically rollback the entire set of changes since they can’t know which part of the release went bad. It’s a huge hammer to solve any issue they didn’t account for.

PunchyHamster 9mo ago

nice way to 100% the router CPUs for all your peers

eqvinox 9mo ago

> It gives you a cheap way to preflight what will happen when you make a globally impacting config change. Your "1-minute flap" can propagate and trigger load on every single DFZ BGP router on the planet. That's not cheap. And 1 minute is too short to even propagate across carriers. There are all kinds of timers working to reduce previous point; your update can still be propagating half an hour later. It can also change state for when you do it for real. And worst of all, BGP routes can get stuck. It's rare, but a real problem.

vlovich123 9mo ago
Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes (again - you’d batch the new route advertisements in one cohort rather than updating each individual route back and forth). And stuck routes are a problem but not one this would make worse since those routes would get stuck from normal changes anyway. The propagation problem isn’t real because clearly most route advertisements that handle most of the traffic actually happen quickly. You shouldn’t care about the long tail - you want to minimize the risk of your new route. The old route being present isn’t a problem and the new route disappearing back to the old also shouldn’t be a problem UNLESS the new route was buggy in which case you wanted to rollback anyway. TLDR: these don’t feel like risks unique to advertising and then undoing it given the route publishing already has to be handled anyway AND cloudflare is a major Tier 1 ISP and handles a good chunk of the entire internet’s traffic. This isn’t about a strategy for some random tier 2/3 ISP.
- eqvinox 9mo ago
  > This isn’t about a strategy for some random tier 2/3 ISP. That's not a constraint you mentioned in your original post. > Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes […] I see you haven't had to deal with the operational reality of devices handling things they weren't quite designed for, and/or have been overdue for replacement, and/or were just designed to the limit to begin with. Good for you. But your solution would affect the entire internet. If you're serious, you could try posting your suggestion to the NANOG or RIPE mailing lists. At the very least you'll probably learn a whole new set of expletives and curses… but I'd recommend against it.