3 ms·

I have both my own multihomed ASN and operate my own nameservers. The latter has usually been about as fast for failover overall in practice. BGP may look to co

I have both my own multihomed ASN and operate my own nameservers. The latter has usually been about as fast for failover overall in practice. BGP may look to converge near instantly from your 2-3 peer outbound perspective but the inbound convergence from the 100k networks on the rest of the internet is much slower and has a long tail very akin to trying to set your DNS TTL to 0 and having the rest of the internet decide to do it slower for cache/churn reasons anyways. The bigger problem, and where BGP multihoming is most handy, is it's just so much easier to get a holistic in+out failover where nothing really changes vs in DNS where it's more about getting the future inbound stuff to change where it goes. E.g. it's a pain to break an active session because the address had to change, even if DNS can update where the new service is quickly.

kortilla 9mo ago
The long tail of routers receiving your update doesn’t matter. Once the common transit networks get it, that’s where the rest would dump the traffic to reach you anyway. The only time slow propagation to the edges matters is the first time announcing a prefix after it has been fully withdrawn. Using the wrong route to get the packet in your general direction still gets you the packet as long as it hits an ISP along the way that got the update. We could fully drain traffic from a transit provider in <60s with a withdrawal with all of the major providers you get at the internet exchanges. If you weren’t seeing that your upstream ISPs may have penalized you for flapping too much and put in explicit delays.
- zamadatix 9mo ago
  <60s sounds about right as a general safe estimate. I just mean people should expect 1-2ish orders of magnitude more than <1s from a downed link with internet BGP upstreams in a multihomed situation.
  - kortilla 9mo ago
    I’m saying that’s not a correctly configured link for fast failure. <1 second was normal for hard link down events or explicit withdrawals. Anything above that was waiting for some BGP peer timeout or some IGP event. If your ISP is taking longer than 1 second to propagate your change, you’ve been put in some dunce protection box.
    - zamadatix 9mo ago
      If it were flap suppression/slow peer detection/"the dunce bucket" there wouldn't be a long tail of convergence - it'd just be nothing until all at once. This also isn't something I've only seen on my personal AS alone, it's what I've come to expect in many enterprise cutovers while previously working at a network VAR. The personal AS is however much more carefree to move around to different random providers on a whimthough of course :). I found some data from an oldish post by benjojo https://blog.benjojo.co.uk/post/speed-of-bgp-network-propagation https://blog.benjojo.co.uk/post/speed-of-bgp-network-propaga... which confirm various tirr 1s do propagate updates across their networks very fast (<2ish seconds) while others certainly do not. Notably, Level 3 (now Lumen) is the largest BGP presence by prefix count and was the worst tested in the list - starting to apply at ~20s after to finishing at ~50s after. This was for announce specifically, which should be the clearer case.

kortilla 9mo ago

The long tail of routers receiving your update doesn’t matter. Once the common transit networks get it, that’s where the rest would dump the traffic to reach you anyway. The only time slow propagation to the edges matters is the first time announcing a prefix after it has been fully withdrawn. Using the wrong route to get the packet in your general direction still gets you the packet as long as it hits an ISP along the way that got the update. We could fully drain traffic from a transit provider in <60s with a withdrawal with all of the major providers you get at the internet exchanges. If you weren’t seeing that your upstream ISPs may have penalized you for flapping too much and put in explicit delays.

zamadatix 9mo ago
<60s sounds about right as a general safe estimate. I just mean people should expect 1-2ish orders of magnitude more than <1s from a downed link with internet BGP upstreams in a multihomed situation.
- kortilla 9mo ago
  I’m saying that’s not a correctly configured link for fast failure. <1 second was normal for hard link down events or explicit withdrawals. Anything above that was waiting for some BGP peer timeout or some IGP event. If your ISP is taking longer than 1 second to propagate your change, you’ve been put in some dunce protection box.
  - zamadatix 9mo ago
    If it were flap suppression/slow peer detection/"the dunce bucket" there wouldn't be a long tail of convergence - it'd just be nothing until all at once. This also isn't something I've only seen on my personal AS alone, it's what I've come to expect in many enterprise cutovers while previously working at a network VAR. The personal AS is however much more carefree to move around to different random providers on a whimthough of course :). I found some data from an oldish post by benjojo https://blog.benjojo.co.uk/post/speed-of-bgp-network-propagation https://blog.benjojo.co.uk/post/speed-of-bgp-network-propaga... which confirm various tirr 1s do propagate updates across their networks very fast (<2ish seconds) while others certainly do not. Notably, Level 3 (now Lumen) is the largest BGP presence by prefix count and was the worst tested in the list - starting to apply at ~20s after to finishing at ~50s after. This was for announce specifically, which should be the clearer case.