4 ms·

The Internet hit 512K BGP routes today, causing widespread issues

21 points by mprovost 12y ago

yebyen 12y ago
The linked page "CIDR Report" shows a rrdtool graph that seems to show we had just as many routes 3 days ago. The page doesn't say anything about widespread issues, and I don't find any source for that when googling. So, where is your source about these widespread issues? (Link title at the time of this post: The Internet hit 512K BGP routes today, causing widespread issues)
- mburns 12y ago
  NANOG[0] warned about it in May, and there is a blog post [1] that expands on the risk. Cisco 6500/7600 (if they are set to hold the complete BGP table) are known to be affected. [0] http://mailman.nanog.org/pipermail/nanog/2014-May/066880.html http://mailman.nanog.org/pipermail/nanog/2014-May/066880.htm... [1] http://packetlife.net/blog/2014/may/6/global-ipv4-routing-table-hits-500k-routes/ http://packetlife.net/blog/2014/may/6/global-ipv4-routing-ta...
  - cpncrunch 12y ago
    Neither of those sources confirms the 'widespread issues' today.
mprovost 12y ago
As I understand it we came close a few days ago but finally rolled over 512K today at which point some routers hit their hardware limits and either stop learning new routes or start forwarding in software. There is a lot of chatter about it on Twitter and some network mailing lists: https://puck.nether.net/pipermail/outages/2014-August/007091.html https://puck.nether.net/pipermail/outages/2014-August/007091... It looks like some providers like Level3 have been having a bad day.
Zikes 12y ago
Can someone provide a lay explanation?
- namecast 12y ago
  Sure. Lots of old/older routers can't hold more than 512K IPv4 routes in their "really fast routing table" memory (TCAM). This means that they'll have to either drop some routes - meaning some places on the internet will not be accessible from those routers - or that they'll have to use slower memory to store those routes - which means routing in general will be slow across the board. Many, many ISPs, hosting and content providers use routers with this limitation. So their customers are going to be impacted, in some indeterminable way. (Network geeks - I've glossed over a point or two, I know. Caveat lector.)
  - Zikes 12y ago
    Is this limitation specific to IPv4? Would those same routers be able to handle a larger quantity of IPv6 addresses (if any), or does this highlight the need to upgrade those routers?
    - namecast 12y ago
      Ah, there's the rub! Assuming a Cisco 6500/7200 model router - the one most commonly affected by this issue - there's a certain amount of TCAM available by default for IPv4 routing, IPv6 routes, and multicast. By default, IPv4 only gets 512K routes. You can re-allocate more space to IPv4, but there's a potential cost... given the current rate of IPv6 adoption though, you could say that cost would be negligible for most people. More details here: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116132-problem-catalyst6500-00.html http://www.cisco.com/c/en/us/support/docs/switches/catalyst-... (The question I have is, why are people with these routers taking a full BGP feed of 512K routes? I'd assume that 90% of the time a default route will do, or even just default to your transit + their direct peer routes...)
    - yebyen 12y ago
      As a person of similar layness but some parts bit-wise... I think the answer is no, plain and simple because IPv6 addresses are literally four times as large. If we were using IPv6 for years leading up to this, would it still be an issue? Obviously not the same question, but IPv6 having much larger potential address space and more opportunities for abbreviation in expressions of "what is a netblock range" means for one thing that you don't always need as many network blocks, you can have larger network blocks where a smaller one would be used in IPv4 because of limits and efficiency with the overall available and provisioned blocks... Disclaimer: I have never been in charge of a large router, let alone a large network.

21 points by mprovost 12y ago

The linked page "CIDR Report" shows a rrdtool graph that seems to show we had just as many routes 3 days ago. The page doesn't say anything about widespread issues, and I don't find any source for that when googling. So, where is your source about these widespread issues? (Link title at the time of this post: The Internet hit 512K BGP routes today, causing widespread issues)

mburns 12y ago
NANOG[0] warned about it in May, and there is a blog post [1] that expands on the risk. Cisco 6500/7600 (if they are set to hold the complete BGP table) are known to be affected. [0] http://mailman.nanog.org/pipermail/nanog/2014-May/066880.html http://mailman.nanog.org/pipermail/nanog/2014-May/066880.htm... [1] http://packetlife.net/blog/2014/may/6/global-ipv4-routing-table-hits-500k-routes/ http://packetlife.net/blog/2014/may/6/global-ipv4-routing-ta...
- cpncrunch 12y ago
  Neither of those sources confirms the 'widespread issues' today.

mprovost 12y ago

As I understand it we came close a few days ago but finally rolled over 512K today at which point some routers hit their hardware limits and either stop learning new routes or start forwarding in software. There is a lot of chatter about it on Twitter and some network mailing lists: https://puck.nether.net/pipermail/outages/2014-August/007091.html https://puck.nether.net/pipermail/outages/2014-August/007091... It looks like some providers like Level3 have been having a bad day.

Zikes 12y ago

Can someone provide a lay explanation?

namecast 12y ago
Sure. Lots of old/older routers can't hold more than 512K IPv4 routes in their "really fast routing table" memory (TCAM). This means that they'll have to either drop some routes - meaning some places on the internet will not be accessible from those routers - or that they'll have to use slower memory to store those routes - which means routing in general will be slow across the board. Many, many ISPs, hosting and content providers use routers with this limitation. So their customers are going to be impacted, in some indeterminable way. (Network geeks - I've glossed over a point or two, I know. Caveat lector.)
- Zikes 12y ago
  Is this limitation specific to IPv4? Would those same routers be able to handle a larger quantity of IPv6 addresses (if any), or does this highlight the need to upgrade those routers?
  - namecast 12y ago
    Ah, there's the rub! Assuming a Cisco 6500/7200 model router - the one most commonly affected by this issue - there's a certain amount of TCAM available by default for IPv4 routing, IPv6 routes, and multicast. By default, IPv4 only gets 512K routes. You can re-allocate more space to IPv4, but there's a potential cost... given the current rate of IPv6 adoption though, you could say that cost would be negligible for most people. More details here: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116132-problem-catalyst6500-00.html http://www.cisco.com/c/en/us/support/docs/switches/catalyst-... (The question I have is, why are people with these routers taking a full BGP feed of 512K routes? I'd assume that 90% of the time a default route will do, or even just default to your transit + their direct peer routes...)
  - yebyen 12y ago
    As a person of similar layness but some parts bit-wise... I think the answer is no, plain and simple because IPv6 addresses are literally four times as large. If we were using IPv6 for years leading up to this, would it still be an issue? Obviously not the same question, but IPv6 having much larger potential address space and more opportunities for abbreviation in expressions of "what is a netblock range" means for one thing that you don't always need as many network blocks, you can have larger network blocks where a smaller one would be used in IPv4 because of limits and efficiency with the overall available and provisioned blocks... Disclaimer: I have never been in charge of a large router, let alone a large network.