3 ms·

I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most

by hhw 9y ago

I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today. Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables. Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing. So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration. Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.

linsomniac 9y ago
So what is our option for securing BGP-like functionality? The thing that triggered me to tell this story in reply was the part about how most routers don't have the CPU to do the security necessary for more enhanced security. At the time I was deploying Pentium D in the multi-GHz range routers, the hot-shot Cisco routers were running 100MHz MIPS CPUs, IIRC. I know they put a ton of emphasis on the ASICs, but seems like they maybe need to splurge a bit on the CPUs. :-) That being said, I haven't really been doing much networking these days, I now just work for a single org, we let the facility do the heavy routing.
- hhw 9y ago
  Not sure how far back you're going, or which model of Cisco router you're referring to. I first cut my teeth on networking working at a Tier 2 carrier back in 2005 on Cisco GSR's (12000 series) which use a 200MHz R5000 MIPS CPU, but they were already quite long in the tooth at the time and were one of the few remaining networks still running them. And I do recall implementing ACL's to be an issue. Not sure if you were referring to ACL's or BGP policies themselves when referring to securing BGP. These days, a full BGP capable router starts with the ASR1001 for Cisco which starts with a 32bit 1.5GHz CPU + 4GB RAM on the RP1 and goes up to a 64bit quad-core 2.2GHz CPU + 8-64GB on the RP3. Juniper land, the lowest end non-EOS router would be the MX80 which comes with a 1.33GHz PPC CPU and 2GB RAM, but unofficially Juniper will always steer you towards the MX104 with a 1.8GHz CPU and 4GB of RAM. I can't comment on the ASR's, but we have MX80's and we can do line rate ACL's with hundreds of terms in ASICs without issue. BGP policies themselves are still handled by the CPU though, and are indeed quite slow on the MX80's with full convergence taking up to 20 minutes or so. We've mostly relegated them to core switching duty at this point though, and are waiting for the new MX204's (800Gbps in 1U) to become mature enough before replacing them. Moving up a bit, we use MX480's as well which we currently have routing engines with 2GHz Intel CPU's and 4GB of RAM, which I believe these are already EOS, but not enough of an issue for us to buy upgraded routing engines for this platform. These do full table re-convergence in about a minute or two. I believe quad-core 1.8GHz is standard now though, with up to hex-core 2GHz available. I don't think CPU's are really much of an issue anymore, although obviously still not as fast as what you can find easily enough in commodity hardware. The ASICs handle most things flawlessly though.
  - PatchMonkey 9y ago
    Well... As someone outside of that arm of the industry, I have to wonder about what exposure there is to spectre, or what kind of patches are coming out for affected machines.
    - hhw 9y ago
      JunOS is based on FreeBSD, which doesn't have a fix yet. Not sure about the different variants of Cisco IOS. You wouldn't run untrusted code on a router though, so spectre would not be a concern. In fact, it's probably better not to patch for it given the performance degradation.

by hhw 9y ago

linsomniac 9y ago

So what is our option for securing BGP-like functionality? The thing that triggered me to tell this story in reply was the part about how most routers don't have the CPU to do the security necessary for more enhanced security. At the time I was deploying Pentium D in the multi-GHz range routers, the hot-shot Cisco routers were running 100MHz MIPS CPUs, IIRC. I know they put a ton of emphasis on the ASICs, but seems like they maybe need to splurge a bit on the CPUs. :-) That being said, I haven't really been doing much networking these days, I now just work for a single org, we let the facility do the heavy routing.

hhw 9y ago
Not sure how far back you're going, or which model of Cisco router you're referring to. I first cut my teeth on networking working at a Tier 2 carrier back in 2005 on Cisco GSR's (12000 series) which use a 200MHz R5000 MIPS CPU, but they were already quite long in the tooth at the time and were one of the few remaining networks still running them. And I do recall implementing ACL's to be an issue. Not sure if you were referring to ACL's or BGP policies themselves when referring to securing BGP. These days, a full BGP capable router starts with the ASR1001 for Cisco which starts with a 32bit 1.5GHz CPU + 4GB RAM on the RP1 and goes up to a 64bit quad-core 2.2GHz CPU + 8-64GB on the RP3. Juniper land, the lowest end non-EOS router would be the MX80 which comes with a 1.33GHz PPC CPU and 2GB RAM, but unofficially Juniper will always steer you towards the MX104 with a 1.8GHz CPU and 4GB of RAM. I can't comment on the ASR's, but we have MX80's and we can do line rate ACL's with hundreds of terms in ASICs without issue. BGP policies themselves are still handled by the CPU though, and are indeed quite slow on the MX80's with full convergence taking up to 20 minutes or so. We've mostly relegated them to core switching duty at this point though, and are waiting for the new MX204's (800Gbps in 1U) to become mature enough before replacing them. Moving up a bit, we use MX480's as well which we currently have routing engines with 2GHz Intel CPU's and 4GB of RAM, which I believe these are already EOS, but not enough of an issue for us to buy upgraded routing engines for this platform. These do full table re-convergence in about a minute or two. I believe quad-core 1.8GHz is standard now though, with up to hex-core 2GHz available. I don't think CPU's are really much of an issue anymore, although obviously still not as fast as what you can find easily enough in commodity hardware. The ASICs handle most things flawlessly though.
- PatchMonkey 9y ago
  Well... As someone outside of that arm of the industry, I have to wonder about what exposure there is to spectre, or what kind of patches are coming out for affected machines.
  - hhw 9y ago
    JunOS is based on FreeBSD, which doesn't have a fix yet. Not sure about the different variants of Cisco IOS. You wouldn't run untrusted code on a router though, so spectre would not be a concern. In fact, it's probably better not to patch for it given the performance degradation.