4 ms·

Many years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers. We could throw tons of CP

Many years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers. We could throw tons of CPU horsepower at it and the fast path and the slow path were the same thing. We had deep Linux expertise, but only passing Cisco expertise, so staying with the Linux networking stack required less specialized knowledge. But, while most people were worrying about the size of the BGP tables, or trimming the announcements they were receiving, we could get full feeds and never had memory problems. We could run strong filtering, to make sure we weren't sending or receiving any junk like bogons or our users weren't sending with spoofed addresses, without worrying about adding one rule running past the ASIC memory and causing everything to go slow path. We had high availability. All for around $2500 in hardware. It always seemed crazy to me that they would put such small CPUs in the high end routers. I know they tried to have that CPU do almost nothing, and the ASICs do all the heavy lifting. But for our needs, having an insanely fast CPU and dumb interfaces was perfectly adequate, and let us do full BGP feeds on a $1200 router. The only real trick we did was we used one network interface and VLANs, and had the multiple network connections terminated at the switch. We were able to get feeds via fiber that we could just terminate in an SPDIF module in the switch. We had 4 core CPUs, so this allowed us plenty of horsepower to manage packet storms or DDoS instead of livelocking the kernel with too many interrupts (it took a while for the kernel to switch from interrupts to polling, and if packets ramped up too quickly the kernel couldn't switch if you didn't have more CPUs than interfaces).

dephekt 9y ago
I've done mostly the same thing you describe and it absolutely works and works well. You throw much less money into hardware, you don't need to rely on vendor-specific experts and expensive certification training and you don't have to pay out the nose for vendor maintenance agreements and licensing. It also requires that your people have real understanding of how routing and switching protocols actually work rather than being glorified [insert vendor] sales consultants. Add some programming and you have SDN and an incredible amount of power and flexibility with very little siloed, non-adaptable expertise.
hhw 9y ago
I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today. Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables. Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing. So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration. Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.
- linsomniac 9y ago
  So what is our option for securing BGP-like functionality? The thing that triggered me to tell this story in reply was the part about how most routers don't have the CPU to do the security necessary for more enhanced security. At the time I was deploying Pentium D in the multi-GHz range routers, the hot-shot Cisco routers were running 100MHz MIPS CPUs, IIRC. I know they put a ton of emphasis on the ASICs, but seems like they maybe need to splurge a bit on the CPUs. :-) That being said, I haven't really been doing much networking these days, I now just work for a single org, we let the facility do the heavy routing.
  - 9y ago
lima 9y ago
That worked fine back then, but in-kernel routing/switching does not scale beyond 1-2 Mpps. Much less if you have complicated iptables rules. We're getting there with DPDK and similar technologies, but right now, if you're running a high-bandwidth network, there's really no alternative to buying ASIC-based routing equipment from the big vendors. Even more if you're targeted by DDoS attacks, which represent the worst case: tens of gigabits of <100B packets. Source: work for a SaaS company, we tried it all

dephekt 9y ago

I've done mostly the same thing you describe and it absolutely works and works well. You throw much less money into hardware, you don't need to rely on vendor-specific experts and expensive certification training and you don't have to pay out the nose for vendor maintenance agreements and licensing. It also requires that your people have real understanding of how routing and switching protocols actually work rather than being glorified [insert vendor] sales consultants. Add some programming and you have SDN and an incredible amount of power and flexibility with very little siloed, non-adaptable expertise.

hhw 9y ago

I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today. Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables. Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing. So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration. Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.

linsomniac 9y ago
So what is our option for securing BGP-like functionality? The thing that triggered me to tell this story in reply was the part about how most routers don't have the CPU to do the security necessary for more enhanced security. At the time I was deploying Pentium D in the multi-GHz range routers, the hot-shot Cisco routers were running 100MHz MIPS CPUs, IIRC. I know they put a ton of emphasis on the ASICs, but seems like they maybe need to splurge a bit on the CPUs. :-) That being said, I haven't really been doing much networking these days, I now just work for a single org, we let the facility do the heavy routing.
- 9y ago

lima 9y ago

That worked fine back then, but in-kernel routing/switching does not scale beyond 1-2 Mpps. Much less if you have complicated iptables rules. We're getting there with DPDK and similar technologies, but right now, if you're running a high-bandwidth network, there's really no alternative to buying ASIC-based routing equipment from the big vendors. Even more if you're targeted by DDoS attacks, which represent the worst case: tens of gigabits of <100B packets. Source: work for a SaaS company, we tried it all