4 ms·

Yes we do, there are problems which beg to be fixed. Most of current complaints are about security, in one way or another require plenty of CPU horsepower (whic

Yes we do, there are problems which beg to be fixed. Most of current complaints are about security, in one way or another require plenty of CPU horsepower (which most routers don't have - CPU peaks while suddenly switching upling and retriving fullview is still a typical picture in a modern ISP). What it takes to prevent something like NLRI spoofing? Plenty of layers of data security. And it still won't fix the human problem underneath it. Security-aware BGP versions like S-BGP, soBGP and psBGP address most of security concerns, add plenty of router load and still... don't solve some of the human problems. But current publicly made suggestions on replacing it with something completely different even lack proper understanding of design goals of original BGP standard.

linsomniac 9y ago
Many years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers. We could throw tons of CPU horsepower at it and the fast path and the slow path were the same thing. We had deep Linux expertise, but only passing Cisco expertise, so staying with the Linux networking stack required less specialized knowledge. But, while most people were worrying about the size of the BGP tables, or trimming the announcements they were receiving, we could get full feeds and never had memory problems. We could run strong filtering, to make sure we weren't sending or receiving any junk like bogons or our users weren't sending with spoofed addresses, without worrying about adding one rule running past the ASIC memory and causing everything to go slow path. We had high availability. All for around $2500 in hardware. It always seemed crazy to me that they would put such small CPUs in the high end routers. I know they tried to have that CPU do almost nothing, and the ASICs do all the heavy lifting. But for our needs, having an insanely fast CPU and dumb interfaces was perfectly adequate, and let us do full BGP feeds on a $1200 router. The only real trick we did was we used one network interface and VLANs, and had the multiple network connections terminated at the switch. We were able to get feeds via fiber that we could just terminate in an SPDIF module in the switch. We had 4 core CPUs, so this allowed us plenty of horsepower to manage packet storms or DDoS instead of livelocking the kernel with too many interrupts (it took a while for the kernel to switch from interrupts to polling, and if packets ramped up too quickly the kernel couldn't switch if you didn't have more CPUs than interfaces).
- dephekt 9y ago
  I've done mostly the same thing you describe and it absolutely works and works well. You throw much less money into hardware, you don't need to rely on vendor-specific experts and expensive certification training and you don't have to pay out the nose for vendor maintenance agreements and licensing. It also requires that your people have real understanding of how routing and switching protocols actually work rather than being glorified [insert vendor] sales consultants. Add some programming and you have SDN and an incredible amount of power and flexibility with very little siloed, non-adaptable expertise.
- hhw 9y ago
  I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today. Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables. Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing. So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration. Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.
  - 9y ago

linsomniac 9y ago

Many years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers. We could throw tons of CPU horsepower at it and the fast path and the slow path were the same thing. We had deep Linux expertise, but only passing Cisco expertise, so staying with the Linux networking stack required less specialized knowledge. But, while most people were worrying about the size of the BGP tables, or trimming the announcements they were receiving, we could get full feeds and never had memory problems. We could run strong filtering, to make sure we weren't sending or receiving any junk like bogons or our users weren't sending with spoofed addresses, without worrying about adding one rule running past the ASIC memory and causing everything to go slow path. We had high availability. All for around $2500 in hardware. It always seemed crazy to me that they would put such small CPUs in the high end routers. I know they tried to have that CPU do almost nothing, and the ASICs do all the heavy lifting. But for our needs, having an insanely fast CPU and dumb interfaces was perfectly adequate, and let us do full BGP feeds on a $1200 router. The only real trick we did was we used one network interface and VLANs, and had the multiple network connections terminated at the switch. We were able to get feeds via fiber that we could just terminate in an SPDIF module in the switch. We had 4 core CPUs, so this allowed us plenty of horsepower to manage packet storms or DDoS instead of livelocking the kernel with too many interrupts (it took a while for the kernel to switch from interrupts to polling, and if packets ramped up too quickly the kernel couldn't switch if you didn't have more CPUs than interfaces).

dephekt 9y ago
I've done mostly the same thing you describe and it absolutely works and works well. You throw much less money into hardware, you don't need to rely on vendor-specific experts and expensive certification training and you don't have to pay out the nose for vendor maintenance agreements and licensing. It also requires that your people have real understanding of how routing and switching protocols actually work rather than being glorified [insert vendor] sales consultants. Add some programming and you have SDN and an incredible amount of power and flexibility with very little siloed, non-adaptable expertise.
hhw 9y ago
I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today. Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables. Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing. So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration. Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.
- 9y ago