4 ms·

How do you propose to modify all of the ASICs in routers that are optimized to do lookups on packet headers at the speeds they need to achieve? Routers today a

by Nrsolis 14y ago

How do you propose to modify all of the ASICs in routers that are optimized to do lookups on packet headers at the speeds they need to achieve? Routers today are doing 32x100GbE in a single chassis. That's an investment in hardware that isn't going to go away any time soon. SCTP already solved this problem. Multipath TCP is an also-ran.

vy8vWJlco 14y ago
That's the clearest case I've seen for not buying "smart" hardware (firewalls, routers, etc) that know any more of the higher levels of the stack than necessary. I think the parent comment is valid though. If we wind up holding 2 or more TCP connections to multiplex, with failover, we are basically creating an overlay network that will have the same needs as the lower levels. Namely, routing, link cost metrics, fragmentation, etc... This is effectively what IP attempts to add on top of 802.3. With that in mind, it seems like it would be more faithful to the point (possible, if not practical) to use the Ethernet+IP stack directly on top of two TCP connections (providing simulated physical links), creating a throw-away network segment between two hosts that could re-use existing routing and bonding logic. I don't think Linux could handle that many virtual interfaces or taps, but that feels like the right place for it...
- jlgaddis 14y ago
  The devices handling 32x100GbE connections are only able to do that because of the very specialized ASICs that have been custom developed. I'm not aware of any device that is software-based that could come remotely close to handling that about of traffic. Most current PC-level hardware has trouble w/ 2x10GbE links (if they can even support that).
  - vy8vWJlco 14y ago
    I agree that specialization is often necessary, but I would prefer to think in terms of add-ons to general purpose commodity architectures. For example, GPUs are specialized for the task of 3D math, but their particular purpose is decided by the CPU. Similarly, FPGAs can be dynamically programmed for a special circuit. Neither is ideal, but in hindsight I would have liked to see something similar develop for the task of routing - and the economic pressure applied by not buying ASICs (that break the stack seperation) would have helped (yes we have benefited from them, but we have benefited from taking a specific path - we don't know if there was a better one). Routing probably would have benefited from an earlier and stronger push towards multicore processors - only there was no reason to do it in consumer CPUs because we view routing and computing largely as exclusive domains (even though the Internet's success has come from connecting many general purpose endpoints... Why is the fabric not also general purpose?). Going forward, I suspect GPUs could also be applied to make routing decisions with acceptable speeds. Have a look at http://www.date-conference.com/proceedings/PAPERS/2010/DATE10/PDFFILES/02.6_4.PDF http://www.date-conference.com/proceedings/PAPERS/2010/DATE1... for some trials.
    - mprovost 14y ago
      The specialised chips that are used by routers are CAMs and TCAMs. Basically, associative arrays (hashes or dictionaries as they're called in most scripting languages) implemented in hardware. You can program an FPGA to run as a CAM. Mostly what they're doing are repeated lookups of where to switch or route packets (based on header fields). If you're in a CPU world with RAM you can try and optimise the data structure that holds the lookup tables so you get decent speed but with CAM it's implemented in hardware so the lookup times are fixed and known. It's not really a limit of computation that can be helped with multiple cores, the CPUs can be pretty slow. Multithreading or parallel processing is typically handled on routers by distributing the workload to packet processors that handle a single port or a group of ports, all looking at the same CAM (or copies of it in the more distributed chassis-based architectures). Modern routers are basically supercomputers in a box. Of course there's an inherent limit at the PHY level where you can only receive or transmit a single packet at a time on an interface so you can't multiprocess beyond a CPU per port basically which limits the complexity of the design. 1000 cores on a GPU won't help you if you only have 100 ports to route traffic on. On a typical Intel PC, the MMU's TLB is implemented in CAM as well but it's tiny in comparison to the ones in routers. CAMs are pretty power hungry compared to RAM. They are becoming commodities though and the router manufacturers are getting away from doing custom ASIC designs. Intel bought Fulcrum which makes these packet processors and companies like Big Switch are doing cool stuff with white box network hardware. It's really going to bring the cost of the hardware down pretty dramatically in the next couple years.
      - vy8vWJlco 14y ago
        I'm looking forward to that!
- Nrsolis 14y ago
  Uh...these routers aren't "smart". They're doing the bare minimum to get a packet from an input interface to an output interface. Basically, ANYTHING you try to do at the speed and scale you'd end up needing to use these routers for would be at least as complicated if not more so. The level of engineering required to de-encapsulate a packet, deal with the L2(Ethernet),L2.5 (MPLS), L3(IP) headers, perform a lookup in the appropriate table (might be more than one), possibly deal with any encapsulations INSIDE that packet (cough-cough GRE) which starts the whole process over again is IMPRESSIVE. You're not going to do that in software, my friend. And anything you build that even begins to look like a general purpose processor will be unable to perform to the degree to which the custom ASICs already do. NPU (Network Processor Units) were supposed to make this easy but they never took off so now what few remain are doing things like NAT or Application Firewalls, and doing a relatively poor job for the money compared to a reasonable x86 server with appropriate I/O engineering. tl;dr: you're woefully ignorant about what it takes to make the Internet run fast.
  - vy8vWJlco 14y ago
    I didn't say it would be fast, only that it seemed like the right place for the specific goals of multipath TCP in order to minimize code duplication and overcome the various "smart" devices (a marketing word, not an accurate description) that pierce layer boundaries (like smart switches that look at IP headers, and routers that look at TCP/UDP flows) and have effectively rendered it impossible to develop new protocol semantics in lower layers (because they won't make it across the Internet; IPv6 could be depolyed by now if lots of ISPs didn't have "smart" devices blocking everything but IP+TCP+some special types of UDP). I blame such devices for making it unnecessarily complicated to develop multipath semantics over the current Internet. The rest of your comment is pretty insulting. :)
    - Nrsolis 14y ago
      Uh, you don't know what you're talking about. ISPs dont block UDP. And IPv6 was held up by a lot more than hardware. Sorry. Go read up on how IP routers actually function and then report back with what you've learned. Source: I've been doing this (networking) since before there was an Internet. I currently don't touch routers that do anything less than 10Gbps/port and really prefer to work with 100Gbps/port. That's a cool couple o'million bucks per chasis, yo.
caf 14y ago
The promise of Multipath TCP is that you don't have to change the applications or the network middleboxes - if the OS on your end and the remote end are both configured to use Multipath TCP, it'll used. SCTP requires either recompiling applications or using a TCP/SCTP proxy, and it doesn't solve the middlebox or discoverability problems. There's room for both solutions.

by Nrsolis 14y ago

vy8vWJlco 14y ago

That's the clearest case I've seen for not buying "smart" hardware (firewalls, routers, etc) that know any more of the higher levels of the stack than necessary. I think the parent comment is valid though. If we wind up holding 2 or more TCP connections to multiplex, with failover, we are basically creating an overlay network that will have the same needs as the lower levels. Namely, routing, link cost metrics, fragmentation, etc... This is effectively what IP attempts to add on top of 802.3. With that in mind, it seems like it would be more faithful to the point (possible, if not practical) to use the Ethernet+IP stack directly on top of two TCP connections (providing simulated physical links), creating a throw-away network segment between two hosts that could re-use existing routing and bonding logic. I don't think Linux could handle that many virtual interfaces or taps, but that feels like the right place for it...

jlgaddis 14y ago
The devices handling 32x100GbE connections are only able to do that because of the very specialized ASICs that have been custom developed. I'm not aware of any device that is software-based that could come remotely close to handling that about of traffic. Most current PC-level hardware has trouble w/ 2x10GbE links (if they can even support that).
- vy8vWJlco 14y ago
  I agree that specialization is often necessary, but I would prefer to think in terms of add-ons to general purpose commodity architectures. For example, GPUs are specialized for the task of 3D math, but their particular purpose is decided by the CPU. Similarly, FPGAs can be dynamically programmed for a special circuit. Neither is ideal, but in hindsight I would have liked to see something similar develop for the task of routing - and the economic pressure applied by not buying ASICs (that break the stack seperation) would have helped (yes we have benefited from them, but we have benefited from taking a specific path - we don't know if there was a better one). Routing probably would have benefited from an earlier and stronger push towards multicore processors - only there was no reason to do it in consumer CPUs because we view routing and computing largely as exclusive domains (even though the Internet's success has come from connecting many general purpose endpoints... Why is the fabric not also general purpose?). Going forward, I suspect GPUs could also be applied to make routing decisions with acceptable speeds. Have a look at http://www.date-conference.com/proceedings/PAPERS/2010/DATE10/PDFFILES/02.6_4.PDF http://www.date-conference.com/proceedings/PAPERS/2010/DATE1... for some trials.
  - mprovost 14y ago
    The specialised chips that are used by routers are CAMs and TCAMs. Basically, associative arrays (hashes or dictionaries as they're called in most scripting languages) implemented in hardware. You can program an FPGA to run as a CAM. Mostly what they're doing are repeated lookups of where to switch or route packets (based on header fields). If you're in a CPU world with RAM you can try and optimise the data structure that holds the lookup tables so you get decent speed but with CAM it's implemented in hardware so the lookup times are fixed and known. It's not really a limit of computation that can be helped with multiple cores, the CPUs can be pretty slow. Multithreading or parallel processing is typically handled on routers by distributing the workload to packet processors that handle a single port or a group of ports, all looking at the same CAM (or copies of it in the more distributed chassis-based architectures). Modern routers are basically supercomputers in a box. Of course there's an inherent limit at the PHY level where you can only receive or transmit a single packet at a time on an interface so you can't multiprocess beyond a CPU per port basically which limits the complexity of the design. 1000 cores on a GPU won't help you if you only have 100 ports to route traffic on. On a typical Intel PC, the MMU's TLB is implemented in CAM as well but it's tiny in comparison to the ones in routers. CAMs are pretty power hungry compared to RAM. They are becoming commodities though and the router manufacturers are getting away from doing custom ASIC designs. Intel bought Fulcrum which makes these packet processors and companies like Big Switch are doing cool stuff with white box network hardware. It's really going to bring the cost of the hardware down pretty dramatically in the next couple years.
    - vy8vWJlco 14y ago
      I'm looking forward to that!
Nrsolis 14y ago
Uh...these routers aren't "smart". They're doing the bare minimum to get a packet from an input interface to an output interface. Basically, ANYTHING you try to do at the speed and scale you'd end up needing to use these routers for would be at least as complicated if not more so. The level of engineering required to de-encapsulate a packet, deal with the L2(Ethernet),L2.5 (MPLS), L3(IP) headers, perform a lookup in the appropriate table (might be more than one), possibly deal with any encapsulations INSIDE that packet (cough-cough GRE) which starts the whole process over again is IMPRESSIVE. You're not going to do that in software, my friend. And anything you build that even begins to look like a general purpose processor will be unable to perform to the degree to which the custom ASICs already do. NPU (Network Processor Units) were supposed to make this easy but they never took off so now what few remain are doing things like NAT or Application Firewalls, and doing a relatively poor job for the money compared to a reasonable x86 server with appropriate I/O engineering. tl;dr: you're woefully ignorant about what it takes to make the Internet run fast.
- vy8vWJlco 14y ago
  I didn't say it would be fast, only that it seemed like the right place for the specific goals of multipath TCP in order to minimize code duplication and overcome the various "smart" devices (a marketing word, not an accurate description) that pierce layer boundaries (like smart switches that look at IP headers, and routers that look at TCP/UDP flows) and have effectively rendered it impossible to develop new protocol semantics in lower layers (because they won't make it across the Internet; IPv6 could be depolyed by now if lots of ISPs didn't have "smart" devices blocking everything but IP+TCP+some special types of UDP). I blame such devices for making it unnecessarily complicated to develop multipath semantics over the current Internet. The rest of your comment is pretty insulting. :)
  - Nrsolis 14y ago
    Uh, you don't know what you're talking about. ISPs dont block UDP. And IPv6 was held up by a lot more than hardware. Sorry. Go read up on how IP routers actually function and then report back with what you've learned. Source: I've been doing this (networking) since before there was an Internet. I currently don't touch routers that do anything less than 10Gbps/port and really prefer to work with 100Gbps/port. That's a cool couple o'million bucks per chasis, yo.

caf 14y ago

The promise of Multipath TCP is that you don't have to change the applications or the network middleboxes - if the OS on your end and the remote end are both configured to use Multipath TCP, it'll used. SCTP requires either recompiling applications or using a TCP/SCTP proxy, and it doesn't solve the middlebox or discoverability problems. There's room for both solutions.