5 ms·

The transport definition is nice and all but I'd love to see what the plan is for routing/switching at these speeds. Given the speed at which 100Gbe has limped

by mbell 14y ago

The transport definition is nice and all but I'd love to see what the plan is for routing/switching at these speeds. Given the speed at which 100Gbe has limped into the market I wouldn't expect to see anything capable of handling single links at these speeds for quite a long time. 1Tbps with with a 1500MTU is ~83 million packets a second. Just for illustrative comparison a CPU at 3.5Ghz would have 42 clock cycles per packet[0]. [0] I know a general purpose CPU would never be used for something like this, just putting the numbers into context.

jsnell 14y ago
You might be selling general purpose CPUs short. To me the computing hardly seems like the daunting part, it's the IO (both the wire interface, and all the internal buses). To verify that intuition, I just did a quick experiment with an Intel 10Gbps NIC and a Xeon E5506. The workload was roughly similar to switching: read packet, parse the headers, extract some fields from the headers and use that as a key to a hash table lookup, write packet out to a different interface. This was done using code from a much more complicated system that I didn't spend any time tuning for just this case, so it's absurdly wasteful (e.g. spending 10% of the runtime just on useless calls to RDTSC). The result was 4.3 Mpps, or about 500 cycles / packet. That seems far from 83 Mpps / 42 cycles, but it was using a single core and a single rx queue on the NIC. Make it parallel with 16 cores / queues, and you're in the right ballpark. And then remember that we're talking of a 3.5 year old CPU, which was already low end when it launched.
- noselasd 14y ago
  You're right that it's the IO (buses and memory) that the main limit. The problem is that a design around a central CPU doesn't scale, there's not enough IO, not enough pins, if you want to DMA packets from 24 interfaces onto main memory and have different cores operate on it is just infeasible with current designs.

by mbell 14y ago

jsnell 14y ago

You might be selling general purpose CPUs short. To me the computing hardly seems like the daunting part, it's the IO (both the wire interface, and all the internal buses). To verify that intuition, I just did a quick experiment with an Intel 10Gbps NIC and a Xeon E5506. The workload was roughly similar to switching: read packet, parse the headers, extract some fields from the headers and use that as a key to a hash table lookup, write packet out to a different interface. This was done using code from a much more complicated system that I didn't spend any time tuning for just this case, so it's absurdly wasteful (e.g. spending 10% of the runtime just on useless calls to RDTSC). The result was 4.3 Mpps, or about 500 cycles / packet. That seems far from 83 Mpps / 42 cycles, but it was using a single core and a single rx queue on the NIC. Make it parallel with 16 cores / queues, and you're in the right ballpark. And then remember that we're talking of a 3.5 year old CPU, which was already low end when it launched.

noselasd 14y ago
You're right that it's the IO (buses and memory) that the main limit. The problem is that a design around a central CPU doesn't scale, there's not enough IO, not enough pins, if you want to DMA packets from 24 interfaces onto main memory and have different cores operate on it is just infeasible with current designs.