4 ms·

PCI-E 3.0 x16 has 16GB/s bandwidth, being 200 Gbps 25 GB/s, so yes, it is not enough. Even having enough bandwidth, handling 25 GB/s data, except for hardware-

PCI-E 3.0 x16 has 16GB/s bandwidth, being 200 Gbps 25 GB/s, so yes, it is not enough. Even having enough bandwidth, handling 25 GB/s data, except for hardware-accelerated network processing, is very hard when current processors have 60 GB/s of RAM bandwidth (DDR4, dual channel), and even with zero-copy modern OS is going to be hard for the CPU to process the packets on continuous full load. That's mind blowing, and amazing :-)

greglindahl 9y ago
The way that networking at high rates is done in in the HPC/Supercomputing world is with user-level networking and OS-bypass, not "zero-copy modern OS".
- faragon 9y ago
  Sure, using hardware-accelerated packet routing for accelerating network protocols (e.g. Cavium SoCs).
  - shaklee3 9y ago
    He's referring to dpdk and the likes. Not Hardware off loads such as cavium or other special chips.
    - faragon 9y ago
      Ok, thank you.
    - jabl 9y ago
      Not really dpdk either, though I suppose there are similarities (Disclaimer: I know very little about dpdk). Think Infiniband (IB), not ethernet. Demanding applications such as MPI libraries or Lustre are written directly against the IB verbs interface, not TCP/IP with the sockets API. And yes, IB is designed such that the NIC HW can offload quite a lot, and the rest is indeed done in userspace without kernel involvement in the hot paths. That being said, it's possible to run more or less the IB protocol stack on ethernet hardware, it's called ROCE (RDMA over converged ethernet). Somewhat amazingly, latency is actually quite competitive with IB.
      - shaklee3 9y ago
        He mentioned hardware-accelerated packet routing. RoCE is not a routing protocol -- it's a UDP packet using IB verbs to DMA directly to hardware on the PCIe bus and bypass the processor. But yes, RoCE is a really interesting protocol and seems to have won the RoCE versus iWarp war.
shaklee3 9y ago
DDR has a much higher bandwidth than this. The old V4 Xeons could get close to 100 GBps, and the new skylakes out now have a 50% increase by going to six channels. On top of that, Intel has a feature where the packets coming off the wire are stored in cache first, so you could see much higher packet processing performance, depending on the type of application.
- snnn 9y ago
  Have you tested the numbers?
  - shaklee3 9y ago
    http://www.anandtech.com/show/8423/intel-xeon-e5-version-3-up-to-18-haswell http://www.anandtech.com/show/8423/intel-xeon-e5-version-3-u... https://www.pugetsystems.com/labs/hpc/Memory-Performance-for-Intel-Xeon-Haswell-EP-DDR4-596/ https://www.pugetsystems.com/labs/hpc/Memory-Performance-for...

greglindahl 9y ago

The way that networking at high rates is done in in the HPC/Supercomputing world is with user-level networking and OS-bypass, not "zero-copy modern OS".

faragon 9y ago
Sure, using hardware-accelerated packet routing for accelerating network protocols (e.g. Cavium SoCs).
- shaklee3 9y ago
  He's referring to dpdk and the likes. Not Hardware off loads such as cavium or other special chips.
  - faragon 9y ago
    Ok, thank you.
  - jabl 9y ago
    Not really dpdk either, though I suppose there are similarities (Disclaimer: I know very little about dpdk). Think Infiniband (IB), not ethernet. Demanding applications such as MPI libraries or Lustre are written directly against the IB verbs interface, not TCP/IP with the sockets API. And yes, IB is designed such that the NIC HW can offload quite a lot, and the rest is indeed done in userspace without kernel involvement in the hot paths. That being said, it's possible to run more or less the IB protocol stack on ethernet hardware, it's called ROCE (RDMA over converged ethernet). Somewhat amazingly, latency is actually quite competitive with IB.
    - shaklee3 9y ago
      He mentioned hardware-accelerated packet routing. RoCE is not a routing protocol -- it's a UDP packet using IB verbs to DMA directly to hardware on the PCIe bus and bypass the processor. But yes, RoCE is a really interesting protocol and seems to have won the RoCE versus iWarp war.

shaklee3 9y ago

DDR has a much higher bandwidth than this. The old V4 Xeons could get close to 100 GBps, and the new skylakes out now have a 50% increase by going to six channels. On top of that, Intel has a feature where the packets coming off the wire are stored in cache first, so you could see much higher packet processing performance, depending on the type of application.

snnn 9y ago
Have you tested the numbers?
- shaklee3 9y ago
  http://www.anandtech.com/show/8423/intel-xeon-e5-version-3-up-to-18-haswell http://www.anandtech.com/show/8423/intel-xeon-e5-version-3-u... https://www.pugetsystems.com/labs/hpc/Memory-Performance-for-Intel-Xeon-Haswell-EP-DDR4-596/ https://www.pugetsystems.com/labs/hpc/Memory-Performance-for...