3 ms·

DPDK Lib is good when you want to have user space app working with the IO bypassing the kernel. however it may greatly harm the system performance in the cases

DPDK Lib is good when you want to have user space app working with the IO bypassing the kernel. however it may greatly harm the system performance in the cases when kernel thread will hook the CPU resources and throw away you user space thread. This can introduce milliseconds of latency in your routing path. Also one question do you think to support L3 PFC support ? This is very important in the infrastructures where number of nodes > 1K.

alex_saroyan 5y ago
Right. We instruct Kernel to reserve 8 CPU cores exclusively for our user space application that is responsible for traffic forwarding. Here's performance results. Max latency is 0.31 msec. #################################################### min_latency_ns: 708 avg_latency_ns: 63723 max_latency_ns: 310795 jitter_ns: 30940 #################################################### Regarding L3 PFC support. In my understanding it's a ROCE requirement. I view RoCE as east-west traffic, since its not leaving the boundaries of the switch fabric, I think non-blocking leaf/spine fabric should be sufficient there. Please correct me if'm missing something. SoftGate, the Linux router where we use DPDK, is designed for North/South traffic, sitting on the border of the data center, so we don't expect RoCE traffic there, but again, please let me know if you see it differently.
- artavazd_rk 5y ago
  You have got a very good latency numbers. It's impressive. You are right about RoCE traffic. It is mainly East-West traffic. However NTT has a project called IOWN and in the scope of that project as I know they are trying to build North-South traffic based on the RoCE technology. In the case when you will decide to use your SoftGate routers inside the datacenter it would make sense to have a multipath routing capabilities and also solution for "PFC spreading blocking problem". This problem is not particularly bound with RoCE transport layer but instead for a MPI use cases. A several big infrastructure companies built their own network transport layer and in house hardware to solve those problems.

alex_saroyan 5y ago

Right. We instruct Kernel to reserve 8 CPU cores exclusively for our user space application that is responsible for traffic forwarding. Here's performance results. Max latency is 0.31 msec. #################################################### min_latency_ns: 708 avg_latency_ns: 63723 max_latency_ns: 310795 jitter_ns: 30940 #################################################### Regarding L3 PFC support. In my understanding it's a ROCE requirement. I view RoCE as east-west traffic, since its not leaving the boundaries of the switch fabric, I think non-blocking leaf/spine fabric should be sufficient there. Please correct me if'm missing something. SoftGate, the Linux router where we use DPDK, is designed for North/South traffic, sitting on the border of the data center, so we don't expect RoCE traffic there, but again, please let me know if you see it differently.

artavazd_rk 5y ago
You have got a very good latency numbers. It's impressive. You are right about RoCE traffic. It is mainly East-West traffic. However NTT has a project called IOWN and in the scope of that project as I know they are trying to build North-South traffic based on the RoCE technology. In the case when you will decide to use your SoftGate routers inside the datacenter it would make sense to have a multipath routing capabilities and also solution for "PFC spreading blocking problem". This problem is not particularly bound with RoCE transport layer but instead for a MPI use cases. A several big infrastructure companies built their own network transport layer and in house hardware to solve those problems.