3 ms·

You have to be root/suid, have to implement your netstack yourself (ip, tcp, etc.), have to use the special threads which may or may not play nice with other th

You have to be root/suid, have to implement your netstack yourself (ip, tcp, etc.), have to use the special threads which may or may not play nice with other threading libraries, locking and lock-free stuff, and there are subtle differences between controler behaviours that are smoothed over by the kernel and drivers, but hey you don't want those. For very simple forwarding and L2/L3 stuff it's easy to start and you can get to high perf quick enough. For anything applicative it can quickly become hard. I lost so much time with e.g. jumbo frames support, arp, frame re-ordering and low-level footguns, honestly sometimes I miss the pain of plain sockets. And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq on a (bunch of) 10Gb link(s). Or send to gpu via gpudirect, and the joy of programming DMAs instead of 'normal' cudaMemcpyAsync. And don't forget to reserve a (bunch of) physical machines for CI. I'm not saying it's not useful, on the contrary. You get to send/receive 800Gbit/s of traffic on a single socket system! But it feels like a setback in usability, learning curve. I'm saying I'd like a simpler API over it, to transmit/receive pub/sub large messages.

bogomipz 5y ago
Thanks for the wonderfully detailed response. I had a couple questions: >"And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq" How does the avx512 come into play here? I know its mainly used for HPC type workloads. Is there a more general usage pattern for it as well? I know Linux was very vocal about his disdain for avx512 and mentioned people were using it for memcpy[1] Is that the case for DPDK? Second is "zmq" here ZeroMQ, the messaging library? I would type of application space you work in. [1] https://www.techradar.com/news/linux-founder-chastises-intel-for-using-power-virus-tech https://www.techradar.com/news/linux-founder-chastises-intel...
- touisteur 5y ago
  Ah it's mostly signal/video processing, very high throughput, compute-heavy, very little branching. AVX was kind of off-topic there, sorry... AVX512 (and vector instructions) makes sense if you have very repetitive dense operations to do on your data. Think convolutions, filters, BLAS, FFT... Even better with dpdk since they run with hugepages so no tlb thrashing. DPDK has an option to use AVX512 for memory copy but I mostly do the computing + copy results in one pass, to maximize throughput. Zmq for zeromq yes, sorry. The abstraction (send arbitrary sized messages, receive messages, publish, subscribe, route messages... Same API for in-process, inter-process, inter-node) is so nice I miss it when we go down some levels of abstraction, especially with large large messages. I'm not sure I agree with Linus there. Intel is/was in a pinch, unable to ramp up FLOPS/mm2 and core count, they reached for vector instructions, in an extremely pragmatic 'what do you need them for - OK you'll get exactly those! And if you need something else, well... Have fun with the fun instruction set'. Intel also had (understandable) frequency throttling problems with how dense/hot the fused-multiply-add could get, but still painful and not detailed (discover weird frequency quirks on your 8000USD chip yay!) and have also been (to me) very unclear on the AVX512 roadmap (see recent Alderlake adventures). I get that Linus would like simpler, less kludgy solutions (more cores, more efficient cores, etc.) but if they exist in the cpu world, I'm waiting. Maybe ARM SVE whenever it will hit mainstream, maybe... The way for more raw flops is for the moment GPUs/accelerators - TPU, tenstorrent, FPGAs, mining ASICs being the extreme there - but they're their own cans of worms :-). So if you want very low latency (no going back and forth through PCIe and not motivated enough to use GPUdirect...) and still in the PC world, you're stuck. But AVX512 is actually a very good instruction set! Branch registers, most operations available, cross-lane instructions, all the shuffles you can imagine (well more than I can imagine) and recently avx512-fp16, the vectorized popcount... It's all getting better and better. A real shame we don't have more hardware with it - starting TigerLake you get it in consumer hardware! good luck with thermals but it gives a taste.

bogomipz 5y ago

Thanks for the wonderfully detailed response. I had a couple questions: >"And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq" How does the avx512 come into play here? I know its mainly used for HPC type workloads. Is there a more general usage pattern for it as well? I know Linux was very vocal about his disdain for avx512 and mentioned people were using it for memcpy[1] Is that the case for DPDK? Second is "zmq" here ZeroMQ, the messaging library? I would type of application space you work in. [1] https://www.techradar.com/news/linux-founder-chastises-intel-for-using-power-virus-tech https://www.techradar.com/news/linux-founder-chastises-intel...

touisteur 5y ago
Ah it's mostly signal/video processing, very high throughput, compute-heavy, very little branching. AVX was kind of off-topic there, sorry... AVX512 (and vector instructions) makes sense if you have very repetitive dense operations to do on your data. Think convolutions, filters, BLAS, FFT... Even better with dpdk since they run with hugepages so no tlb thrashing. DPDK has an option to use AVX512 for memory copy but I mostly do the computing + copy results in one pass, to maximize throughput. Zmq for zeromq yes, sorry. The abstraction (send arbitrary sized messages, receive messages, publish, subscribe, route messages... Same API for in-process, inter-process, inter-node) is so nice I miss it when we go down some levels of abstraction, especially with large large messages. I'm not sure I agree with Linus there. Intel is/was in a pinch, unable to ramp up FLOPS/mm2 and core count, they reached for vector instructions, in an extremely pragmatic 'what do you need them for - OK you'll get exactly those! And if you need something else, well... Have fun with the fun instruction set'. Intel also had (understandable) frequency throttling problems with how dense/hot the fused-multiply-add could get, but still painful and not detailed (discover weird frequency quirks on your 8000USD chip yay!) and have also been (to me) very unclear on the AVX512 roadmap (see recent Alderlake adventures). I get that Linus would like simpler, less kludgy solutions (more cores, more efficient cores, etc.) but if they exist in the cpu world, I'm waiting. Maybe ARM SVE whenever it will hit mainstream, maybe... The way for more raw flops is for the moment GPUs/accelerators - TPU, tenstorrent, FPGAs, mining ASICs being the extreme there - but they're their own cans of worms :-). So if you want very low latency (no going back and forth through PCIe and not motivated enough to use GPUdirect...) and still in the PC world, you're stuck. But AVX512 is actually a very good instruction set! Branch registers, most operations available, cross-lane instructions, all the shuffles you can imagine (well more than I can imagine) and recently avx512-fp16, the vectorized popcount... It's all getting better and better. A real shame we don't have more hardware with it - starting TigerLake you get it in consumer hardware! good luck with thermals but it gives a taste.