4 ms·

DPDK is a very recent thing...

by xjia 8y ago

shaklee3 8y ago
It's been out for over 5 years, so not too recent.
corndoge 8y ago
As another said, been out for quite a while, more or less the standard kernel stack bypass, and certainly less recent than this BBC thing. Wish they'd spent their time contributing better MLX support to DPDK.
lttlrck 8y ago
Also netmap is extremely easy to get going. It’s a man page and some great examples and the deva are superbly helpful. I found DPDK daunting in comparison (it really is a different beast though, a real toolkit). One nice thing about Netmap is it can fall-back to emulation mode and work with any NIC. This can be really helpful if you want a single codepath but don’t always need the performance. DPDK might support this too now, I haven’t looked at it for quite a while.
- shaklee3 8y ago
  DPDK has great documentation and examples. I agree that it's more of a beast compared to netmap and pf_ring, but it's not lacking good documentation. Their documentation on each example are really well-done. DPDK can fall back to emulation mode as well. See the AF_PACKET driver.
  - abainbridge 8y ago
    Hmmm, I found /lots/ of documentation, but lots of gaps. For example, I still have no idea how many rings, how many queues and how many cores I should use for my packet forwarding app. Presumably there's a mental model I need to learn here. It's among the first thing any DPDK user needs to know. It's not the first thing in the docs. In the examples, its magic numbers all the way with no explanation. eg http://dpdk.org/doc/guides/sample_app_ug/skeleton.html http://dpdk.org/doc/guides/sample_app_ug/skeleton.html Also, I can't find a hello world like "send a packet" application, and it's complement of "receive a packet", only forwarding apps. That's not great when I'm trying to test if my config works. I still don't know how many huge pages I need to reserve. The quick start guide says 64: http://dpdk.org/doc/quick-start http://dpdk.org/doc/quick-start That didn't work. The Mellanox rep said 8192, which did work. That's a big difference. I can't find an explanation. I invested about 1% as much time in Netmap to get to the same level of "my app nearly works".
    - shaklee3 8y ago
      How many is really a question of how fast your processor is, how many cores, and what rates you're doing. They can't really give a solid answer since it varies so much, but their examples have defaults. Hello world apps are testpmd or dpdk-pktgen. Hugepages depends on how much memory your program is going to allocate. They also come in different sizes, so that example you gave is only 64 2MB pages, which is pretty small. I usually go for the largest hugepages possible in the processor I'm on (2GB) and allocate a small number of those. The reason being that most of my DPDK applications are the only things running on that machine, so I don't need to worry about sharing memory. I agree it's tougher to learn, but it's much more powerful. They likely need some kind of beginners' guide, since once you get past the tough parts, it's great.
      - abainbridge 8y ago
        > How many is really a question of how fast your processor is, how many cores, and what rates you're doing. FWIW, as far as I can tell, if you have no significant CPU work to do per packet, then you never need more than one core. A single core on a 5 year old laptop CPU can memcpy at a few hundred Gbit/s. In my tests I could even do the UDP checksum in software at 40 Gbit/s on one core. Mostly for me the bottleneck is the DDR RAM bandwidth, and using more cores doesn't increase that. The docs should explain why/how the defaults values were chosen in the example apps. I still don't know if there is any advantage to using more than 1 queue (aka ring) when using a single core. When I tried it, I found I could send nearly twice as many packets, but got lots of packet loss. Weird. I'm working on a public cloud, so the rules of what "line rate" really is are not available. There's some kind of fairness rules that either the hypervisor or the TOR enforces (I think). It's not at all clear if/how back-pressure is asserted to the NICs. > Hello world apps are testpmd or dpdk-pktgen. testpmd is 34,000 lines of code. dpdk-pktgen is 59,000. These are not hello world apps. I cannot read them to learn how to write a DPDK app. To understand if I had the basics right, I wrote a simple UDP sender and UDP echoer. They were 350 lines each. It seems like something similar should be the first apps everyone beginner sees. > Hugepages depends on how much memory your program is going to allocate Yeah, fair enough. I didn't explain my actual problem. My hello world programs worked fine with 64 huge pages on Intel cards. But I found no matter how small I made the ring buffers, 64 huge pages wasn't enough with Mellanox cards. And the error messages I got were gibberish from the bowels of the Mellanox driver (which isn't DPDK's fault). The Mellanox docs should have said how many hugepages their driver needs as an overhead (it appears to be more than 64).
- wmf 8y ago
  Netmap is easier, but when everyone other than you is on DPDK that's kind of a hint that you're painting yourself into a corner.
  - kierank 8y ago
    Except for minor services such as Verisign's DNS which uses netmap, sure.
    - shaklee3 8y ago
      Source? Couldn't find anything saying that.
  - lohohohobet 8y ago
    Netmap cannot (and does not want to) achieve the same performance of DPDK, but it has other advantages more than simpler installation, learning curve and flexibility (think of netmap pipes, netmap monitor ports and the VALE switch). Netmap applications can save CPU (sleep) when no traffic is needed, while DPDK applications must resort to busy waiting and need dedicated cores. This may be very handy for some applications, as you can find a sweet spot between high packet rate and CPU consumption. Moreover, Netmap uses the user<-->kernel interface to implement a layer of protection between the user-space application and the NIC hardware (performing validation on the application operations, which obviously comes with some cost). AFAIK, DPDK does not pay this cost, but allows unprotected access to the NIC hardware (which could do wild memory access through DMA). In terms of performance, benchmarks clearly show that DPDK has lower per-packet overhead (because it does less things), although netmap is not really far from those numbers. Point here is that benchmarks are often like free running engines; they show performance with no useful work. When you write a real application, you want to perform some useful work, which usually includes touching the packets, copying, compute packet headers etc; this results in expensive cache misses and CPU cycles that adds up to the basic I/O cost. If the real-work cost is something like 100ns per-packet (or more), then a difference in I/O cost of 5-10 nanoseconds per-packet may not really matter that much.

DPDK is a very recent thing...

by xjia 8y ago

DPDK is a very recent thing...

shaklee3 8y ago

It's been out for over 5 years, so not too recent.

corndoge 8y ago

As another said, been out for quite a while, more or less the standard kernel stack bypass, and certainly less recent than this BBC thing. Wish they'd spent their time contributing better MLX support to DPDK.

lttlrck 8y ago

Also netmap is extremely easy to get going. It’s a man page and some great examples and the deva are superbly helpful. I found DPDK daunting in comparison (it really is a different beast though, a real toolkit). One nice thing about Netmap is it can fall-back to emulation mode and work with any NIC. This can be really helpful if you want a single codepath but don’t always need the performance. DPDK might support this too now, I haven’t looked at it for quite a while.

shaklee3 8y ago
DPDK has great documentation and examples. I agree that it's more of a beast compared to netmap and pf_ring, but it's not lacking good documentation. Their documentation on each example are really well-done. DPDK can fall back to emulation mode as well. See the AF_PACKET driver.
- abainbridge 8y ago
  Hmmm, I found /lots/ of documentation, but lots of gaps. For example, I still have no idea how many rings, how many queues and how many cores I should use for my packet forwarding app. Presumably there's a mental model I need to learn here. It's among the first thing any DPDK user needs to know. It's not the first thing in the docs. In the examples, its magic numbers all the way with no explanation. eg http://dpdk.org/doc/guides/sample_app_ug/skeleton.html http://dpdk.org/doc/guides/sample_app_ug/skeleton.html Also, I can't find a hello world like "send a packet" application, and it's complement of "receive a packet", only forwarding apps. That's not great when I'm trying to test if my config works. I still don't know how many huge pages I need to reserve. The quick start guide says 64: http://dpdk.org/doc/quick-start http://dpdk.org/doc/quick-start That didn't work. The Mellanox rep said 8192, which did work. That's a big difference. I can't find an explanation. I invested about 1% as much time in Netmap to get to the same level of "my app nearly works".
  - shaklee3 8y ago
    How many is really a question of how fast your processor is, how many cores, and what rates you're doing. They can't really give a solid answer since it varies so much, but their examples have defaults. Hello world apps are testpmd or dpdk-pktgen. Hugepages depends on how much memory your program is going to allocate. They also come in different sizes, so that example you gave is only 64 2MB pages, which is pretty small. I usually go for the largest hugepages possible in the processor I'm on (2GB) and allocate a small number of those. The reason being that most of my DPDK applications are the only things running on that machine, so I don't need to worry about sharing memory. I agree it's tougher to learn, but it's much more powerful. They likely need some kind of beginners' guide, since once you get past the tough parts, it's great.
    - abainbridge 8y ago
      > How many is really a question of how fast your processor is, how many cores, and what rates you're doing. FWIW, as far as I can tell, if you have no significant CPU work to do per packet, then you never need more than one core. A single core on a 5 year old laptop CPU can memcpy at a few hundred Gbit/s. In my tests I could even do the UDP checksum in software at 40 Gbit/s on one core. Mostly for me the bottleneck is the DDR RAM bandwidth, and using more cores doesn't increase that. The docs should explain why/how the defaults values were chosen in the example apps. I still don't know if there is any advantage to using more than 1 queue (aka ring) when using a single core. When I tried it, I found I could send nearly twice as many packets, but got lots of packet loss. Weird. I'm working on a public cloud, so the rules of what "line rate" really is are not available. There's some kind of fairness rules that either the hypervisor or the TOR enforces (I think). It's not at all clear if/how back-pressure is asserted to the NICs. > Hello world apps are testpmd or dpdk-pktgen. testpmd is 34,000 lines of code. dpdk-pktgen is 59,000. These are not hello world apps. I cannot read them to learn how to write a DPDK app. To understand if I had the basics right, I wrote a simple UDP sender and UDP echoer. They were 350 lines each. It seems like something similar should be the first apps everyone beginner sees. > Hugepages depends on how much memory your program is going to allocate Yeah, fair enough. I didn't explain my actual problem. My hello world programs worked fine with 64 huge pages on Intel cards. But I found no matter how small I made the ring buffers, 64 huge pages wasn't enough with Mellanox cards. And the error messages I got were gibberish from the bowels of the Mellanox driver (which isn't DPDK's fault). The Mellanox docs should have said how many hugepages their driver needs as an overhead (it appears to be more than 64).
wmf 8y ago
Netmap is easier, but when everyone other than you is on DPDK that's kind of a hint that you're painting yourself into a corner.
- kierank 8y ago
  Except for minor services such as Verisign's DNS which uses netmap, sure.
  - shaklee3 8y ago
    Source? Couldn't find anything saying that.
- lohohohobet 8y ago
  Netmap cannot (and does not want to) achieve the same performance of DPDK, but it has other advantages more than simpler installation, learning curve and flexibility (think of netmap pipes, netmap monitor ports and the VALE switch). Netmap applications can save CPU (sleep) when no traffic is needed, while DPDK applications must resort to busy waiting and need dedicated cores. This may be very handy for some applications, as you can find a sweet spot between high packet rate and CPU consumption. Moreover, Netmap uses the user<-->kernel interface to implement a layer of protection between the user-space application and the NIC hardware (performing validation on the application operations, which obviously comes with some cost). AFAIK, DPDK does not pay this cost, but allows unprotected access to the NIC hardware (which could do wild memory access through DMA). In terms of performance, benchmarks clearly show that DPDK has lower per-packet overhead (because it does less things), although netmap is not really far from those numbers. Point here is that benchmarks are often like free running engines; they show performance with no useful work. When you write a real application, you want to perform some useful work, which usually includes touching the packets, copying, compute packet headers etc; this results in expensive cache misses and CPU cycles that adds up to the basic I/O cost. If the real-work cost is something like 100ns per-packet (or more), then a difference in I/O cost of 5-10 nanoseconds per-packet may not really matter that much.