5 ms·

The article appears to be inaccurate. Why? AFAIK it's possible to make use of netmap on a box with a single nic. I tried this out for myself about 2 years ago o

by s1m0n 11y ago

The article appears to be inaccurate. Why? AFAIK it's possible to make use of netmap on a box with a single nic. I tried this out for myself about 2 years ago on a VMware virtual machine. How does it work? A user land packet filter can "forward" certain packets on to the kernel -- e.g. ssh packets in my case -- while others stick around in shared memory for kernel bypass. This means that I can ssh to the box and the ssh packets flow via the kernel, while the rest of the packets bypass the kernel, but all packets floor over the same nic. Nice :-)

majke 11y ago
Well, sure. You can use netmap and use a "host ring" to inject the packets back to the kernel. While good for toy app, this won't work in real workloads. For example host ring doesn't support multiple RX queues. The article goes into details how to work around it and just leave most of the network flows to be dealt with by the kernel, while opting-in to the kernel bypass for only selected flows.
- s1m0n 11y ago
  This is the inaccurate sentence: "Snabbswitch, DPDK and netmap take over the whole network card, not allowing any traffic on that NIC to reach the kernel." Obviously with netmap traffic to the NIC may reach the kernel...
  - majke 11y ago
    There are many ways to inject packets back to kernel. Tuntap, raw socket on loopback, "dummy" device, etc. So by this count you can always make packets reach the kernel. There are two problems with doing the "take over the nic" techniques: 1) I don't believe you can actually push, say 2M pps back to the kernel with any of this techniques. There is a reason RSS exists, and even if you can process 10M pps on one CPU, it doesn't mean it's easy to insert them back to kernel. 2) I don't think putting a piece of custom code between CloudFlare kernel and network card is feasible on the architectural level. You really want to stand in the way and have to actively forward all these packets?
    - s1m0n 11y ago
      The title of the article does not mention CloudFlare; only bypassing. The fact that the CloudFlare architecture pushes a higher bandwidth of packets into the network kernel and bypasses the rest does not make it a good technique or to be recommended. If you are primarily interested in the best performance with a single NIC solution then I believe it is suboptimal. Why? You are asking the CPU to do two different types of work; optimized and unoptimized. Because of cache line pollution then the "unoptimized" work via the network kernel will pollute the other work. I may be wrong but I would bet you'd get better performance by separating your CloudFlare specific workload onto two boxes, each with one NIC. In this scenario then no cache line pollution can occur. Of course, these two boxes might not be easily possible within three existing CloudFlare architecture. But this has nothing to do with the general idea of packets bypassing the kernel. After the bypass you want the CPU to process those packets in the most efficient way...
- s1m0n 11y ago
  If you are only interested in pushing infrequently used ssh packets into the kernel for e.g. low bandwidth health monitoring -- while all other packets bypass there kernel -- then why would this be considered a "toy app"? Surely it's a useful technique because it allows netmap to be used on very many cheap dedicated hosts for rent where only one NIC is available and you have no control over the hardware, or?

by s1m0n 11y ago

majke 11y ago

Well, sure. You can use netmap and use a "host ring" to inject the packets back to the kernel. While good for toy app, this won't work in real workloads. For example host ring doesn't support multiple RX queues. The article goes into details how to work around it and just leave most of the network flows to be dealt with by the kernel, while opting-in to the kernel bypass for only selected flows.

s1m0n 11y ago
This is the inaccurate sentence: "Snabbswitch, DPDK and netmap take over the whole network card, not allowing any traffic on that NIC to reach the kernel." Obviously with netmap traffic to the NIC may reach the kernel...
- majke 11y ago
  There are many ways to inject packets back to kernel. Tuntap, raw socket on loopback, "dummy" device, etc. So by this count you can always make packets reach the kernel. There are two problems with doing the "take over the nic" techniques: 1) I don't believe you can actually push, say 2M pps back to the kernel with any of this techniques. There is a reason RSS exists, and even if you can process 10M pps on one CPU, it doesn't mean it's easy to insert them back to kernel. 2) I don't think putting a piece of custom code between CloudFlare kernel and network card is feasible on the architectural level. You really want to stand in the way and have to actively forward all these packets?
  - s1m0n 11y ago
    The title of the article does not mention CloudFlare; only bypassing. The fact that the CloudFlare architecture pushes a higher bandwidth of packets into the network kernel and bypasses the rest does not make it a good technique or to be recommended. If you are primarily interested in the best performance with a single NIC solution then I believe it is suboptimal. Why? You are asking the CPU to do two different types of work; optimized and unoptimized. Because of cache line pollution then the "unoptimized" work via the network kernel will pollute the other work. I may be wrong but I would bet you'd get better performance by separating your CloudFlare specific workload onto two boxes, each with one NIC. In this scenario then no cache line pollution can occur. Of course, these two boxes might not be easily possible within three existing CloudFlare architecture. But this has nothing to do with the general idea of packets bypassing the kernel. After the bypass you want the CPU to process those packets in the most efficient way...
s1m0n 11y ago
If you are only interested in pushing infrequently used ssh packets into the kernel for e.g. low bandwidth health monitoring -- while all other packets bypass there kernel -- then why would this be considered a "toy app"? Surely it's a useful technique because it allows netmap to be used on very many cheap dedicated hosts for rent where only one NIC is available and you have no control over the hardware, or?