5 ms·

Ixy – a userspace network driver in 1000 lines of code

122 points by ttflee 9y ago

userbinator 9y ago
Implement at least one other driver beside ixgbe Other NICs with documented and relatively simple interfaces include the (rather old) Ne2000, 8254x, and some of the Realtek ones: http://wiki.osdev.org/Category:Network_Hardware http://wiki.osdev.org/Category:Network_Hardware
- emmericp 9y ago
  The problem with these NICs is that they are PCI or even ISA. Also, they are quite rare compared to, e.g., an igb/e1000e NIC. A driver for an igb/e1000e NIC would be quite similar to the ixgbe driver. However, since the main goal of using these NICs seems to be support in virtual environments: just using virtio is better.
iamtew 9y ago
There was a talk about this at 34c4 earlier today. Demystifying Network Cards - https://events.ccc.de/congress/2017/Fahrplan/events/9159.html https://events.ccc.de/congress/2017/Fahrplan/events/9159.htm... - https://streaming.media.ccc.de/34c3/relive/9159 https://streaming.media.ccc.de/34c3/relive/9159
- bogomipz 9y ago
  Thanks for posting these links. On page 20 of the slide deck the author enumerates problems with existing user mode frameworks and one bullet point states: • Limited support for interrupts • Interrupts not considered useful at >= 0.1 Mpps Does anyone have any insight into why Interrupts aren't considers "useful" at this rate? Is this a reference to NAPI and the threshold at which its better to poll?
  - emmericp 9y ago
    The core of the problem is an optimization problem between saving power and optimizing for latency. Lets say you are receiving around 100k packets. You can fire 100k interrupts per second. Sure, no problem. But you'll probably be running at 100% CPU load. NAPI doesn't really help here. You'll see a quite high CPU load with NAPI (if measuring it correctly, CONFIG_IRQ_TIME_ACCOUNTING is often disabled), just not the horrible pre-NAPI live locks. What will help is a driver that limits the interrupt rate (Intel drivers do this by default, the option is called ITR). Now you've got a high latency instead (for some definitions of high, you'll see 40-100 µs with ixgbe). Note that the interrupts are now basically just a hardware timer. We don't need to use a NIC for hardware timers. This is of course only true at quite high packet rates (0.1 Mpps was maybe the wrong figure, let's say 1 Mpps). Interrupts are great at low packet rates. I've looked into this in some more detail in the Linux kernel a few years ago, see this paper for lots of details: https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/SPECTS15NAPIoptimization.pdf https://www.net.in.tum.de/fileadmin/bibtex/publications/pape... All the DPDK stuff about power saving is also quite interesting. Power saving is a relevant topic and DPDK is still quite bad at it. I think the most promising approach is dynamically adding and removing worker threads as well as controlling the CPU frequency from the application (that knows how loaded it actually is!) is currently the most promising approach. Unfortunately, most DPDK allocate threads and cores statically at the moment.
    - bogomipz 9y ago
      Thank you for the detailed explanation. Those bullets points make sense to me now. Cheers.
dvdplm 9y ago
Is it possible to try this out in a VM? I.e. are there any VMs out there that emulates the hardware well enough to be usable with Ixy?
- emmericp 9y ago
  I'm not aware of any hypervisors that can emulate a 82599 (ixgbe) NIC. I've a student who will start to work on a virtio driver soon to get a VM setup running.
- sigjuice 9y ago
  If you have a system with an extra 82599 perhaps you can use it in a VM via PCIe passthrough?
- en4bz 9y ago
  DPDK, which this library seems to be inspired by, has a virtio driver that will work for VMs. You can get into Intel 10GbE hardware for $100 USD [1]. [1] https://www.ebay.com/itm/Intel-Dell-X520-DA2-10Gb-10Gbe-10-Gigabit-Network-Adapter-NIC-Dual-E10G42BTDA/182583302702?epid=1272538247&hash=item2a82d01a2e:g:nxoAAOSwIQdZHwEX https://www.ebay.com/itm/Intel-Dell-X520-DA2-10Gb-10Gbe-10-G...
  - voltagex_ 9y ago
    How much is the cabling going to cost you? A switch?
    - tecleandor 9y ago
      Most expensive thing will be the switch, the cabling is not that expensive. For a 10g card to work you'll need a pair of SFP+ transceivers, one for the card, and another one for the other element you're connecting to (a switch, a router, another computer...) This transceiver can be chosen for copper or fiber depending on your needs, and each transceiver will be from 50 to 100$. Then you buy fiber or copper Cat6a cable. But, if your two devices are near (less than 10 meters), you can use a Twin Axial DAC (Direct Attach Cable) : you won't need the transceivers, you can just directly connect the two devices with this cable into the SFP+ port, and it's kind of cheap, for example, 12€ a 3meter run: https://www.fs.com/en/products/40141.html?currency=EUR https://www.fs.com/en/products/40141.html?currency=EUR
abecedarius 9y ago
From the draft paper: "It should be noted that the Snabb project [16] has similar design goals, ixy tries to be one order of magnitude simpler than Snabb. For example, Snabb targets 10,000 lines of code [15], we target 1,000 lines of code and Snabb builds on Lua with LuaJIT instead of C limiting accessibility." Snabb aims at practicality while Ixy aims at education, as I understand it, though I haven't read either one yet.
feelin_googley 9y ago
I listened to this talk and enjoyed it. Question for the author. It reminded me of something else, a number of years ago. Did you ever consider this: http://wiki.rumpkernel.org/Repo:-drv-netif-dpdk http://wiki.rumpkernel.org/Repo:-drv-netif-dpdk

122 points by ttflee 9y ago

Implement at least one other driver beside ixgbe Other NICs with documented and relatively simple interfaces include the (rather old) Ne2000, 8254x, and some of the Realtek ones: http://wiki.osdev.org/Category:Network_Hardware http://wiki.osdev.org/Category:Network_Hardware

emmericp 9y ago
The problem with these NICs is that they are PCI or even ISA. Also, they are quite rare compared to, e.g., an igb/e1000e NIC. A driver for an igb/e1000e NIC would be quite similar to the ixgbe driver. However, since the main goal of using these NICs seems to be support in virtual environments: just using virtio is better.

iamtew 9y ago

There was a talk about this at 34c4 earlier today. Demystifying Network Cards - https://events.ccc.de/congress/2017/Fahrplan/events/9159.html https://events.ccc.de/congress/2017/Fahrplan/events/9159.htm... - https://streaming.media.ccc.de/34c3/relive/9159 https://streaming.media.ccc.de/34c3/relive/9159

bogomipz 9y ago
Thanks for posting these links. On page 20 of the slide deck the author enumerates problems with existing user mode frameworks and one bullet point states: • Limited support for interrupts • Interrupts not considered useful at >= 0.1 Mpps Does anyone have any insight into why Interrupts aren't considers "useful" at this rate? Is this a reference to NAPI and the threshold at which its better to poll?
- emmericp 9y ago
  The core of the problem is an optimization problem between saving power and optimizing for latency. Lets say you are receiving around 100k packets. You can fire 100k interrupts per second. Sure, no problem. But you'll probably be running at 100% CPU load. NAPI doesn't really help here. You'll see a quite high CPU load with NAPI (if measuring it correctly, CONFIG_IRQ_TIME_ACCOUNTING is often disabled), just not the horrible pre-NAPI live locks. What will help is a driver that limits the interrupt rate (Intel drivers do this by default, the option is called ITR). Now you've got a high latency instead (for some definitions of high, you'll see 40-100 µs with ixgbe). Note that the interrupts are now basically just a hardware timer. We don't need to use a NIC for hardware timers. This is of course only true at quite high packet rates (0.1 Mpps was maybe the wrong figure, let's say 1 Mpps). Interrupts are great at low packet rates. I've looked into this in some more detail in the Linux kernel a few years ago, see this paper for lots of details: https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/SPECTS15NAPIoptimization.pdf https://www.net.in.tum.de/fileadmin/bibtex/publications/pape... All the DPDK stuff about power saving is also quite interesting. Power saving is a relevant topic and DPDK is still quite bad at it. I think the most promising approach is dynamically adding and removing worker threads as well as controlling the CPU frequency from the application (that knows how loaded it actually is!) is currently the most promising approach. Unfortunately, most DPDK allocate threads and cores statically at the moment.
  - bogomipz 9y ago
    Thank you for the detailed explanation. Those bullets points make sense to me now. Cheers.

dvdplm 9y ago

Is it possible to try this out in a VM? I.e. are there any VMs out there that emulates the hardware well enough to be usable with Ixy?

emmericp 9y ago
I'm not aware of any hypervisors that can emulate a 82599 (ixgbe) NIC. I've a student who will start to work on a virtio driver soon to get a VM setup running.
sigjuice 9y ago
If you have a system with an extra 82599 perhaps you can use it in a VM via PCIe passthrough?
en4bz 9y ago
DPDK, which this library seems to be inspired by, has a virtio driver that will work for VMs. You can get into Intel 10GbE hardware for $100 USD [1]. [1] https://www.ebay.com/itm/Intel-Dell-X520-DA2-10Gb-10Gbe-10-Gigabit-Network-Adapter-NIC-Dual-E10G42BTDA/182583302702?epid=1272538247&hash=item2a82d01a2e:g:nxoAAOSwIQdZHwEX https://www.ebay.com/itm/Intel-Dell-X520-DA2-10Gb-10Gbe-10-G...
- voltagex_ 9y ago
  How much is the cabling going to cost you? A switch?
  - tecleandor 9y ago
    Most expensive thing will be the switch, the cabling is not that expensive. For a 10g card to work you'll need a pair of SFP+ transceivers, one for the card, and another one for the other element you're connecting to (a switch, a router, another computer...) This transceiver can be chosen for copper or fiber depending on your needs, and each transceiver will be from 50 to 100$. Then you buy fiber or copper Cat6a cable. But, if your two devices are near (less than 10 meters), you can use a Twin Axial DAC (Direct Attach Cable) : you won't need the transceivers, you can just directly connect the two devices with this cable into the SFP+ port, and it's kind of cheap, for example, 12€ a 3meter run: https://www.fs.com/en/products/40141.html?currency=EUR https://www.fs.com/en/products/40141.html?currency=EUR

abecedarius 9y ago

From the draft paper: "It should be noted that the Snabb project [16] has similar design goals, ixy tries to be one order of magnitude simpler than Snabb. For example, Snabb targets 10,000 lines of code [15], we target 1,000 lines of code and Snabb builds on Lua with LuaJIT instead of C limiting accessibility." Snabb aims at practicality while Ixy aims at education, as I understand it, though I haven't read either one yet.

feelin_googley 9y ago

I listened to this talk and enjoyed it. Question for the author. It reminded me of something else, a number of years ago. Did you ever consider this: http://wiki.rumpkernel.org/Repo:-drv-netif-dpdk http://wiki.rumpkernel.org/Repo:-drv-netif-dpdk