5 ms·

What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc. You can SSH

What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc. You can SSH into your box. You can debug with gdb. Valgrind. Everything is normal...except the performance, which is just insane. Given how easy it is, there isn't really a good excuse anymore to not write data plane applications the "right" way, instead of jamming everything through the kernel like we've been doing. Especially with Intel's latest E5 processors, the performance is just phenomenal. If you want a fun, accessible project to play around with these concepts, Snabb Switch[0] makes it easy to write these kinds of apps with LuaJIT, which also has a super easy way to bind to C libraries. It's fast too: 40 million packets a second using a scripting language(!). I wrote a little bit about a recent project I completed that used these principles here: https://news.ycombinator.com/item?id=7231407 https://news.ycombinator.com/item?id=7231407 [0] https://github.com/SnabbCo/snabbswitch https://github.com/SnabbCo/snabbswitch

stefan_kendall 13y ago
Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.
- erichocean 13y ago
  The entire purpose is to NOT route stuff through the kernel. TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores. Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor, bi-passing the kernel (Ubuntu, Fedora, etc. isn't relevant in the least). So, you launch your app as a normal userland app. For each of your app's threads, schedule them on the remaining CPU cores however you want (I schedule one thread per core). Linux will not schedule its own threads or threads from any other process on those cores, so you own them completely—it'll never context switch to another thread. That means when you SSH in, it's running on a thread on cores 1 or 2 only. Same with every other Linux process but your own. Other than sucking up available memory bandwidth and potentially trashing your L2/L3 cache, these other processes don't impact your own app at all. Thus, even though you're running stock Linux, and SSH and gdb works, and you've got a normal userland app, your app is the ONLY app running on the remaining cores, and you're talking directly to the hardware. It's just as fast as doing everything without a kernel, except it cost you 2 cores. IMO, it's more than worth it for the convenience. This approach is so easy that there's really no reason not to do it. There are so many situations in the past where I wanted the performance of those single-app kernels, but it just wasn't worth the dev effort. That's no longer true.
  - rdtsc 13y ago
    Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Also cache effects. If your architecture shares caches between cores, sometimes it would be worth "wasting" a neighboring core to avoid ssh and gdb thrashing your cache. Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle. There is an interesting research done by Siemens, that takes this kind of isolation a step further and uses virtualization extensions to isolate resources (cores for example): https://github.com/siemens/jailhouse https://github.com/siemens/jailhouse
    - erichocean 13y ago
      > Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Have a look at cpuset[0]. You can forcibly move kernel threads into a given set (even pinned threads), and then force that set onto whatever cores you want. [0] http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset.7.html http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset...
      - rdtsc 13y ago
        Ah good stuff thanks posting that. I'll have to look into it, I haven't tried cpusets yet. In the past I tried migrating kernel threads, the migration did work but system became unstable, so I gave up on that.
    - nkurz 13y ago
      Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle. Perhaps this is what you meant, but this is straightforward. Simply disabling 'irqbalance' is a simple way to do this. Alternatively, you can also configure it to cooperate with 'isolcpus' by using 'FOLLOW_ISOLCPUS'.
      - rdtsc 13y ago
        Disabling it will disable auto-balancing it, it will just become static, which might not be the exact configuration you want.
- hueving 13y ago
  Do you have any data about what the bottleneck was?
nl 13y ago
Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).
- jsnell 13y ago
  Some NICs support virtualizing raw hardare access, so there's no fundamental reason for why Snabb couldn't support it. Just a simple matter of programming (and last I heard it was high on the priority list, so the support might even be there by now).
- _delirium 13y ago
  Another downside is that if you do networking in userspace, portability becomes your problem. If you do TCP/IP in the kernel, it works on nearly everything: just about any brand of Ethernet card, and even exotic non-Ethernet stuff (ISDN, whatever). If you do it in Snabb Switch, according to the wiki it currently supports exactly one class of interface: Intel-branded ethernet cards. Of course, I expect that will expand, but each one of these user-space networking stacks will have to ship with its own complete driver set to reach the same level of portability.
  - lukego 13y ago
    True. Or we might end up sharing drivers between projects too. In my experience high-end systems choose hardware to match software, so you only need to support one suitable option. The main reason we would add support for a new NIC is if it turns out to be better in some way e.g. as Mellanox support 40G now and Intel don't. In Snabb Switch have a lot of 10G/40G NICs online for people to play with now: https://groups.google.com/d/msg/snabb-devel/PXOsv0uLQCE/HjPjKT9h9q8J https://groups.google.com/d/msg/snabb-devel/PXOsv0uLQCE/HjPj...
  - nousernamesleft 13y ago
    Realistically that isn't much of an issue. If you are doing something at this level, you pick your hardware for the task. You don't need to support every random old dell PC someone wants to use like a general purpose OS has too (linux, the BSDs, etc). And of course, intel cards are not hard to find.
stock_toaster 13y ago
Userspace network stacks (like this[1]) on top of things like netmap[2] are also very cool. [1]: http://conferences.sigcomm.org/hotnets/2013/papers/hotnets-final43.pdf [2]: http://info.iet.unipi.it/~luigi/netmap/

stefan_kendall 13y ago

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

erichocean 13y ago
The entire purpose is to NOT route stuff through the kernel. TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores. Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor, bi-passing the kernel (Ubuntu, Fedora, etc. isn't relevant in the least). So, you launch your app as a normal userland app. For each of your app's threads, schedule them on the remaining CPU cores however you want (I schedule one thread per core). Linux will not schedule its own threads or threads from any other process on those cores, so you own them completely—it'll never context switch to another thread. That means when you SSH in, it's running on a thread on cores 1 or 2 only. Same with every other Linux process but your own. Other than sucking up available memory bandwidth and potentially trashing your L2/L3 cache, these other processes don't impact your own app at all. Thus, even though you're running stock Linux, and SSH and gdb works, and you've got a normal userland app, your app is the ONLY app running on the remaining cores, and you're talking directly to the hardware. It's just as fast as doing everything without a kernel, except it cost you 2 cores. IMO, it's more than worth it for the convenience. This approach is so easy that there's really no reason not to do it. There are so many situations in the past where I wanted the performance of those single-app kernels, but it just wasn't worth the dev effort. That's no longer true.
- rdtsc 13y ago
  Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Also cache effects. If your architecture shares caches between cores, sometimes it would be worth "wasting" a neighboring core to avoid ssh and gdb thrashing your cache. Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle. There is an interesting research done by Siemens, that takes this kind of isolation a step further and uses virtualization extensions to isolate resources (cores for example): https://github.com/siemens/jailhouse https://github.com/siemens/jailhouse
  - erichocean 13y ago
    > Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Have a look at cpuset[0]. You can forcibly move kernel threads into a given set (even pinned threads), and then force that set onto whatever cores you want. [0] http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset.7.html http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset...
    - rdtsc 13y ago
      Ah good stuff thanks posting that. I'll have to look into it, I haven't tried cpusets yet. In the past I tried migrating kernel threads, the migration did work but system became unstable, so I gave up on that.
  - nkurz 13y ago
    Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle. Perhaps this is what you meant, but this is straightforward. Simply disabling 'irqbalance' is a simple way to do this. Alternatively, you can also configure it to cooperate with 'isolcpus' by using 'FOLLOW_ISOLCPUS'.
    - rdtsc 13y ago
      Disabling it will disable auto-balancing it, it will just become static, which might not be the exact configuration you want.
hueving 13y ago
Do you have any data about what the bottleneck was?

nl 13y ago

Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

jsnell 13y ago
Some NICs support virtualizing raw hardare access, so there's no fundamental reason for why Snabb couldn't support it. Just a simple matter of programming (and last I heard it was high on the priority list, so the support might even be there by now).
_delirium 13y ago
Another downside is that if you do networking in userspace, portability becomes your problem. If you do TCP/IP in the kernel, it works on nearly everything: just about any brand of Ethernet card, and even exotic non-Ethernet stuff (ISDN, whatever). If you do it in Snabb Switch, according to the wiki it currently supports exactly one class of interface: Intel-branded ethernet cards. Of course, I expect that will expand, but each one of these user-space networking stacks will have to ship with its own complete driver set to reach the same level of portability.
- lukego 13y ago
  True. Or we might end up sharing drivers between projects too. In my experience high-end systems choose hardware to match software, so you only need to support one suitable option. The main reason we would add support for a new NIC is if it turns out to be better in some way e.g. as Mellanox support 40G now and Intel don't. In Snabb Switch have a lot of 10G/40G NICs online for people to play with now: https://groups.google.com/d/msg/snabb-devel/PXOsv0uLQCE/HjPjKT9h9q8J https://groups.google.com/d/msg/snabb-devel/PXOsv0uLQCE/HjPj...
- nousernamesleft 13y ago
  Realistically that isn't much of an issue. If you are doing something at this level, you pick your hardware for the task. You don't need to support every random old dell PC someone wants to use like a general purpose OS has too (linux, the BSDs, etc). And of course, intel cards are not hard to find.

stock_toaster 13y ago

Userspace network stacks (like this[1]) on top of things like netmap[2] are also very cool. [1]: http://conferences.sigcomm.org/hotnets/2013/papers/hotnets-final43.pdf [2]: http://info.iet.unipi.it/~luigi/netmap/