4 ms·

When you have to bypass your operating system to get your hardware to perform, perhaps it is time to re-assess your choice of Operating system.

bwoj 11y ago
This isn't quite so much bypassing the OS as it is redefining the boundary of the privileged space to not include the network traffic. This lets your filtering application get the network packets directly without having to copy them out of kernel space and into user space. This is exactly the same technique all high performance network devices follow presently. The ones that aren't doing it in userspace, are doing it in some sort of RTOS that doesn't even have protected memory spaces.
- SixSigma 11y ago
  In hpc circles this scheme is called OS bypass http://blogs.cisco.com/performance/mpi-newbie-what-is-operating-system-bypass http://blogs.cisco.com/performance/mpi-newbie-what-is-operat...
  - lrizzo 11y ago
    (netmap author here) I prefer to define netmap as a "network stack bypass" scheme because we use as much as possible of the OS -- all the things it does well, we do not want to reinvent. Device drivers, system calls, synchronization support etc. are part of the kernel. Native netmap support for a NIC only involves 3-400 lines of code, or 10% of the typical device driver. Processes do ioctl(), mmap() and poll() for I/O - all standard system calls implemented by the OS, there is no NIC-specific code in the application. NICs can be switched in and out of netmap mode without reloading modules (and with the cloudflare patch, even sharing the two modes). There are no custom memory pools or hugepages to reserve. Device configuration relies on ethtool and ifconfig etc. This approach is what let the cloudflare folks implement their traffic steering with zero new code, just a couple of ethtool lines; the change they contributed back to support the split mode is completely agnostic of the specific NIC being used.
wglb 11y ago
Which OS would handle the load he speaks of?
- SixSigma 11y ago
  HPC also use OS Bypass so the plan9 team, who develop for Blue Gene and other large clusters, worked on currying system calls to maximize throughput http://4e.iwp9.org/papers/usecsys.pdf http://4e.iwp9.org/papers/usecsys.pdf
motoboi 11y ago
Greatest problem is copying data in and out kernel-space to user-space. This causes huge traffic on RAM and trashes caches all along the way.
- signa11 11y ago
  > Greatest problem is copying data in and out kernel-space to user-space. even if you could do away with copying e.g. packet_mmap (on linux), context switch will kill you...

bwoj 11y ago

This isn't quite so much bypassing the OS as it is redefining the boundary of the privileged space to not include the network traffic. This lets your filtering application get the network packets directly without having to copy them out of kernel space and into user space. This is exactly the same technique all high performance network devices follow presently. The ones that aren't doing it in userspace, are doing it in some sort of RTOS that doesn't even have protected memory spaces.

SixSigma 11y ago
In hpc circles this scheme is called OS bypass http://blogs.cisco.com/performance/mpi-newbie-what-is-operating-system-bypass http://blogs.cisco.com/performance/mpi-newbie-what-is-operat...
- lrizzo 11y ago
  (netmap author here) I prefer to define netmap as a "network stack bypass" scheme because we use as much as possible of the OS -- all the things it does well, we do not want to reinvent. Device drivers, system calls, synchronization support etc. are part of the kernel. Native netmap support for a NIC only involves 3-400 lines of code, or 10% of the typical device driver. Processes do ioctl(), mmap() and poll() for I/O - all standard system calls implemented by the OS, there is no NIC-specific code in the application. NICs can be switched in and out of netmap mode without reloading modules (and with the cloudflare patch, even sharing the two modes). There are no custom memory pools or hugepages to reserve. Device configuration relies on ethtool and ifconfig etc. This approach is what let the cloudflare folks implement their traffic steering with zero new code, just a couple of ethtool lines; the change they contributed back to support the split mode is completely agnostic of the specific NIC being used.

wglb 11y ago

Which OS would handle the load he speaks of?

SixSigma 11y ago
HPC also use OS Bypass so the plan9 team, who develop for Blue Gene and other large clusters, worked on currying system calls to maximize throughput http://4e.iwp9.org/papers/usecsys.pdf http://4e.iwp9.org/papers/usecsys.pdf

motoboi 11y ago

Greatest problem is copying data in and out kernel-space to user-space. This causes huge traffic on RAM and trashes caches all along the way.

signa11 11y ago
> Greatest problem is copying data in and out kernel-space to user-space. even if you could do away with copying e.g. packet_mmap (on linux), context switch will kill you...