4 ms·

Hmm, I might be missing something here, but don't most high performance network applications skip the kernel for this exact reason? (IE http://highscalability.c

by edude03 11y ago

Hmm, I might be missing something here, but don't most high performance network applications skip the kernel for this exact reason? (IE http://highscalability.com/blog/2014/2/13/snabb-switch-skip-the-os-and-get-40-million-requests-per-sec.html http://highscalability.com/blog/2014/2/13/snabb-switch-skip-...) Makes me wonder how often bypassing the kernel is used in production networked applications.

kev009 11y ago
It's built into FreeBSD: https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 Netmap is commonly used in HFT as well as packet filtering applications. I believe Verisign is running some of the root DNS servers with netmap as well, getting millions of connections per second.
bobmagoo 11y ago
Yeah it is, Robert Graham gave a great talk about it over at https://www.youtube.com/watch?v=D09jdbS6oSI https://www.youtube.com/watch?v=D09jdbS6oSI with a writeup at http://c10m.robertgraham.com/2013/02/c10m-in-1990s.html http://c10m.robertgraham.com/2013/02/c10m-in-1990s.html
sj4nz 11y ago
There is a project "Kerlnel" http://kerlnel.org/ http://kerlnel.org/ that is supposed to be Erlang instance running on bare metal. Site appears to be down and the Github https://github.com/kerlnel https://github.com/kerlnel hasn't been touched since 2013. Then there is http://erlangonxen.org/ http://erlangonxen.org/ which puts Erlang on top of Xen instead of another operating system.
easytiger 11y ago
Well he should have compared this with onload enabled. He was using a solarflare after all. Limited benefits on mc compared with TCP though.
- sbanach 11y ago
  Yes, given the solarflare I was expecting the article to end up with a receiver coded against ef_vi, which exposes the NIC memory directly (but you have to do the IP/UDP yourself)
  - easytiger 11y ago
    Yea, solarcapture is essentially, i think, ef_vi consumer off to another thread.
pjc50 11y ago
He's "cheating" by using sendmmsg() to send many messages per system call, reducing the number of context switches.
- amluto 11y ago
  Barely. On a reasonably configured kernel (you need both syscall auditing and context tracking turned off, which is doable at compile time or runtime), a modern CPU should be able to round-trip a syscall in under 40 ns. That only eats 4% CPU at 1M syscalls per second. (It's slightly worse than that due to extra cache and TLB pressure, but I doubt that matters in this workload.)
scurvy 11y ago
Lots of High Frequency Trading (HFT) applications use an userland based IP stack.

by edude03 11y ago

kev009 11y ago

It's built into FreeBSD: https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 Netmap is commonly used in HFT as well as packet filtering applications. I believe Verisign is running some of the root DNS servers with netmap as well, getting millions of connections per second.

bobmagoo 11y ago

Yeah it is, Robert Graham gave a great talk about it over at https://www.youtube.com/watch?v=D09jdbS6oSI https://www.youtube.com/watch?v=D09jdbS6oSI with a writeup at http://c10m.robertgraham.com/2013/02/c10m-in-1990s.html http://c10m.robertgraham.com/2013/02/c10m-in-1990s.html

sj4nz 11y ago

There is a project "Kerlnel" http://kerlnel.org/ http://kerlnel.org/ that is supposed to be Erlang instance running on bare metal. Site appears to be down and the Github https://github.com/kerlnel https://github.com/kerlnel hasn't been touched since 2013. Then there is http://erlangonxen.org/ http://erlangonxen.org/ which puts Erlang on top of Xen instead of another operating system.

easytiger 11y ago

Well he should have compared this with onload enabled. He was using a solarflare after all. Limited benefits on mc compared with TCP though.

sbanach 11y ago
Yes, given the solarflare I was expecting the article to end up with a receiver coded against ef_vi, which exposes the NIC memory directly (but you have to do the IP/UDP yourself)
- easytiger 11y ago
  Yea, solarcapture is essentially, i think, ef_vi consumer off to another thread.

pjc50 11y ago

He's "cheating" by using sendmmsg() to send many messages per system call, reducing the number of context switches.

amluto 11y ago
Barely. On a reasonably configured kernel (you need both syscall auditing and context tracking turned off, which is doable at compile time or runtime), a modern CPU should be able to round-trip a syscall in under 40 ns. That only eats 4% CPU at 1M syscalls per second. (It's slightly worse than that due to extra cache and TLB pressure, but I doubt that matters in this workload.)

scurvy 11y ago

Lots of High Frequency Trading (HFT) applications use an userland based IP stack.