4 ms·

So,curious here: I'v been reading about how the focus these days is to move networking code to userspace because you can squeeze out more PPS performance,does t

So,curious here: I'v been reading about how the focus these days is to move networking code to userspace because you can squeeze out more PPS performance,does the fact that WC makes use of kernel code heavily give it a performance disadvantage?

ahmedalsudani 7y ago
I don’t know what PPS is, but that is an inaccurate sentiment. Running as a kernel module allows you to achieve higher throughout and lower latency.
- lstamour 7y ago
  PPS means Packets Per Second. Here’s a press release from the FD.io project (“Fido”), take it as you will: https://fd.io/latest/singles/kubernetes/ https://fd.io/latest/singles/kubernetes/
  - dahfizz 7y ago
    FD.io uses the VPP stack under the hood, which ultimately uses DPDK[1] for the actual IO acceleration. The reason it's faster is because it's polling your hardware. You spend a lot of cpu time to buy the lower latency. [1] http://doc.dpdk.org/guides/prog_guide/overview.html http://doc.dpdk.org/guides/prog_guide/overview.html
    - lstamour 7y ago
      It's possible to write a driver that doesn't use polling - for example https://arxiv.org/pdf/1901.10664.pdf https://arxiv.org/pdf/1901.10664.pdf which discusses using vfio though doing so is apparently much harder than polling alone due to the complexity of the vfio interface and that most users care more about maximizing PPS performance at peak vs saving CPU cycles when less busy. Presumably one other option would be to try and tune the CPU to run slower when less demand is expected, and only run at faster peak performance when absolutely required. Also, it sounds like you might want to look at pinning cores to workloads, such that only one of your CPU cores is polling while the rest act as dynamic worker cores and could do other activities. Note that I'm largely speaking theoretically, for my use cases (mostly at home), the incompatibility of user space networking stacks often gets in the way.
      - dahfizz 7y ago
        > It's possible to write a driver that doesn't use polling Yup, definitely possible. It just doesn't give the PPS numbers as polling does. The project you linked to is really cool, but it is an academic project. It's not trying to compete with DPDK or anything. IO is one of those things in computer science that forces us to make trade ofs. What is more important to you? Raw throughput? Latency? Overall resource usage? You can't optimize all three. > it sounds like you might want to look at pinning cores to workloads, such that only one of your CPU cores is polling Correct again. It may take more than one cpu (DPDK is commercial software running on 88+ cpu systems) but cpu pinning is definitely the recommended way to go. In the spirit of spending CPU to buy IO speed, dedicating physical cores to the task always gives you the best bang for your buck.
- lern_too_spel 7y ago
  You're correct. I don't know why you're downvoted. Anything you can do to get performance with kernel bypass techniques can be done inside the kernel as well.
  - ahmedalsudani 7y ago
    ¯\_(ツ)_/¯ It’s alright. You sometimes get downvoted by the early two people who are offended that I’d dare comment without knowing what PPS is (or something equally offensive).
wmf 7y ago
A polling/batching implementation of WireGuard would be faster than the Linux kernel implementation although it would be much less convenient to use. If WireGuard spends, say, 90% of its time in ChaCha20-Poly1305 (which is already highly optimized) then there's only room for less than 10% speedup.
- Hello71 7y ago
  it could be faster with a hardware chacha20 implementation. if wireguard gets really popular, we might see that eventually. I'd wager a guess that we would get hardware wireguard first though, with fast path acceleration, similar to hardware TLS.
  - tatersolid 7y ago
    Modern CPUs are pretty close to being ASICs for ARX ciphers like ChaCha. Add, rotate, and XOR all have dedicated die area. Bernstein designed it that way. Are there any hardware ASIC implementations of ChaCha or are they basically unnecessary?
e_y_ 7y ago
You could always use a userspace implementation of WireGuard like https://blog.cloudflare.com/boringtun-userspace-wireguard-rust/ https://blog.cloudflare.com/boringtun-userspace-wireguard-ru... . I imagine the advantage of the kernel module is good performance for normal users who are not trying to wring every last bit of performance out of their network setup.
vbezhenar 7y ago
If all networking happens inside kernel, it'll be as fast as possible. People write userspace code because they don't want to put their application into kernel for obvious reasons. And with userspace code it makes sense to move everything into userspace, including network drivers, etc. But it's not a standard configuration and compromises on security.
lvh 7y ago
Most of the userspace high-performance networking stuff is still pretty exotic. It requires plenty of tuning, specific network hardware, and sometimes extreme conditions, to be viable. DPDK does not work on the wireless connection I'm using to post this reply, for example (nor any wireless connection for that matter). Meanwhile, if you're not doing that: Wireguard can avoid lots of syscall and copying overhead plus some magic CPU register overhead by being in the kernel. Hence this almost certainly makes wg faster on the net because it can't pick and choose what environments it runs in.
rkangel 7y ago
You're only half right - projects attempting to achieve very high speed networking (packets per second) go to great lengths to remove the traditional syscall per packet that normal linux networking involves. One approach to this is to move everything into userspace and take the kernel out of it completely. Another approach used by some projects is to put everything into the kernel and take userspace out of it completely. Either of these approaches remove syscalls. The kernel isn't an inherently slower place (in some cases it can be faster as you have more control), it is having your network stack split across the kernel/userspace boundary that is slow.

ahmedalsudani 7y ago

I don’t know what PPS is, but that is an inaccurate sentiment. Running as a kernel module allows you to achieve higher throughout and lower latency.

lstamour 7y ago
PPS means Packets Per Second. Here’s a press release from the FD.io project (“Fido”), take it as you will: https://fd.io/latest/singles/kubernetes/ https://fd.io/latest/singles/kubernetes/
- dahfizz 7y ago
  FD.io uses the VPP stack under the hood, which ultimately uses DPDK[1] for the actual IO acceleration. The reason it's faster is because it's polling your hardware. You spend a lot of cpu time to buy the lower latency. [1] http://doc.dpdk.org/guides/prog_guide/overview.html http://doc.dpdk.org/guides/prog_guide/overview.html
  - lstamour 7y ago
    It's possible to write a driver that doesn't use polling - for example https://arxiv.org/pdf/1901.10664.pdf https://arxiv.org/pdf/1901.10664.pdf which discusses using vfio though doing so is apparently much harder than polling alone due to the complexity of the vfio interface and that most users care more about maximizing PPS performance at peak vs saving CPU cycles when less busy. Presumably one other option would be to try and tune the CPU to run slower when less demand is expected, and only run at faster peak performance when absolutely required. Also, it sounds like you might want to look at pinning cores to workloads, such that only one of your CPU cores is polling while the rest act as dynamic worker cores and could do other activities. Note that I'm largely speaking theoretically, for my use cases (mostly at home), the incompatibility of user space networking stacks often gets in the way.
    - dahfizz 7y ago
      > It's possible to write a driver that doesn't use polling Yup, definitely possible. It just doesn't give the PPS numbers as polling does. The project you linked to is really cool, but it is an academic project. It's not trying to compete with DPDK or anything. IO is one of those things in computer science that forces us to make trade ofs. What is more important to you? Raw throughput? Latency? Overall resource usage? You can't optimize all three. > it sounds like you might want to look at pinning cores to workloads, such that only one of your CPU cores is polling Correct again. It may take more than one cpu (DPDK is commercial software running on 88+ cpu systems) but cpu pinning is definitely the recommended way to go. In the spirit of spending CPU to buy IO speed, dedicating physical cores to the task always gives you the best bang for your buck.
lern_too_spel 7y ago
You're correct. I don't know why you're downvoted. Anything you can do to get performance with kernel bypass techniques can be done inside the kernel as well.
- ahmedalsudani 7y ago
  ¯\_(ツ)_/¯ It’s alright. You sometimes get downvoted by the early two people who are offended that I’d dare comment without knowing what PPS is (or something equally offensive).

wmf 7y ago

A polling/batching implementation of WireGuard would be faster than the Linux kernel implementation although it would be much less convenient to use. If WireGuard spends, say, 90% of its time in ChaCha20-Poly1305 (which is already highly optimized) then there's only room for less than 10% speedup.

Hello71 7y ago
it could be faster with a hardware chacha20 implementation. if wireguard gets really popular, we might see that eventually. I'd wager a guess that we would get hardware wireguard first though, with fast path acceleration, similar to hardware TLS.
- tatersolid 7y ago
  Modern CPUs are pretty close to being ASICs for ARX ciphers like ChaCha. Add, rotate, and XOR all have dedicated die area. Bernstein designed it that way. Are there any hardware ASIC implementations of ChaCha or are they basically unnecessary?

e_y_ 7y ago

You could always use a userspace implementation of WireGuard like https://blog.cloudflare.com/boringtun-userspace-wireguard-rust/ https://blog.cloudflare.com/boringtun-userspace-wireguard-ru... . I imagine the advantage of the kernel module is good performance for normal users who are not trying to wring every last bit of performance out of their network setup.

vbezhenar 7y ago

If all networking happens inside kernel, it'll be as fast as possible. People write userspace code because they don't want to put their application into kernel for obvious reasons. And with userspace code it makes sense to move everything into userspace, including network drivers, etc. But it's not a standard configuration and compromises on security.

lvh 7y ago

Most of the userspace high-performance networking stuff is still pretty exotic. It requires plenty of tuning, specific network hardware, and sometimes extreme conditions, to be viable. DPDK does not work on the wireless connection I'm using to post this reply, for example (nor any wireless connection for that matter). Meanwhile, if you're not doing that: Wireguard can avoid lots of syscall and copying overhead plus some magic CPU register overhead by being in the kernel. Hence this almost certainly makes wg faster on the net because it can't pick and choose what environments it runs in.

rkangel 7y ago

You're only half right - projects attempting to achieve very high speed networking (packets per second) go to great lengths to remove the traditional syscall per packet that normal linux networking involves. One approach to this is to move everything into userspace and take the kernel out of it completely. Another approach used by some projects is to put everything into the kernel and take userspace out of it completely. Either of these approaches remove syscalls. The kernel isn't an inherently slower place (in some cases it can be faster as you have more control), it is having your network stack split across the kernel/userspace boundary that is slow.