3 ms·

Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article: > Who says

Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article: > Who says you need to do networking in the kernel though? I thought it implied they were bypassing the kernel entirely. This isn't true at all if it's still using a UDP socket to send... there's still a ton of code in the kernel to handle sending and receiving UDP. It's also a helluva lot less efficient in terms of CPU consumed, unless one is using sendmmsg, because it's a syscall per tiny datagram between user space and kernel space to get the data onto the UDP socket...

tptacek 2y ago
When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything. There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care. Long story short: sometimes this is about perf, here though not so much.
- akira2501 2y ago
  To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.
- xxpor 2y ago
  If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/ https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).
  - ignoramous 2y ago
    > If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org I hope, by "you", you mean the rest of us journeymen and not this tptacek [0] [0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy https://archive.md/TVIyy
    - nickpsecurity 2y ago
      None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.
    - tptacek 2y ago
      I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.
  - cout 2y ago
    I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket). https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload
- GoblinSlayer 2y ago
  Also: packet parser vulnerabilities.

tptacek 2y ago

When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything. There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care. Long story short: sometimes this is about perf, here though not so much.

akira2501 2y ago
To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.
xxpor 2y ago
If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/ https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).
- ignoramous 2y ago
  > If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org I hope, by "you", you mean the rest of us journeymen and not this tptacek [0] [0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy https://archive.md/TVIyy
  - nickpsecurity 2y ago
    None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.
  - tptacek 2y ago
    I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.
- cout 2y ago
  I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket). https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload
GoblinSlayer 2y ago
Also: packet parser vulnerabilities.