3 ms·

Didn't really convey where the magic happens to get whatever frames are produced by the userspace TCP/IP stack to whatever mechanism puts those frames on some w

Didn't really convey where the magic happens to get whatever frames are produced by the userspace TCP/IP stack to whatever mechanism puts those frames on some wire... isn't it still making some syscall? IOW, where's the I/O happening?

tptacek 2y ago
The simplest way to do this, and the way I think this post is referring to, is to bring up a WireGuard session. WireGuard is essentially just a UDP protocol (so driveable from userland SOCK_DGRAM) that encapsulates whole IP packets, so you can run a TCP stack in userland over it.
- morning-coffee 2y ago
  Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article: > Who says you need to do networking in the kernel though? I thought it implied they were bypassing the kernel entirely. This isn't true at all if it's still using a UDP socket to send... there's still a ton of code in the kernel to handle sending and receiving UDP. It's also a helluva lot less efficient in terms of CPU consumed, unless one is using sendmmsg, because it's a syscall per tiny datagram between user space and kernel space to get the data onto the UDP socket...
  - tptacek 2y ago
    When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything. There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care. Long story short: sometimes this is about perf, here though not so much.
    - akira2501 2y ago
      To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.
    - xxpor 2y ago
      If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/ https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).
      - ignoramous 2y ago
        > If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org I hope, by "you", you mean the rest of us journeymen and not this tptacek [0] [0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy https://archive.md/TVIyy
        nickpsecurity 2y ago
        None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.
        tptacek 2y ago
        I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.
      - cout 2y ago
        I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket). https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload
RandomThoughts3 2y ago
I haven’t followed for a long time but last time I checked the state of the art was getting your packet in and out via an eBPF module like XDP actually bypassing the kernel network stack (that’s not what the article is doing). You would then be directly talking to the NIC through the module which is akin to sending packets directly through the wire.
- stefan_ 2y ago
  Theres also the people that just map the PCIe device into a userland process: https://www.dpdk.org/ https://www.dpdk.org/
  - RandomThoughts3 2y ago
    You can't really just map a PCIe device into userland (by that I mean that the just here does a lot of work). As far as I know, DPDK uses either a BSD NIC driver running in userland (somewhat slow) or XDP on Linux.
    - stefan_ 2y ago
      No, that is indeed the working principle (plus minus some yak shaving with iommus and what not). It certainly doesn't use XDP, that would defeat the purpose.
      - RandomThoughts3 2y ago
        I don’t see in anyway how it would defeat the purpose especially considering it certainly does use XDP [1] for everything it doesn’t have a driver for and you can opt for it instead of an existing driver if you actually want to have something more performant. [1] https://doc.dpdk.org/guides/nics/af_xdp.html https://doc.dpdk.org/guides/nics/af_xdp.html
      - RandomThoughts3 2y ago
        I don’t see in anyway how it would defeat the purpose especially considering it certainly does use XDP [1] for everything it doesn’t have a driver for and you can opt for it instead of an existing driver if you actually want to have something performant. Also, "no" about what? DPDK doesn’t just expose the PCIE device to user land. There is what you call the yak shaving with memory and then it also has a full driver from the NIC so you can actually you know do networking. [1] https://doc.dpdk.org/guides/nics/af_xdp.html https://doc.dpdk.org/guides/nics/af_xdp.html
jcalvinowens 2y ago
Traditionally on Linux it's going to be AF_PACKET sockets: you can read and write raw ethernet frames via the usual syscalls. https://www.man7.org/linux/man-pages/man7/packet.7.html https://www.man7.org/linux/man-pages/man7/packet.7.html There are more modern interfaces which perform better.
- bartvk 2y ago
  Writing raw ethernet frames, can you do that as an unprivileged user? Or is the point more that we're doing userspace networking, which could nevertheless be run as root?
  - jcalvinowens 2y ago
    You need CAP_NET_RAW, but I'm pretty sure you can drop privileges after bind()

tptacek 2y ago

The simplest way to do this, and the way I think this post is referring to, is to bring up a WireGuard session. WireGuard is essentially just a UDP protocol (so driveable from userland SOCK_DGRAM) that encapsulates whole IP packets, so you can run a TCP stack in userland over it.

morning-coffee 2y ago
Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article: > Who says you need to do networking in the kernel though? I thought it implied they were bypassing the kernel entirely. This isn't true at all if it's still using a UDP socket to send... there's still a ton of code in the kernel to handle sending and receiving UDP. It's also a helluva lot less efficient in terms of CPU consumed, unless one is using sendmmsg, because it's a syscall per tiny datagram between user space and kernel space to get the data onto the UDP socket...
- tptacek 2y ago
  When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything. There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care. Long story short: sometimes this is about perf, here though not so much.
  - akira2501 2y ago
    To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.
  - xxpor 2y ago
    If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/ https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).
    - ignoramous 2y ago
      > If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org I hope, by "you", you mean the rest of us journeymen and not this tptacek [0] [0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy https://archive.md/TVIyy
      - nickpsecurity 2y ago
        None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.
      - tptacek 2y ago
        I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.
    - cout 2y ago
      I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket). https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload

RandomThoughts3 2y ago

I haven’t followed for a long time but last time I checked the state of the art was getting your packet in and out via an eBPF module like XDP actually bypassing the kernel network stack (that’s not what the article is doing). You would then be directly talking to the NIC through the module which is akin to sending packets directly through the wire.

stefan_ 2y ago
Theres also the people that just map the PCIe device into a userland process: https://www.dpdk.org/ https://www.dpdk.org/
- RandomThoughts3 2y ago
  You can't really just map a PCIe device into userland (by that I mean that the just here does a lot of work). As far as I know, DPDK uses either a BSD NIC driver running in userland (somewhat slow) or XDP on Linux.
  - stefan_ 2y ago
    No, that is indeed the working principle (plus minus some yak shaving with iommus and what not). It certainly doesn't use XDP, that would defeat the purpose.
    - RandomThoughts3 2y ago
      I don’t see in anyway how it would defeat the purpose especially considering it certainly does use XDP [1] for everything it doesn’t have a driver for and you can opt for it instead of an existing driver if you actually want to have something more performant. [1] https://doc.dpdk.org/guides/nics/af_xdp.html https://doc.dpdk.org/guides/nics/af_xdp.html
    - RandomThoughts3 2y ago
      I don’t see in anyway how it would defeat the purpose especially considering it certainly does use XDP [1] for everything it doesn’t have a driver for and you can opt for it instead of an existing driver if you actually want to have something performant. Also, "no" about what? DPDK doesn’t just expose the PCIE device to user land. There is what you call the yak shaving with memory and then it also has a full driver from the NIC so you can actually you know do networking. [1] https://doc.dpdk.org/guides/nics/af_xdp.html https://doc.dpdk.org/guides/nics/af_xdp.html

jcalvinowens 2y ago

Traditionally on Linux it's going to be AF_PACKET sockets: you can read and write raw ethernet frames via the usual syscalls. https://www.man7.org/linux/man-pages/man7/packet.7.html https://www.man7.org/linux/man-pages/man7/packet.7.html There are more modern interfaces which perform better.

bartvk 2y ago
Writing raw ethernet frames, can you do that as an unprivileged user? Or is the point more that we're doing userspace networking, which could nevertheless be run as root?
- jcalvinowens 2y ago
  You need CAP_NET_RAW, but I'm pretty sure you can drop privileges after bind()