4 ms·

The Linux network stack has been rapidly evolving; nowadays we have XDP for a fast path through the kernel. But there's also a lot of moving parts that ensure h

The Linux network stack has been rapidly evolving; nowadays we have XDP for a fast path through the kernel. But there's also a lot of moving parts that ensure high throughput on noisy networks, for different application workload patterns, to avoid bufferbloat, and to do this fairly across multiple processes/containers. I'd imagine one can develop a user-space stack that appeared better in a simple iperf microbenchmark in a lab, but would lose for real-world applications and networks (unless your real-world use case was very narrow). I'd recommend understanding the stack before choosing to replace it (the old programmer's advice about understanding a gate before removing it). I summarized the TCP send path in my next book (sysperf2, with help from Linux network engineers); it involves: - Socket send buffers - Pacing - TSO - Congestion controls - Nagle - TSQ - qdisc (optional) - GSQ - BQL - TSO (NIC) That's just the TCP send path. I'm happy to expand all these acronyms, but if any are new to you, that's kind-of my point. These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve.

dilyevsky 6y ago
Big fan of your books and blog! Would you say usefulness of dpdk and xdp has diminished purely from performance standpoint? I hear rumors you can do line rate on modern linux kernels these days but haven’t had a chance to try it personally yet. I think at the time maglev was originally developed google was still on 2.6 kernels...
- brendangregg 6y ago
  Thanks. I'd expect more future uses of xdp over dpdk. As for max performance (I saw a Linux server max out multiple 10 GbE ports years ago) I'm not sure anyone right now knows what it is. We've recently added XDP, io_uring, BPF congestion control algorithms, and more. There are performance numbers for these in isolation, but I don't think anyone has performance tested (and tuned) them all working together yet. And that poses a problem: to reach maximum performance requires many engineering hours to learn and configure all of these. You'd need a large gain to justify doing this: E.g., for a system doing over 1M packets/s. This complexity may change. Look at BPF for observability: we built BCC and bpftrace so that using it was trivial. What trivial front-end/technology could we build upon XDP and io_uring to make deployment easy?
  - bogomipz 6y ago
    >"I'd expect more future uses of xdp over dpdk." Can say why that is? I haven't really kept up with DPDK but I know there was lots of interest in it a few years back. What has changed?
    - deleted 6y ago
      [deleted]
    - wtallis 6y ago
      I haven't been watching the networking side too closely, but I expect the same thing to play out for storage with io_uring vs SPDK. When the in-kernel solution is almost as fast, has much broader availability and compatibility, and imposes fewer constraints on hardware purchasing decisions and system administration, the userspace offload solution gets consigned to a fairly small niche.
      - brendangregg 6y ago
        Right; there's a better presentation I can't find right now, but here's one from Intel showing how XDP was catching up in 2018 http://vger.kernel.org/lpc_net2018_talks/lpc18_pres_af_xdp_perf-v3.pdf http://vger.kernel.org/lpc_net2018_talks/lpc18_pres_af_xdp_p...
  - LunaSea 6y ago
    Do you know how far along protocol implementation for higher layers like HTTP and TLS are?
  - maartenh 6y ago
    How would you debug programs using ioring? With the traditional I/O syscalls, one can use tools like strace to observe the behavior of a program. Would using BPF to instrument the ioring be the way to go?
amelius 6y ago
> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve. But if you invent them in a different order, you might just end up with a better system ...
- takeda 6y ago
  I don't know... generally when things are developed by someone that doesn't understand the topic end up being quite bad. Just look at some databases that were created by people who know nothing about databases when they started.
techman9 6y ago
This is super useful! I've been hacking recently on qdisc classifiers, and trying to set a filter up with BPF, but there's a paucity of documentation on the subject. The tc tools are sorta documented here (https://man7.org/linux/man-pages/man8/tc-bpf.8.html https://man7.org/linux/man-pages/man8/tc-bpf.8.html), but it's quite hard to find docs on the underlying syscalls and the behavior of `bpf_prog_type_sched_cls`. I bought a copy of BPF performance tools to learn more about the BPF interface, and it's really useful but quite focused on performance! I wish there was a resource of similar depth and breath about eBPF for classification/XDP.
- brendangregg 6y ago
  Thanks for getting the BPF book. Further references for BPF bits would be the kernel docs (Documentation/bpf), kernel source (kernel/bpf, include/.../bpf), the self tests (tools/testing/selftests/bpf), Cilium's docs (https://docs.cilium.io/en/stable/bpf/ https://docs.cilium.io/en/stable/bpf/), and if all else fails, mailing lists like iovisor-dev. It sounds like there should be a deep book on BPF internals (written by maintainers Alexei & Daniel), but it would be a tough sell: you really do need the maintainers to author it or be heavily involved to do it well, and it's a lot of work for a changing target, and without a large audience (lots of people will use BPF, but few will code it at that level. I'd guess 100 world-wide. Not a great case for a book!) Compare it to my BPF perf tools book: big audience (end users: developers, SREs, perf engineers, etc), and has a long shelf life (the tools are done, and work, and should keep working).
bogomipz 6y ago
I didn't recognize GSQ and Google wasn't much help unfortunately. Could you say a few words about that and also maybe what "pacing" is? Thanks.
- brendangregg 6y ago
  Ah, sorry, that should be GSO (typo!). Generic segmentation offload, which aggregates "super packets" (up to 64 Kbytes) within the network stack to reduce overheads. Pacing controls when to send packets, spreading out transmissions to avoid bursts that can hurt performance.
kstenerud 6y ago
> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve. But is that really such a bad thing? Right now your choices are take it or leave it, or change operating systems. What if we could instead choose libraries suited to purpose, written by experts, like we do with encryption or compression or allocation or any number of common tasks? You could then have a default implementation suitable for 80% of use cases, and specialized implementations to handle more complex requirements. You could even have per-NIC exclusive device access for a single application!
ksec 6y ago
>The Linux network stack has been rapidly evolving Are FreeBSD no longer getting any of those? It seems even Network Stack talks from Netflix are now Linux Focused. It also makes me wonder..... does any of these matter once BPF takes over, quite literally.
- cpach 6y ago
  I really like the FreeBSD philosophy but AFAICT the Linux ecosystem moves much much faster than FreeBSD. That means FreeBSD is more stable and has less weird surprises – but also that Linux get new features at a much faster rate.
- brendangregg 6y ago
  Netflix is lots of Linux and lots of FreeBSD. The Netflix FreeBSD network stack also has various optimizations (we call it the "rack" stack). I've seen a good presentation on how it works, by Randall Stewart, but I don't know that it's public yet.

dilyevsky 6y ago

Big fan of your books and blog! Would you say usefulness of dpdk and xdp has diminished purely from performance standpoint? I hear rumors you can do line rate on modern linux kernels these days but haven’t had a chance to try it personally yet. I think at the time maglev was originally developed google was still on 2.6 kernels...

brendangregg 6y ago
Thanks. I'd expect more future uses of xdp over dpdk. As for max performance (I saw a Linux server max out multiple 10 GbE ports years ago) I'm not sure anyone right now knows what it is. We've recently added XDP, io_uring, BPF congestion control algorithms, and more. There are performance numbers for these in isolation, but I don't think anyone has performance tested (and tuned) them all working together yet. And that poses a problem: to reach maximum performance requires many engineering hours to learn and configure all of these. You'd need a large gain to justify doing this: E.g., for a system doing over 1M packets/s. This complexity may change. Look at BPF for observability: we built BCC and bpftrace so that using it was trivial. What trivial front-end/technology could we build upon XDP and io_uring to make deployment easy?
- bogomipz 6y ago
  >"I'd expect more future uses of xdp over dpdk." Can say why that is? I haven't really kept up with DPDK but I know there was lots of interest in it a few years back. What has changed?
  - deleted 6y ago
    [deleted]
  - wtallis 6y ago
    I haven't been watching the networking side too closely, but I expect the same thing to play out for storage with io_uring vs SPDK. When the in-kernel solution is almost as fast, has much broader availability and compatibility, and imposes fewer constraints on hardware purchasing decisions and system administration, the userspace offload solution gets consigned to a fairly small niche.
    - brendangregg 6y ago
      Right; there's a better presentation I can't find right now, but here's one from Intel showing how XDP was catching up in 2018 http://vger.kernel.org/lpc_net2018_talks/lpc18_pres_af_xdp_perf-v3.pdf http://vger.kernel.org/lpc_net2018_talks/lpc18_pres_af_xdp_p...
- LunaSea 6y ago
  Do you know how far along protocol implementation for higher layers like HTTP and TLS are?
- maartenh 6y ago
  How would you debug programs using ioring? With the traditional I/O syscalls, one can use tools like strace to observe the behavior of a program. Would using BPF to instrument the ioring be the way to go?

amelius 6y ago

> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve. But if you invent them in a different order, you might just end up with a better system ...

takeda 6y ago
I don't know... generally when things are developed by someone that doesn't understand the topic end up being quite bad. Just look at some databases that were created by people who know nothing about databases when they started.

techman9 6y ago

This is super useful! I've been hacking recently on qdisc classifiers, and trying to set a filter up with BPF, but there's a paucity of documentation on the subject. The tc tools are sorta documented here (https://man7.org/linux/man-pages/man8/tc-bpf.8.html https://man7.org/linux/man-pages/man8/tc-bpf.8.html), but it's quite hard to find docs on the underlying syscalls and the behavior of `bpf_prog_type_sched_cls`. I bought a copy of BPF performance tools to learn more about the BPF interface, and it's really useful but quite focused on performance! I wish there was a resource of similar depth and breath about eBPF for classification/XDP.

brendangregg 6y ago
Thanks for getting the BPF book. Further references for BPF bits would be the kernel docs (Documentation/bpf), kernel source (kernel/bpf, include/.../bpf), the self tests (tools/testing/selftests/bpf), Cilium's docs (https://docs.cilium.io/en/stable/bpf/ https://docs.cilium.io/en/stable/bpf/), and if all else fails, mailing lists like iovisor-dev. It sounds like there should be a deep book on BPF internals (written by maintainers Alexei & Daniel), but it would be a tough sell: you really do need the maintainers to author it or be heavily involved to do it well, and it's a lot of work for a changing target, and without a large audience (lots of people will use BPF, but few will code it at that level. I'd guess 100 world-wide. Not a great case for a book!) Compare it to my BPF perf tools book: big audience (end users: developers, SREs, perf engineers, etc), and has a long shelf life (the tools are done, and work, and should keep working).

bogomipz 6y ago

I didn't recognize GSQ and Google wasn't much help unfortunately. Could you say a few words about that and also maybe what "pacing" is? Thanks.

brendangregg 6y ago
Ah, sorry, that should be GSO (typo!). Generic segmentation offload, which aggregates "super packets" (up to 64 Kbytes) within the network stack to reduce overheads. Pacing controls when to send packets, spreading out transmissions to avoid bursts that can hurt performance.

kstenerud 6y ago

> These parts have a purpose, and starting from scratch may lead you to eventually re-invent them all as you encounter the problems they solve. But is that really such a bad thing? Right now your choices are take it or leave it, or change operating systems. What if we could instead choose libraries suited to purpose, written by experts, like we do with encryption or compression or allocation or any number of common tasks? You could then have a default implementation suitable for 80% of use cases, and specialized implementations to handle more complex requirements. You could even have per-NIC exclusive device access for a single application!

ksec 6y ago

>The Linux network stack has been rapidly evolving Are FreeBSD no longer getting any of those? It seems even Network Stack talks from Netflix are now Linux Focused. It also makes me wonder..... does any of these matter once BPF takes over, quite literally.

cpach 6y ago
I really like the FreeBSD philosophy but AFAICT the Linux ecosystem moves much much faster than FreeBSD. That means FreeBSD is more stable and has less weird surprises – but also that Linux get new features at a much faster rate.
brendangregg 6y ago
Netflix is lots of Linux and lots of FreeBSD. The Netflix FreeBSD network stack also has various optimizations (we call it the "rack" stack). I've seen a good presentation on how it works, by Randall Stewart, but I don't know that it's public yet.