4 ms·

Packet, where are you? – eBPF-based Linux kernel networking debugger

143 points by timetoogo 4y ago

totetsu 4y ago
Is 'tracing network packets in the kernel' also what tcpdump/snoop/nettl dp?
- aeonik 4y ago
  I can speak for tcpdump and the answer is no. It only looks at the network interface. Something I often want to do is catch which process sent a few UDP packets. Netstat and ss won't catch it because it's too short of a time frame, and tcpdump doesn't contain any information about the kernel. I've been using opensnitch which uses eBPF rules to track this information lately, but I'm looking for something more flexible.
  - dhaavi 3y ago
    I don't know what you mean by "flexible", but you could check if what we are building - Portmaster Privacy Suite - goes in that direction: https://safing.io/ https://safing.io/
  - elesiuta 3y ago
    > Something I often want to do is catch which process sent a few UDP packets. Catching short lived processes and packets is one of the things I specialized picosnitch [1] for, which focuses strictly on monitoring. [1] https://github.com/elesiuta/picosnitch https://github.com/elesiuta/picosnitch
- unmole 4y ago
  No. Tcpdump and others are packet sniffers. They clone network packets so the contents can be examined. pwru on the other hand is used to trace what the kernel is doing with your packets.
bmo-at 4y ago
This is a great codebase to read for me currently, as I am working on an ebpf based packet aquisition mechanism for a network monitoring program. Anyone got other good projects to look at?
ralgozino 4y ago
Asking as someone with almost no experience with eBPF. How do you read the otuput of the tool? In the README's example GIF, If I didn't know there's an IPTables rule dropping the packages I would not know that the package is being dropped.
- zokier 4y ago
  You'll want to have kernel sources handy. In this case you can see the packet ending up in nf_hook_slow function: https://elixir.bootlin.com/linux/latest/source/net/netfilter/core.c#L607 https://elixir.bootlin.com/linux/latest/source/net/netfilter... from there you can see only one branch leading to kfree_skb, so you can make a guess that some netfilter "hook" returned NF_DROP. From that you might already be able to make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc
  - ralgozino 4y ago
    thank you! TIL. I guess you need a completely different mindset to approach this.
  - liotier 4y ago
    > make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc Would it be a good idea to let pwru dig into the iptables detour to provide the whole view of the packet's journey - or is iptables at a different layer whose observability is best left to another tool ?
- bazzargh 4y ago
  if you have a recent enough kernel, this change https://github.com/cilium/pwru/pull/148 https://github.com/cilium/pwru/pull/148 means that it will print the reason the packet was dropped in the output, as logged by kfree_skb_reason - see https://lwn.net/Articles/885729/ https://lwn.net/Articles/885729/ There's a whole heap of reasons a packet can be dropped: https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b5298caedd5af0c374a8/include/net/dropreason.h#L84 https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b... In any case, this makes it less painful than going back to read the kernel source. I don't think that the gif in the README is up to date with this change - it's 2 years old and this feature only appeared 2 months ago.
martinohansen 4y ago
Love the name!
cpufry 3y ago
this is pretty neat
nicevalue 3y ago
What is the difference b.w this tool and Dropwatch [https://linux.die.net/man/1/dropwatch https://linux.die.net/man/1/dropwatch?]
- tptacek 3y ago
  dropwatch is much more limited, and effectively obsolete. It's a purpose-designed kernel feature with its own netlink protocol that has one serious consumer (the accursed dropwatch.c client). Leave pwru aside for a minute, and everything dropwatch can do, you can do better just with bpftrace.
- suprjami 3y ago
  Dropwatch just tells you where packets were freed from memory by the "bad" path of `kfree_skb()`. It relies on the "good" packet free path to use `consume_skb()` which isn't always true, so you get a number of false positives with dropwatch. It's hard to spot small individual packet loss among all the noise. pwru offers src/dst IP/port filtering, so it's easier to exclude uninteresting traffic, and it tracks an skb's journey through the entire network stack, printing each function the skb touches over its lifetime. They are quite different tools, they probably have some overlap depending on situation. Also the actual dropwatch command is obsolete, it's more common and useful to use the SystemTap script, or perf, or bpftrace.
tinus_hn 3y ago
I wish there was a way to describe a packet and then trace how it is handled by the firewall, what rules it matches and why it is allowed, dropped or blocked.
suprjami 3y ago
I have dreamed about a tool like this for at least 10 years.

143 points by timetoogo 4y ago

totetsu 4y ago

Is 'tracing network packets in the kernel' also what tcpdump/snoop/nettl dp?

aeonik 4y ago
I can speak for tcpdump and the answer is no. It only looks at the network interface. Something I often want to do is catch which process sent a few UDP packets. Netstat and ss won't catch it because it's too short of a time frame, and tcpdump doesn't contain any information about the kernel. I've been using opensnitch which uses eBPF rules to track this information lately, but I'm looking for something more flexible.
- dhaavi 3y ago
  I don't know what you mean by "flexible", but you could check if what we are building - Portmaster Privacy Suite - goes in that direction: https://safing.io/ https://safing.io/
- elesiuta 3y ago
  > Something I often want to do is catch which process sent a few UDP packets. Catching short lived processes and packets is one of the things I specialized picosnitch [1] for, which focuses strictly on monitoring. [1] https://github.com/elesiuta/picosnitch https://github.com/elesiuta/picosnitch
unmole 4y ago
No. Tcpdump and others are packet sniffers. They clone network packets so the contents can be examined. pwru on the other hand is used to trace what the kernel is doing with your packets.

bmo-at 4y ago

This is a great codebase to read for me currently, as I am working on an ebpf based packet aquisition mechanism for a network monitoring program. Anyone got other good projects to look at?

ralgozino 4y ago

Asking as someone with almost no experience with eBPF. How do you read the otuput of the tool? In the README's example GIF, If I didn't know there's an IPTables rule dropping the packages I would not know that the package is being dropped.

zokier 4y ago
You'll want to have kernel sources handy. In this case you can see the packet ending up in nf_hook_slow function: https://elixir.bootlin.com/linux/latest/source/net/netfilter/core.c#L607 https://elixir.bootlin.com/linux/latest/source/net/netfilter... from there you can see only one branch leading to kfree_skb, so you can make a guess that some netfilter "hook" returned NF_DROP. From that you might already be able to make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc
- ralgozino 4y ago
  thank you! TIL. I guess you need a completely different mindset to approach this.
- liotier 4y ago
  > make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc Would it be a good idea to let pwru dig into the iptables detour to provide the whole view of the packet's journey - or is iptables at a different layer whose observability is best left to another tool ?
bazzargh 4y ago
if you have a recent enough kernel, this change https://github.com/cilium/pwru/pull/148 https://github.com/cilium/pwru/pull/148 means that it will print the reason the packet was dropped in the output, as logged by kfree_skb_reason - see https://lwn.net/Articles/885729/ https://lwn.net/Articles/885729/ There's a whole heap of reasons a packet can be dropped: https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b5298caedd5af0c374a8/include/net/dropreason.h#L84 https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b... In any case, this makes it less painful than going back to read the kernel source. I don't think that the gif in the README is up to date with this change - it's 2 years old and this feature only appeared 2 months ago.

martinohansen 4y ago

Love the name!

cpufry 3y ago

this is pretty neat

nicevalue 3y ago

What is the difference b.w this tool and Dropwatch [https://linux.die.net/man/1/dropwatch https://linux.die.net/man/1/dropwatch?]

tptacek 3y ago
dropwatch is much more limited, and effectively obsolete. It's a purpose-designed kernel feature with its own netlink protocol that has one serious consumer (the accursed dropwatch.c client). Leave pwru aside for a minute, and everything dropwatch can do, you can do better just with bpftrace.
suprjami 3y ago
Dropwatch just tells you where packets were freed from memory by the "bad" path of `kfree_skb()`. It relies on the "good" packet free path to use `consume_skb()` which isn't always true, so you get a number of false positives with dropwatch. It's hard to spot small individual packet loss among all the noise. pwru offers src/dst IP/port filtering, so it's easier to exclude uninteresting traffic, and it tracks an skb's journey through the entire network stack, printing each function the skb touches over its lifetime. They are quite different tools, they probably have some overlap depending on situation. Also the actual dropwatch command is obsolete, it's more common and useful to use the SystemTap script, or perf, or bpftrace.

tinus_hn 3y ago

I wish there was a way to describe a packet and then trace how it is handled by the firewall, what rules it matches and why it is allowed, dropped or blocked.

suprjami 3y ago

I have dreamed about a tool like this for at least 10 years.