5 ms·

Linux Observability with BPF

193 points by dankohn1 7y ago

zaphar 7y ago
I've been a fan of dtrace for a long time and so I've been meaning to learn BPF to get similar functionality native to the linux kernel. I may pick this book up.
- danobi 7y ago
  I suggest taking a look at bpftrace ( https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace ). The features and stability are getting quite good and better over time.
  - _wmd 7y ago
    hey! Do you know if anyone is working on getting bpftrace to use BTF yet? Looks like one of the final chunks landed in Linux 5.2
    - leodido 7y ago
      AFAIK not.
    - dustfinger 7y ago
      See Bpftrace for Linux 2018: > https://news.ycombinator.com/item?id=18168137 https://news.ycombinator.com/item?id=18168137 pzakah asks: > You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet. Brendan responds: > That's the old repo (we should add a note to it pointing people to https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace instead!) Alastair added struct support for kprobes yesterday, based on the functionality in bcc (which bpftrace uses). That was the final missing piece, and why I'm posting about it now. See the last example here: https://github.com/iovisor/bpftrace/blob/ma https://github.com/iovisor/bpftrace/blob/ma --- .... I took a look and according to the last example mentioned they have not added full struct support yet.
    - mmarchini 7y ago
      Yes, we have an open Pull Request for that: https://github.com/iovisor/bpftrace/pull/734 https://github.com/iovisor/bpftrace/pull/734
  - fntlnz 7y ago
    There’s also a kubernetes version of it: https://github.com/iovisor/kubectl-trace https://github.com/iovisor/kubectl-trace
- otoburb 7y ago
  Brendan Gregg has a post that provides an overview[1] of the various eBPF wrappers that's worth a look if you want to dive further. [1] http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci...
  - zaphar 7y ago
    Thanks!, That looks quite helpful.
linsomniac 7y ago
As someone who has gone through the ipfwadm -> ipchains -> iptables history, I would generally be pretty meh about another firewalling change. But, I have high hopes that it'll give me the ability to do reasonable traffic shaping in Linux. I once set up a FreeBSD box to do shaping for an ISP's entire DSL customer base, and it was a couple commands and worked brilliantly! By contrast, I've dabbled with Linux shaping for ~2 decades, and the best I've gotten is one recipe that works well for one limited use case. Maybe it's just me, but I've never been able to make it do my bidding, despite several occasions sitting down and giving it the old college try. For other kernel paths, I'm really interested in using it to monitor for file modifications in the way that inotify is not very good. And I often resort to strace to figure out low level what is going on, it might be really good for that too. Looks like an interesting book!
- Fnoord 7y ago
  Wondershaper [1] should make it easy on Linux. On *BSD there is AltQ for this which also has PF integration (or well, back when I looked into this in 2004). I'm curious to hear which use cases didn't work out for you. Regarding BPF, is it possible to convert IPT rules to BPF for backwards compatibility? [1] https://github.com/magnific0/wondershaper/ https://github.com/magnific0/wondershaper/
  - wtallis 7y ago
    Be wary of poorly-designed wrappers around tc. They often make very wrong decisions: https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/ https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper... sqm-scripts is generally a much better option than wondershaper derivatives: https://github.com/tohojo/sqm-scripts https://github.com/tohojo/sqm-scripts
  - linsomniac 7y ago
    Thanks for that pointer, I had only used wondershaper as provided in the Ubuntu package, which is version 1.1a, and horribly broken. I recently had a system that was downloading around 1TB of archive data, and killing the network for other use, so I set up wondershaper on that VM. It did indeed limit it, but I couldn't get it to limit it to anything other than the 768kbps it seemed built to do. I really needed more like 50-100Mbps. I'll have to try version 1.4 on github.
dustfinger 7y ago
Thanks for sharing Jessie! I am definitely going to order this book :-) EDIT: Why did I get down voted? I am not affiliated with this book, the authors of the book, or the OP in anyway. It just so happens that I will find this book very useful, so I thanked the OP for sharing. It is sad we live in a world where there is so much manipulation that we are suspicious of each other's sincerity. I guess I understand where the downvoter was coming from :-(
- monocasa 7y ago
  This sort of comment isn't looked highly upon on HN as it doesn't really add to the discussion. It's what the upvote button is for.
  - mirashii 7y ago
    And in addition, from the guidelines: > Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
  - dustfinger 7y ago
    Fair enough. I will remember for next time.
TwoNineFive 7y ago
It's an advertisement for a book. Cilium is on there and that's a name I associate with blogverts. There are eleven exclamation marks on that page. BPF is the new paradigm for synergistic application of strategic opportunities in the use of acronyms.

193 points by dankohn1 7y ago

I've been a fan of dtrace for a long time and so I've been meaning to learn BPF to get similar functionality native to the linux kernel. I may pick this book up.

danobi 7y ago
I suggest taking a look at bpftrace ( https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace ). The features and stability are getting quite good and better over time.
- _wmd 7y ago
  hey! Do you know if anyone is working on getting bpftrace to use BTF yet? Looks like one of the final chunks landed in Linux 5.2
  - leodido 7y ago
    AFAIK not.
  - dustfinger 7y ago
    See Bpftrace for Linux 2018: > https://news.ycombinator.com/item?id=18168137 https://news.ycombinator.com/item?id=18168137 pzakah asks: > You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet. Brendan responds: > That's the old repo (we should add a note to it pointing people to https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace instead!) Alastair added struct support for kprobes yesterday, based on the functionality in bcc (which bpftrace uses). That was the final missing piece, and why I'm posting about it now. See the last example here: https://github.com/iovisor/bpftrace/blob/ma https://github.com/iovisor/bpftrace/blob/ma --- .... I took a look and according to the last example mentioned they have not added full struct support yet.
  - mmarchini 7y ago
    Yes, we have an open Pull Request for that: https://github.com/iovisor/bpftrace/pull/734 https://github.com/iovisor/bpftrace/pull/734
- fntlnz 7y ago
  There’s also a kubernetes version of it: https://github.com/iovisor/kubectl-trace https://github.com/iovisor/kubectl-trace
otoburb 7y ago
Brendan Gregg has a post that provides an overview[1] of the various eBPF wrappers that's worth a look if you want to dive further. [1] http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci...
- zaphar 7y ago
  Thanks!, That looks quite helpful.

linsomniac 7y ago

As someone who has gone through the ipfwadm -> ipchains -> iptables history, I would generally be pretty meh about another firewalling change. But, I have high hopes that it'll give me the ability to do reasonable traffic shaping in Linux. I once set up a FreeBSD box to do shaping for an ISP's entire DSL customer base, and it was a couple commands and worked brilliantly! By contrast, I've dabbled with Linux shaping for ~2 decades, and the best I've gotten is one recipe that works well for one limited use case. Maybe it's just me, but I've never been able to make it do my bidding, despite several occasions sitting down and giving it the old college try. For other kernel paths, I'm really interested in using it to monitor for file modifications in the way that inotify is not very good. And I often resort to strace to figure out low level what is going on, it might be really good for that too. Looks like an interesting book!

Fnoord 7y ago
Wondershaper [1] should make it easy on Linux. On *BSD there is AltQ for this which also has PF integration (or well, back when I looked into this in 2004). I'm curious to hear which use cases didn't work out for you. Regarding BPF, is it possible to convert IPT rules to BPF for backwards compatibility? [1] https://github.com/magnific0/wondershaper/ https://github.com/magnific0/wondershaper/
- wtallis 7y ago
  Be wary of poorly-designed wrappers around tc. They often make very wrong decisions: https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/ https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper... sqm-scripts is generally a much better option than wondershaper derivatives: https://github.com/tohojo/sqm-scripts https://github.com/tohojo/sqm-scripts
- linsomniac 7y ago
  Thanks for that pointer, I had only used wondershaper as provided in the Ubuntu package, which is version 1.1a, and horribly broken. I recently had a system that was downloading around 1TB of archive data, and killing the network for other use, so I set up wondershaper on that VM. It did indeed limit it, but I couldn't get it to limit it to anything other than the 768kbps it seemed built to do. I really needed more like 50-100Mbps. I'll have to try version 1.4 on github.

dustfinger 7y ago

Thanks for sharing Jessie! I am definitely going to order this book :-) EDIT: Why did I get down voted? I am not affiliated with this book, the authors of the book, or the OP in anyway. It just so happens that I will find this book very useful, so I thanked the OP for sharing. It is sad we live in a world where there is so much manipulation that we are suspicious of each other's sincerity. I guess I understand where the downvoter was coming from :-(

monocasa 7y ago
This sort of comment isn't looked highly upon on HN as it doesn't really add to the discussion. It's what the upvote button is for.
- mirashii 7y ago
  And in addition, from the guidelines: > Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
- dustfinger 7y ago
  Fair enough. I will remember for next time.

TwoNineFive 7y ago

It's an advertisement for a book. Cilium is on there and that's a name I associate with blogverts. There are eleven exclamation marks on that page. BPF is the new paradigm for synergistic application of strategic opportunities in the use of acronyms.