6 ms·
Give me 15 minutes and I'll change your view of Linux tracing [video]
- johnrivera 9y agoI can't help but giggle like a schoolchild because of those keyboard sounds.
- kzahel 9y agoPretty funny. Found this in homebrew casks (https://github.com/yingDev/Tickeys https://github.com/yingDev/Tickeys)
- etblg 9y agoThis is amazing. Now I'm swinging around a virtual sword every time I type.
- pmoriarty 9y agoIt was cute for about 2 seconds, then got incredibly annoying.
- vacri 9y agoI get the feeling that it's more satisfying when it's on your own keyboard, but it's definitely annoying on another person's.
- ycombimike 9y agoI'll bet he can't even hear it.
- burkaman 9y agoHe turns it on at the beginning and says it's because he likes having audio feedback.
- justinjlynn 9y agoIt's like playing a video game, I'd imagine. The sound effects are the bane of everyone's existence but your own. Seriously though, you don't play your _game boy_ in public without headphones; let alone present while playing one! I'm amazed -- so, so tempted to turn of the preso even if I was fascinated by the content.
- jachee 9y agoBrenden Gregg was already my favorite Linux internals experts. This would've taken him to the top of the list if he weren't already there.
- boondaburrah 9y agoSuddenly I'm revisiting memories of Platinum Sounds.
- lotyrin 9y agoBPF really seems nice. Ramifications to me though are: if I was willing to pay a few percent overhead on all my production instances, what I would be able to monitor 24/7 and get a return on the investment, and I haven't found much writing in that area. Seems like there could be a lot of opportunity, hopefully I'll get a chance to dive in and find out myself.
- tomsthumb 9y ago> I haven't found much writing in that area This book would _probably_ get you moving in the right direction: http://www.brendangregg.com/sysperfbook.html http://www.brendangregg.com/sysperfbook.html It should be something like: look at your bottlenecks and utilization, look at your costs, look at (cost effective) ways to reduce or remove those bottlenecks or that utilization. Pick the cheapest place to have a bottleneck. Using SSD at an extra 30$ a month lets you use half the CPU and RAM, saving 60$ a month? Go for it.
- wyldfire 9y agoperf is awesome. ftrace is awesomer still for finding great stuff but I'm often on a system with a kernel with no or limited support enabled for it.
- erikb 9y agookay, after 15 minutes of this. Previous view: Linux tracing is so complicated. Without a personally important usecase I probably won't invest the time to learn it. Current view: Linux tracing is so complicated. Without a personally important usecase I probably won't invest the time to learn it. Soo... What I can say is I really hate these keyboard sounds. Otherwise I'm not sure I learned much.
- piyush_soni 9y agoAnd I'm jealous of his keyboard.
- pbhjpbhj 9y agoIt was on his blog before, http://www.brendangregg.com/blog/2016-12-27/linux-tracing-in-15-minutes.html http://www.brendangregg.com/blog/2016-12-27/linux-tracing-in... and has been posted here a couple of times. Interesting this is the most traction it's got AFAICS.
- chicago_wade 9y agoWill BPF replace ftrace? From what I understood he was able to do everything ftrace could by using BPF and BPF was more efficient.
- cyphar 9y agoBPF was used to do the aggregation and calculation in-kernel. You still need ftrace to actually run the BPF program in that context. You can read the cover page for the patch that added this in 2015[1]. [1]: https://lwn.net/Articles/630965/ https://lwn.net/Articles/630965/
- brendangregg 9y agoRight; plus theres some capabilities where ftrace is (and maybe always will be) better. Eg, function counting: ftrace can count all kernel functions instantly (try my perf-tools funccount tool), whereas the BPF method involves setting a kprobe on everything, which takes much longer (setup and tear down). And function graph tracing from ftrace will likely be better than anything we can do in BPF (as it uses tracing all functions as well).
- viraptor 9y agoI feel like despite all this progress, sysdig is still the most accessible solution at the moment. It even includes a slowish, but super simple way of tracing user space (you can even write traces from bash scripts). I wish there was a built-in Linux equivalent.
- haberman 9y agoArgh. This talk reinforced my existing view of Linux tracing: it's really fragmented. Five years ago I tried to make some sense of this by researching all of the existing technologies. In the kernel I found: - ftrace (https://lwn.net/Articles/290277/) - tracepoints (https://www.kernel.org/doc/Documentation/trace/tracepoints.txt) - kprobes (https://www.kernel.org/doc/Documentation/kprobes.txt) - events (https://www.kernel.org/doc/Documentation/trace/events.txt) Now apparently we can add: - BPF, a packet filter that grew into a tracing framework (https://lwn.net/Articles/599755/) In user-space we have: - perf - systemtap - lttng - other, random, fragmented things This talk seems to add a bunch of other fragmented user-space tools. I don't mean to put down anybody's work, but this stuff will never be user-friendly as long as it remains so fragmented, IMHO.
- db48x 9y agoWhat's fragmented about it? Almost everything he showed there was a script that use ftrace, kprobes, or BPF to measure something specific. Since those are all available in the kernel at the same time, you can certainly think of them as a single API.
- alexnewman 9y agoSome of those are old tools some are old. This will always be the case
- xelxebar 9y agoI'm of a similar mind. However, somewhat recently I came across this article which helped provide a framework to think about all these things. Turns it that it's not just a flat space of competing tools: https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://jvns.ca/blog/2017/07/05/linux-tracing-systems/
- Philipp__ 9y agoOr just learn DTrace and hope it will be eventually ported somehow to Linux... /s
- cryptonector 9y agohttps://github.com/dtrace4linux/linux https://github.com/dtrace4linux/linux I... haven't tried it in ages. I've no idea if it works. It used to crash my system easily, but maybe now it's fine.
- the8472 9y agoperf with source annotation is pretty nice if you're profiling for individual hotspots. But I have not found any solution that lets me spot amdahl bottlenecks which get drowned out in raw cycles spent by the parallel parts. In java this is trivial with thread utilization timlines that incorporate sampling. Maybe this could be solved by weighting samples by the inverse of number of running threads at the time