7 ms·
What is BPF and why is it taking over Linux Performance Analysis?
- forgotpwtomain 10y agoLow quality blogs designed to be used as marketing copy for the company. This would be a much better link/resource: http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.html http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superp...
- binarycrusader 10y agoThat so called low quality blog is one Brendan himself linked to recently: https://twitter.com/brendangregg/status/766314203749638145 https://twitter.com/brendangregg/status/766314203749638145
- deleted 10y ago[deleted]
- diegocg 10y agoWhat is exactly "low quality" about this post?
- wcummings 10y agoIt's a marketing blog. It exists for leadgen.
- daveguy 10y agoFortunately in technology you can have both a high quality informing and informed post that is also sponsored. Posts of the nature "this is a tool we use and how we use it" are informative even if they are from the perspective of a company.
- brendangregg 10y agoThere is something important about this post (even though it looks like marketing): it's the first application vendor to create BPF scripts. It's a milestone for BPF. It won't be the last.
- SFJulie 10y agolong story short. Average is a poor measure of non linear phenomenon like the measure of a performance for a task . Histogram are better to see the repartitions of values, and can hint you on a n-modal behaviour. (a «mode» when everything goes well, and a degraded one for instance which can hint on a potential transition of behavior). I guess the next step is to say this is even more important to know the evolution over time and announce that the equivalent of spectrogram (3D mapped on 2D plus colour) is a good idea to known towards which mode you are tending the more. [EDIT: which can also more significantly graphed as spectrum analysis, colour encoding the amplitude of measure for a given frequency, and frequency being 1/Period thus making sure that slow processes may appear between 0 and average] A shortcut is to hire of of the many jobless guys with a master in science that actually have experience in labs. Beware that for a reason that is beyond my reach, people with background in math, management, literature have problems with scientific concepts that has been discovered after 1800 unless it is packaged like magic.
- static_noise 10y agoReplace the average (mean) with the median and you will have better results in almost all cases. When the data is good the result ist virtually identical. When there are outliers, the median just ignores them unless there lots of them. When the data is that bad, that the median doesn't work anymore you need to up your statistics quite a few notches.
- SFJulie 10y agoWell the median of measure that are only aggregated by average (RRD, carbon ...) is ... biased. Hence the reason I made my own backend for making very high precision measurement using a percentile filter (10%, 50%, 90% and average for the fun). It was fun to watch linux «monotonic» clock not being monotonic.
- omginternets 10y agoIn psychophysics we either use the median for reaction times (Gamma distribution) or we normalize with something like the logarithm. I think some of the techniques might be applicable to system performance analysis, so I submit the following introductory publication, in case any are interested [0]. [0] http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1077&context=tpr http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1077...
- deleted 10y ago[deleted]
- vbernat 10y agoSystemTap enables the same kind of analysis for a few years already. And it works well with older kernels (starting from 3.2, but really from 3.13). Of course, an in-kernel solution is better (and I believe that SystemTap will soon use it). It's unfortunate that SystemTap never had much visbility. It also comes with a huge libraries of tapsets.
- dozzie 10y agoIs SystemTap stable already? As far as I remember from few years ago, it could easily crash your kernel with a valid program that didn't abuse anything, what disqualified it from use in production.
- josefbacik 10y agoSystemtap was fantastic, but super finnicky. You had to have debuginfo for the kernel installed plus the source. This was fine for RHEL but kind of a pain elsewhere. Then you run into other problems like sometimes the distro compiler wasn't used to build the kernel you are running and suddenly you couldn't load any stap script. Also the only way to get output was through stdout, there was no easy way to programmatically pull results from an stap script. BPF/BCC solves most of this. You still have to have the sources so you can get the right targets for kprobes, but you don't have the compiler mismatch problem. You can now access the hash maps directly from user space so you can do things like build system monitoring tools that run in production and record really specific information easily. I used systemtap for years, but bcc/bpf is a whole new world.
- 10y ago
- qwertyuiop924 10y agoeBPF could use a better interface, though. I REALLY don't want to have to write this stuff in C. Think we could get a scripting language?
- okket 10y agoYou only have to write the kernel stuff in C (which is a small part, since you are just collecting data), for the front end you can use BCC (Python/Lua) https://github.com/iovisor/bcc https://github.com/iovisor/bcc Personal favorite: https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt https://github.com/iovisor/bcc/blob/master/tools/sslsniff_ex...
- qwertyuiop924 10y agoInteresting. I'll have to read the docs, but... This looks like a job for... FUNCTIONAL PROGRAMMING!
- brendangregg 10y agoYes, we could, and I imagine we will at some point. It'll have become a user-level problem: the kernel capabilities are there (BPF+everything else), and there's the bcc user-level libraries (written in C) that make using the kernel stuff easier. The easiest way to add a front end would be to consume the bcc libraries. There's plenty of examples of how to use them (the Python and lua interfaces). So if a scripting interface existed, would you use it? For what? Yes, it'd be nice to have, but I think a lot of people will get as far as using the provided bcc tools and be happy with that.
- qwertyuiop924 10y agoIf we could write bpf programs in something other than C, it would go a long way towards opening up eBPF to people who wouldn't be able to write their own scripts otherwise.