5 ms·

Perf is much more than a sampling profiler: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page. It can use uprobe

by 4ad 11y ago

Perf is much more than a sampling profiler: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page. It can use uprobes and kprobes. There are many ways to do dynamic tracing in Linux, apart from perf. Ftrace, raw kprobes/uprobes, eBPF, bcc compiler for eBPF, etc.

teraflop 11y ago
After perusing the perf man page, the only way I could figure out how to make it accurately count userspace function calls was using hardware breakpoints, e.g.: "perf stat -e mem:0xADDRESS:x" Obviously that's not a very good approach, because you're limited to the number of breakpoints that your CPU can handle simultaneously (4 on my machine) and there's a lot of overhead. If you know of a better way to accomplish the same thing with perf, I'd be happy to hear it.
- wmf 11y ago
  Check out http://www.brendangregg.com/perf.html#DynamicTracingEg http://www.brendangregg.com/perf.html#DynamicTracingEg
- cthalupa 11y ago
  https://lwn.net/Articles/499190/ https://lwn.net/Articles/499190/ & https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-space-probe-instrumentation/ https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-spa... http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-upr...
  - teraflop 11y ago
    Ah, cool. I just tested it out and it seems to work as documented. Unfortunately it requires root access, and incurs about 1 microsecond of overhead per function call on my machine.
    - cthalupa 11y ago
      Reading the X-Ray abstract, unless I'm missing it, it doesn't go into detail about the time overhead. I do see mention of CPU and RAM usage, though. I'm not sure that we have any evidence that X-Ray offers less overhead here, though it's certainly cool if it does.
      - DannyBee 11y ago
        Can you give an example where perf works with 10 million+ probe points? :) (and where this does not have an antagonist effect on the other things on the machine?)
SEJeff 11y ago
And for those poor people who haven't seen bcc, do yourself a favor and look into it: http://iovisor.github.io/bcc/ http://iovisor.github.io/bcc/ Besides, with a unicorn for the logo, whats not to like?

by 4ad 11y ago

teraflop 11y ago

After perusing the perf man page, the only way I could figure out how to make it accurately count userspace function calls was using hardware breakpoints, e.g.: "perf stat -e mem:0xADDRESS:x" Obviously that's not a very good approach, because you're limited to the number of breakpoints that your CPU can handle simultaneously (4 on my machine) and there's a lot of overhead. If you know of a better way to accomplish the same thing with perf, I'd be happy to hear it.

wmf 11y ago
Check out http://www.brendangregg.com/perf.html#DynamicTracingEg http://www.brendangregg.com/perf.html#DynamicTracingEg
cthalupa 11y ago
https://lwn.net/Articles/499190/ https://lwn.net/Articles/499190/ & https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-space-probe-instrumentation/ https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-spa... http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-upr...
- teraflop 11y ago
  Ah, cool. I just tested it out and it seems to work as documented. Unfortunately it requires root access, and incurs about 1 microsecond of overhead per function call on my machine.
  - cthalupa 11y ago
    Reading the X-Ray abstract, unless I'm missing it, it doesn't go into detail about the time overhead. I do see mention of CPU and RAM usage, though. I'm not sure that we have any evidence that X-Ray offers less overhead here, though it's certainly cool if it does.
    - DannyBee 11y ago
      Can you give an example where perf works with 10 million+ probe points? :) (and where this does not have an antagonist effect on the other things on the machine?)

SEJeff 11y ago

And for those poor people who haven't seen bcc, do yourself a favor and look into it: http://iovisor.github.io/bcc/ http://iovisor.github.io/bcc/ Besides, with a unicorn for the logo, whats not to like?