4 ms·

Thanks for the reply; some replies: > "anything that takes a context switch per traced function will probably be dramatically slower." Good thing uprobes don'

Thanks for the reply; some replies: > "anything that takes a context switch per traced function will probably be dramatically slower." Good thing uprobes don't context switch: # perf stat -e context-switches -e probe_libc:re_search_internal sed '/./d' /mnt/data.txt Performance counter stats for 'sed /./d /mnt/data.txt': 6 context-switches 15,122,432 probe_libc:re_search_internal 19.744738204 seconds time elapsed You mean mode switch? Cheaper, but yes, still costly. Here's runtime without the probe: # time sed '/./d' /mnt/data.txt real 0m3.349s user 0m3.345s sys 0m0.004s Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? AFAIK, LTTng has done work for user<->user instrumentation. I think uBPF will be doing this (https://github.com/iovisor/ubpf https://github.com/iovisor/ubpf) - although that project is very new. Could use some help from some more good engineers (please do!). > "In "speed while not tracing", anything much more expensive than nop-sleds will be too slow to run in production." I'm not sure anyone is suggesting anything more than nop-sleds. Dynamic tracing is zero, and static tracing is nop-sleds. > "probably won't be able to completely hook functions that get inlined" Sure. Sometimes there's static tracing probes (nop-sled based), sometimes there isn't and it's dynamic probes, sometimes those dynamic probes are inlined and you walk up the stack to find one that isn't. If it is inlined, maybe you need to trace the address rather than the function entry. In my experience it's pretty rare that something is just untracable because inlining is so insane. But yes, it does happen sometimes. Usually I figure out a workaround before giving up. > "a mechanism to execute arbitrary code at function entry or exit in a way that's runtime-customizable and very low overhead when you want it to be" BPF! In-kernel virtual machine that runs JIT'd code on events, and is part of mainline Linux. Lots of enhancements in the Linux 4.x series.

davidtgoldblatt 10y ago
> You mean mode switch? Cheaper, but yes, still costly. Hah, yes. > Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? I'm not at Google any more to check (and haven't touched the code since 2012), but IIRC XRay overhead while tracing was something like 100 cycles per function (which includes an entry/exit pair). Hard to compare across different machines made years apart of course, but I think that a little over an order of magnitude difference sounds about right.
- brendangregg 10y ago
  Ok, if the requirements are 100 cycles per function, then you'll (for current kernels) need user<->user. So, AFAIK, an LTTng-like thing or uBPF. And LTTng does have such an order of magnitude difference: slide 25+ of http://www.efficios.com/pub/endusersummit2012/presentation-enduser2012.pdf http://www.efficios.com/pub/endusersummit2012/presentation-e... I'd also look at what problems need solving, in case they can be solved without function tracing. If 100 cycles per function was needed, it sounds like a high rate of functions, and a lot could be done with just timer-based stack trace sampling.
  - jlebar 10y ago
    > I'd also look at what problems need solving, in case they can be solved without function tracing. Without speaking to the specifics of xray, which I'm not so familiar with, I'll say that the folks who developed xray are actually part of a debugging team. They are some of the best debuggers at Google, finding incredibly complex (and, usually rarely-occurring) issues that span many layers of Google's technology stack. So, if they decided they needed function-level tracing, as opposed to any of the other tools they have available, there's probably a good reason.
  - knweiss 10y ago
    Brendan, did you read this awesome blog post (which mentions your work btw) about Google's tracing framework which may explain the kind of problems they want to solve? http://danluu.com/perf-tracing/ http://danluu.com/perf-tracing/ "Sampling profilers, the most common performance debugging tool, are notoriously bad at debugging problems caused by tail latency because they aggregate events into averages. But tail latency is, by definition, not average."
    - brendangregg 10y ago
      I don't quite get it -- to me, that's like saying a hammer is notoriously bad at screwing in screws. And then going ahead and proving it. Um. Good content, but I don't get the premise. I wasn't using sampling profilers to study tail latency in the first place. Who would? (Maybe it's a different usage of "sampling profilers" than I'm familiar with.) As for tail latency: one way is to dump every function event ; a lower cost way is to summarize latency in-kernel as a histogram. For both you're still tracing every function, and care about overhead down to 100 cycles etc. But think about what causes the tails in the first place. Lock contention? Trace that. Resource I/O? Trace that. If it costs 1 us to trace disk I/O, it's usually not a problem. I like to time scheduler switch events with stack traces -- a catch all (but a bit more expensive). Of course, these approaches require kernel instrumentation. :)

davidtgoldblatt 10y ago

> You mean mode switch? Cheaper, but yes, still costly. Hah, yes. > Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? I'm not at Google any more to check (and haven't touched the code since 2012), but IIRC XRay overhead while tracing was something like 100 cycles per function (which includes an entry/exit pair). Hard to compare across different machines made years apart of course, but I think that a little over an order of magnitude difference sounds about right.

brendangregg 10y ago
Ok, if the requirements are 100 cycles per function, then you'll (for current kernels) need user<->user. So, AFAIK, an LTTng-like thing or uBPF. And LTTng does have such an order of magnitude difference: slide 25+ of http://www.efficios.com/pub/endusersummit2012/presentation-enduser2012.pdf http://www.efficios.com/pub/endusersummit2012/presentation-e... I'd also look at what problems need solving, in case they can be solved without function tracing. If 100 cycles per function was needed, it sounds like a high rate of functions, and a lot could be done with just timer-based stack trace sampling.
- jlebar 10y ago
  > I'd also look at what problems need solving, in case they can be solved without function tracing. Without speaking to the specifics of xray, which I'm not so familiar with, I'll say that the folks who developed xray are actually part of a debugging team. They are some of the best debuggers at Google, finding incredibly complex (and, usually rarely-occurring) issues that span many layers of Google's technology stack. So, if they decided they needed function-level tracing, as opposed to any of the other tools they have available, there's probably a good reason.
- knweiss 10y ago
  Brendan, did you read this awesome blog post (which mentions your work btw) about Google's tracing framework which may explain the kind of problems they want to solve? http://danluu.com/perf-tracing/ http://danluu.com/perf-tracing/ "Sampling profilers, the most common performance debugging tool, are notoriously bad at debugging problems caused by tail latency because they aggregate events into averages. But tail latency is, by definition, not average."
  - brendangregg 10y ago
    I don't quite get it -- to me, that's like saying a hammer is notoriously bad at screwing in screws. And then going ahead and proving it. Um. Good content, but I don't get the premise. I wasn't using sampling profilers to study tail latency in the first place. Who would? (Maybe it's a different usage of "sampling profilers" than I'm familiar with.) As for tail latency: one way is to dump every function event ; a lower cost way is to summarize latency in-kernel as a histogram. For both you're still tracing every function, and care about overhead down to 100 cycles etc. But think about what causes the tails in the first place. Lock contention? Trace that. Resource I/O? Trace that. If it costs 1 us to trace disk I/O, it's usually not a problem. I like to time scheduler switch events with stack traces -- a catch all (but a bit more expensive). Of course, these approaches require kernel instrumentation. :)