4 ms·

Most of XRay was written before uprobes was merged into the Linux kernel (and well before such kernels were widely available). I don't think any of the alterna

Most of XRay was written before uprobes was merged into the Linux kernel (and well before such kernels were widely available). I don't think any of the alternatives you mentioned are Pareto superior to XRay when considering all of "speed while tracing", "speed while not tracing", and "flexibility". E.g.: - In "speed while tracing", anything that takes a context switch per traced function will probably be dramatically slower. Even if there's some fast dispatch mechanism you have in mind that I'm not familiar with when you say dynamic tracing, if it doesn't insert the moral equivalent of a nop-sled, it will have to either choose between logging the whole PC (spending data, which means spending RAM and disk time) or figuring out how to map it to a function-specific unique int (spending cycles). - In "speed while not tracing", anything much more expensive than nop-sleds will be too slow to run in production. - Anything that doesn't have a compile time component probably won't be able to completely hook functions that get inlined, or whose source you aren't able to change, won't be able to pick out information the runtime wants to summarize from function arguments, etc. To me, the neat thing about XRay isn't so much the "function patching" aspect, except insofar as it serves as a mechanism to execute arbitrary code at function entry or exit in a way that's runtime-customizable and very low overhead when you want it to be.

deleted 10y ago
[deleted]
brendangregg 10y ago
Thanks for the reply; some replies: > "anything that takes a context switch per traced function will probably be dramatically slower." Good thing uprobes don't context switch: # perf stat -e context-switches -e probe_libc:re_search_internal sed '/./d' /mnt/data.txt Performance counter stats for 'sed /./d /mnt/data.txt': 6 context-switches 15,122,432 probe_libc:re_search_internal 19.744738204 seconds time elapsed You mean mode switch? Cheaper, but yes, still costly. Here's runtime without the probe: # time sed '/./d' /mnt/data.txt real 0m3.349s user 0m3.345s sys 0m0.004s Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? AFAIK, LTTng has done work for user<->user instrumentation. I think uBPF will be doing this (https://github.com/iovisor/ubpf https://github.com/iovisor/ubpf) - although that project is very new. Could use some help from some more good engineers (please do!). > "In "speed while not tracing", anything much more expensive than nop-sleds will be too slow to run in production." I'm not sure anyone is suggesting anything more than nop-sleds. Dynamic tracing is zero, and static tracing is nop-sleds. > "probably won't be able to completely hook functions that get inlined" Sure. Sometimes there's static tracing probes (nop-sled based), sometimes there isn't and it's dynamic probes, sometimes those dynamic probes are inlined and you walk up the stack to find one that isn't. If it is inlined, maybe you need to trace the address rather than the function entry. In my experience it's pretty rare that something is just untracable because inlining is so insane. But yes, it does happen sometimes. Usually I figure out a workaround before giving up. > "a mechanism to execute arbitrary code at function entry or exit in a way that's runtime-customizable and very low overhead when you want it to be" BPF! In-kernel virtual machine that runs JIT'd code on events, and is part of mainline Linux. Lots of enhancements in the Linux 4.x series.
- davidtgoldblatt 10y ago
  > You mean mode switch? Cheaper, but yes, still costly. Hah, yes. > Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? I'm not at Google any more to check (and haven't touched the code since 2012), but IIRC XRay overhead while tracing was something like 100 cycles per function (which includes an entry/exit pair). Hard to compare across different machines made years apart of course, but I think that a little over an order of magnitude difference sounds about right.
  - brendangregg 10y ago
    Ok, if the requirements are 100 cycles per function, then you'll (for current kernels) need user<->user. So, AFAIK, an LTTng-like thing or uBPF. And LTTng does have such an order of magnitude difference: slide 25+ of http://www.efficios.com/pub/endusersummit2012/presentation-enduser2012.pdf http://www.efficios.com/pub/endusersummit2012/presentation-e... I'd also look at what problems need solving, in case they can be solved without function tracing. If 100 cycles per function was needed, it sounds like a high rate of functions, and a lot could be done with just timer-based stack trace sampling.
    - jlebar 10y ago
      > I'd also look at what problems need solving, in case they can be solved without function tracing. Without speaking to the specifics of xray, which I'm not so familiar with, I'll say that the folks who developed xray are actually part of a debugging team. They are some of the best debuggers at Google, finding incredibly complex (and, usually rarely-occurring) issues that span many layers of Google's technology stack. So, if they decided they needed function-level tracing, as opposed to any of the other tools they have available, there's probably a good reason.
    - knweiss 10y ago
      Brendan, did you read this awesome blog post (which mentions your work btw) about Google's tracing framework which may explain the kind of problems they want to solve? http://danluu.com/perf-tracing/ http://danluu.com/perf-tracing/ "Sampling profilers, the most common performance debugging tool, are notoriously bad at debugging problems caused by tail latency because they aggregate events into averages. But tail latency is, by definition, not average."

deleted 10y ago

[deleted]

brendangregg 10y ago

Thanks for the reply; some replies: > "anything that takes a context switch per traced function will probably be dramatically slower." Good thing uprobes don't context switch: # perf stat -e context-switches -e probe_libc:re_search_internal sed '/./d' /mnt/data.txt Performance counter stats for 'sed /./d /mnt/data.txt': 6 context-switches 15,122,432 probe_libc:re_search_internal 19.744738204 seconds time elapsed You mean mode switch? Cheaper, but yes, still costly. Here's runtime without the probe: # time sed '/./d' /mnt/data.txt real 0m3.349s user 0m3.345s sys 0m0.004s Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? AFAIK, LTTng has done work for user<->user instrumentation. I think uBPF will be doing this (https://github.com/iovisor/ubpf https://github.com/iovisor/ubpf) - although that project is very new. Could use some help from some more good engineers (please do!). > "In "speed while not tracing", anything much more expensive than nop-sleds will be too slow to run in production." I'm not sure anyone is suggesting anything more than nop-sleds. Dynamic tracing is zero, and static tracing is nop-sleds. > "probably won't be able to completely hook functions that get inlined" Sure. Sometimes there's static tracing probes (nop-sled based), sometimes there isn't and it's dynamic probes, sometimes those dynamic probes are inlined and you walk up the stack to find one that isn't. If it is inlined, maybe you need to trace the address rather than the function entry. In my experience it's pretty rare that something is just untracable because inlining is so insane. But yes, it does happen sometimes. Usually I figure out a workaround before giving up. > "a mechanism to execute arbitrary code at function entry or exit in a way that's runtime-customizable and very low overhead when you want it to be" BPF! In-kernel virtual machine that runs JIT'd code on events, and is part of mainline Linux. Lots of enhancements in the Linux 4.x series.

davidtgoldblatt 10y ago
> You mean mode switch? Cheaper, but yes, still costly. Hah, yes. > Which means we can calculate the cost to be ~1.1 us per probe (on my system). Anyone know what XRay is clocking in at? I'm not at Google any more to check (and haven't touched the code since 2012), but IIRC XRay overhead while tracing was something like 100 cycles per function (which includes an entry/exit pair). Hard to compare across different machines made years apart of course, but I think that a little over an order of magnitude difference sounds about right.
- brendangregg 10y ago
  Ok, if the requirements are 100 cycles per function, then you'll (for current kernels) need user<->user. So, AFAIK, an LTTng-like thing or uBPF. And LTTng does have such an order of magnitude difference: slide 25+ of http://www.efficios.com/pub/endusersummit2012/presentation-enduser2012.pdf http://www.efficios.com/pub/endusersummit2012/presentation-e... I'd also look at what problems need solving, in case they can be solved without function tracing. If 100 cycles per function was needed, it sounds like a high rate of functions, and a lot could be done with just timer-based stack trace sampling.
  - jlebar 10y ago
    > I'd also look at what problems need solving, in case they can be solved without function tracing. Without speaking to the specifics of xray, which I'm not so familiar with, I'll say that the folks who developed xray are actually part of a debugging team. They are some of the best debuggers at Google, finding incredibly complex (and, usually rarely-occurring) issues that span many layers of Google's technology stack. So, if they decided they needed function-level tracing, as opposed to any of the other tools they have available, there's probably a good reason.
  - knweiss 10y ago
    Brendan, did you read this awesome blog post (which mentions your work btw) about Google's tracing framework which may explain the kind of problems they want to solve? http://danluu.com/perf-tracing/ http://danluu.com/perf-tracing/ "Sampling profilers, the most common performance debugging tool, are notoriously bad at debugging problems caused by tail latency because they aggregate events into averages. But tail latency is, by definition, not average."