5 ms·

Seems much more restricted than perf or DTrace, or all the other tracing tools. I don't understand the point of this at all. NIH? I'd love to heard brendangreg

by 4ad 11y ago

Seems much more restricted than perf or DTrace, or all the other tracing tools. I don't understand the point of this at all. NIH? I'd love to heard brendangregg's or bcantrill's thoughts on this. (Does HN give users some kind of alert when they are mentioned?).

bitmapbrother 11y ago
Does perf or Dtrace have the same functionality? >XRay allows you to get accurate function call traces with negligible overhead when off and moderate overhead when on, suitable for services deployed in production. XRay enables efficient function call entry/exit logging with high accuracy timestamps, and can be dynamically enabled and disabled.
- 4ad 11y ago
  Yes?
  - DannyBee 11y ago
    First, i always assume dtrace can do something. Dtrace is not that low overhead in the situation that xray is describing. http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provider-overhead/ http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provid... When they traced everything, it was two orders of magnitude slower. At 100k event/s (quite possible in the situations x-ray is used), the app would be about 60% slower. For just a simple probe, not full on tracing. Does this make it better than dtrace? No. It just serves a different set of use cases.
    - dap 11y ago
      > When they traced everything, it was two orders of magnitude slower. > At 100k event/s (quite possible in the situations x-ray is used), the app would be about 60% slower. For just a simple probe, not full on tracing. I think you've misunderstood the post. The actual time reported in that post is 600 ns per probe, with sanity checks reporting as much as 2000ns, which he concluded was in the same ballpark. He measured that by observing a program that just calls two functions in a tight loop, one of which is strlen() on a dozen-character string. That's basically the worst possible case for any function call tracing system -- I don't think it's fair to say that's "a simple probe, not full on tracing". The "60%" slower conclusion is only by construction: if you have N events per second, and the framework adds overhead of T nanoseconds per probe, then you can pick N such that the tracing framework adds whatever percentage overhead you like. In this case, Brendan picked N=100K and came up with 60% overhead for that case, but I think there's a math error in there. For that calculation, he assumes a 6us probe time instead of 600ns. I think the overhead would be 6% for 100,000 events, not 60%.
      - DannyBee 11y ago
        I didn't misunderstand. The problem with this is that you assume that if you probe every function instead of just strlen that you get the same time bounds. The only way to do this is, afaik, something similar to http://docs.oracle.com/cd/E19253-01/819-5488/gcgmc/index.html http://docs.oracle.com/cd/E19253-01/819-5488/gcgmc/index.htm... This really just is going to expand to putting a probe on every function. I'd really like to see numbers on how dtrace handles many millions of probes at once (which is what xray is handling). It's not uncommon to have 10 or 100 million functions in some of these programs. I have strong doubts that dtrace has the same overhead given 200 million probes. (since the probe data structures alone will likely take up gigabytes of memory, accessing them is unlikely to be cache friendly, etc) AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P TL;DR dynamic probing infrastructures are not a panacea
        dap 11y ago
        > I'd really like to see numbers on how dtrace handles many millions of probes at once (which is what xray is handling). It's not uncommon to have 10 or 100 million functions in some of these programs. I have strong doubts that dtrace has the same overhead given 200 million probes. I see now. That's a fair question, and I'm not aware of data either way. I just tried a pretty simple experiment inspired by Brendan's that suggests that on my machine, the overhead is about 1450ns per probe for as many as 140,000 probes: https://gist.github.com/davepacheco/a12a0d45d55f0d7a28c312c2cd3cf234 https://gist.github.com/davepacheco/a12a0d45d55f0d7a28c312c2... > AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P Well, DTrace as architected is always going to pay the cost of a context switch into the kernel for each probe, and I think it's fair to take Brendan's result of 600ns as a lower bound of the per-probe overhead, at least on his machine. However, once in the kernel, for a typical native program (i.e., not JIT), I expect DTrace would only record the current userland thread instruction pointer. Names are typically resolved asynchronously by the consumer. So I would be surprised if it really was much slower, especially given the result above, but I too would like to see data. I'm not saying that DTrace solves all problems or even that the OP should have used it instead. It's certainly true that for the special case of userland function boundary tracing, one might expect to do better by skipping the context switch (at the expense of much functionality, including any ability to correlate with broader system activity). But since DTrace was brought up, I wanted to help clarify the uncertainty about what it can do and what its overhead is.
teraflop 11y ago
I don't know about DTrace, but perf is a sampling profiler, not an accurate call tracer.
- 4ad 11y ago
  Perf is much more than a sampling profiler: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page. It can use uprobes and kprobes. There are many ways to do dynamic tracing in Linux, apart from perf. Ftrace, raw kprobes/uprobes, eBPF, bcc compiler for eBPF, etc.
  - teraflop 11y ago
    After perusing the perf man page, the only way I could figure out how to make it accurately count userspace function calls was using hardware breakpoints, e.g.: "perf stat -e mem:0xADDRESS:x" Obviously that's not a very good approach, because you're limited to the number of breakpoints that your CPU can handle simultaneously (4 on my machine) and there's a lot of overhead. If you know of a better way to accomplish the same thing with perf, I'd be happy to hear it.
    - wmf 11y ago
      Check out http://www.brendangregg.com/perf.html#DynamicTracingEg http://www.brendangregg.com/perf.html#DynamicTracingEg
    - cthalupa 11y ago
      https://lwn.net/Articles/499190/ https://lwn.net/Articles/499190/ & https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-space-probe-instrumentation/ https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-spa... http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-upr...
      - teraflop 11y ago
        Ah, cool. I just tested it out and it seems to work as documented. Unfortunately it requires root access, and incurs about 1 microsecond of overhead per function call on my machine.
asuffield 11y ago
(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google) I recommend reading page 2 of the paper, which discusses the specific set of features that XRay offers. How do I get perf or DTrace to give me the six things listed there? I can only think of ways to get a couple of them.
- dap 11y ago
  > The cost is acceptable when tracing and barely measurable when not tracing. "Acceptable" is obviously relative, but with DTrace's pid provider, the cost is zero when not tracing, and about the cost of a fastcall per probe point when enabled. > Instrumentation is automatic and directed towards functions that are important for understanding the binary’s execution time. I'm not sure what this means, but with DTrace, you enumerate the functions or binary objects (with wildcards and such) that you want to instrument, and the framework takes care of reliably instrumenting them, no matter the state of the process. Is that "automatic" and "directed"? I need to read the rest of the paper more closely. > Tracing is efficient in both space and time -- only recording what is required and what matters. DTrace records exactly what you ask it to. It supports in-situ aggregation for cases where it's not tenable to record a complete log of all interesting events. This is an important part of the design. > Tracing is configurable with thresholds for storage (how much memory to use) and accuracy (whether to log everything or only function calls taking at least some amount of time). With DTrace, it's pretty easy to filter on function execution time. The buffer size is configurable. There are also multiple buffer policies for different use-cases (e.g., ringbuffer of the last N events leading up to some other event). > Tracing does not require changes to the operating system nor super-user privileges. If they're running Linux, as I imagine they are, DTrace isn't necessarily an option. Several other platforms have just ported it. Using it to record user-level state on your own processes does not require superuser privileges. > Tracing can be turned on and off dynamically without having to restart the server. Absolutely -- that's what the "D" is for. I'd strongly recommended checking out the DTrace paper: https://www.usenix.org/legacy/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/ https://www.usenix.org/legacy/event/usenix04/tech/general/fu... There may be good reasons not to use DTrace for this, but I'm not sure which of those six goals would be the sticking point other than OS availability. (edit: I also haven't read beyond that yet!)
evmar 11y ago
It's definitely the sort of paper that would get reviewer feedback if it was submitted to a conference, in that it failed to compare any related work. (But it's unfair to grade it by that metric because it reads like more of a "here's what we did" blog post, not a research paper.) From reading it, I believe no other tool is both (1) non-sampled and (2) supports instrumenting all functions simultaneously. (At least none of the other comments in this thread point at tools that do this.)
- cthalupa 11y ago
  >From reading it, I believe no other tool is both (1) non-sampled and (2) supports instrumenting all functions simultaneously. (At least none of the other comments in this thread point at tools that do this.) Perf supports dynamic tracing of probepoints, and probing multiple points simultaneously. Perhaps I'm misunderstanding things, but I don't see how perf doesn't fit those two requirements. It does a lot more than just sampling
  - DannyBee 11y ago
    1. Errr, perf definitely does not support inserting N probepoints, where N is the number of function entry + exit in a program. 2. Perf is 14x slower with tracepoints, and is definitely not that low overhead with tons and tons of probe points. See http://events.linuxfoundation.org/sites/events/files/slides/perf-collabsummit-2015.pdf http://events.linuxfoundation.org/sites/events/files/slides/...
    - cthalupa 11y ago
      Is there something I'm missing in the XRay paper that mentions speed? The only overhead I can find mentioned in the paper is CPU and memory utilization. If XRay is able to trace millions of probe points with less overhead than uprobe + frontends, that's extremely impressive... But I just don't see any numbers so far that have said this is the case?
      - DannyBee 11y ago
DannyBee 11y ago
So, to be fair to the xray people, what they have done meets a very specific set of use cases you can't meet with most other tracing tools (I haven't looked at dtrace hard enough. at a glance, it looks like it would not meet several things, but dtrace is not a sane option for them for a variety of reasons). Past that, the mechanism used to do this, and the system itself, is not uncommon to build. I don't see any claims that say they think they've broken new ground. Just that they built it :)

by 4ad 11y ago

bitmapbrother 11y ago

Does perf or Dtrace have the same functionality? >XRay allows you to get accurate function call traces with negligible overhead when off and moderate overhead when on, suitable for services deployed in production. XRay enables efficient function call entry/exit logging with high accuracy timestamps, and can be dynamically enabled and disabled.

4ad 11y ago
Yes?
- DannyBee 11y ago
  First, i always assume dtrace can do something. Dtrace is not that low overhead in the situation that xray is describing. http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provider-overhead/ http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provid... When they traced everything, it was two orders of magnitude slower. At 100k event/s (quite possible in the situations x-ray is used), the app would be about 60% slower. For just a simple probe, not full on tracing. Does this make it better than dtrace? No. It just serves a different set of use cases.
  - dap 11y ago
    > When they traced everything, it was two orders of magnitude slower. > At 100k event/s (quite possible in the situations x-ray is used), the app would be about 60% slower. For just a simple probe, not full on tracing. I think you've misunderstood the post. The actual time reported in that post is 600 ns per probe, with sanity checks reporting as much as 2000ns, which he concluded was in the same ballpark. He measured that by observing a program that just calls two functions in a tight loop, one of which is strlen() on a dozen-character string. That's basically the worst possible case for any function call tracing system -- I don't think it's fair to say that's "a simple probe, not full on tracing". The "60%" slower conclusion is only by construction: if you have N events per second, and the framework adds overhead of T nanoseconds per probe, then you can pick N such that the tracing framework adds whatever percentage overhead you like. In this case, Brendan picked N=100K and came up with 60% overhead for that case, but I think there's a math error in there. For that calculation, he assumes a 6us probe time instead of 600ns. I think the overhead would be 6% for 100,000 events, not 60%.
    - DannyBee 11y ago
      I didn't misunderstand. The problem with this is that you assume that if you probe every function instead of just strlen that you get the same time bounds. The only way to do this is, afaik, something similar to http://docs.oracle.com/cd/E19253-01/819-5488/gcgmc/index.html http://docs.oracle.com/cd/E19253-01/819-5488/gcgmc/index.htm... This really just is going to expand to putting a probe on every function. I'd really like to see numbers on how dtrace handles many millions of probes at once (which is what xray is handling). It's not uncommon to have 10 or 100 million functions in some of these programs. I have strong doubts that dtrace has the same overhead given 200 million probes. (since the probe data structures alone will likely take up gigabytes of memory, accessing them is unlikely to be cache friendly, etc) AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P TL;DR dynamic probing infrastructures are not a panacea
      - dap 11y ago
        > I'd really like to see numbers on how dtrace handles many millions of probes at once (which is what xray is handling). It's not uncommon to have 10 or 100 million functions in some of these programs. I have strong doubts that dtrace has the same overhead given 200 million probes. I see now. That's a fair question, and I'm not aware of data either way. I just tried a pretty simple experiment inspired by Brendan's that suggests that on my machine, the overhead is about 1450ns per probe for as many as 140,000 probes: https://gist.github.com/davepacheco/a12a0d45d55f0d7a28c312c2cd3cf234 https://gist.github.com/davepacheco/a12a0d45d55f0d7a28c312c2... > AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P Well, DTrace as architected is always going to pay the cost of a context switch into the kernel for each probe, and I think it's fair to take Brendan's result of 600ns as a lower bound of the per-probe overhead, at least on his machine. However, once in the kernel, for a typical native program (i.e., not JIT), I expect DTrace would only record the current userland thread instruction pointer. Names are typically resolved asynchronously by the consumer. So I would be surprised if it really was much slower, especially given the result above, but I too would like to see data. I'm not saying that DTrace solves all problems or even that the OP should have used it instead. It's certainly true that for the special case of userland function boundary tracing, one might expect to do better by skipping the context switch (at the expense of much functionality, including any ability to correlate with broader system activity). But since DTrace was brought up, I wanted to help clarify the uncertainty about what it can do and what its overhead is.

teraflop 11y ago

I don't know about DTrace, but perf is a sampling profiler, not an accurate call tracer.

4ad 11y ago
Perf is much more than a sampling profiler: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page. It can use uprobes and kprobes. There are many ways to do dynamic tracing in Linux, apart from perf. Ftrace, raw kprobes/uprobes, eBPF, bcc compiler for eBPF, etc.
- teraflop 11y ago
  After perusing the perf man page, the only way I could figure out how to make it accurately count userspace function calls was using hardware breakpoints, e.g.: "perf stat -e mem:0xADDRESS:x" Obviously that's not a very good approach, because you're limited to the number of breakpoints that your CPU can handle simultaneously (4 on my machine) and there's a lot of overhead. If you know of a better way to accomplish the same thing with perf, I'd be happy to hear it.
  - wmf 11y ago
    Check out http://www.brendangregg.com/perf.html#DynamicTracingEg http://www.brendangregg.com/perf.html#DynamicTracingEg
  - cthalupa 11y ago
    https://lwn.net/Articles/499190/ https://lwn.net/Articles/499190/ & https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-space-probe-instrumentation/ https://gnu.wildebeest.org/blog/mjw/2012/05/24/pull-user-spa... http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-upr...
    - teraflop 11y ago
      Ah, cool. I just tested it out and it seems to work as documented. Unfortunately it requires root access, and incurs about 1 microsecond of overhead per function call on my machine.

asuffield 11y ago

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google) I recommend reading page 2 of the paper, which discusses the specific set of features that XRay offers. How do I get perf or DTrace to give me the six things listed there? I can only think of ways to get a couple of them.

dap 11y ago
> The cost is acceptable when tracing and barely measurable when not tracing. "Acceptable" is obviously relative, but with DTrace's pid provider, the cost is zero when not tracing, and about the cost of a fastcall per probe point when enabled. > Instrumentation is automatic and directed towards functions that are important for understanding the binary’s execution time. I'm not sure what this means, but with DTrace, you enumerate the functions or binary objects (with wildcards and such) that you want to instrument, and the framework takes care of reliably instrumenting them, no matter the state of the process. Is that "automatic" and "directed"? I need to read the rest of the paper more closely. > Tracing is efficient in both space and time -- only recording what is required and what matters. DTrace records exactly what you ask it to. It supports in-situ aggregation for cases where it's not tenable to record a complete log of all interesting events. This is an important part of the design. > Tracing is configurable with thresholds for storage (how much memory to use) and accuracy (whether to log everything or only function calls taking at least some amount of time). With DTrace, it's pretty easy to filter on function execution time. The buffer size is configurable. There are also multiple buffer policies for different use-cases (e.g., ringbuffer of the last N events leading up to some other event). > Tracing does not require changes to the operating system nor super-user privileges. If they're running Linux, as I imagine they are, DTrace isn't necessarily an option. Several other platforms have just ported it. Using it to record user-level state on your own processes does not require superuser privileges. > Tracing can be turned on and off dynamically without having to restart the server. Absolutely -- that's what the "D" is for. I'd strongly recommended checking out the DTrace paper: https://www.usenix.org/legacy/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/ https://www.usenix.org/legacy/event/usenix04/tech/general/fu... There may be good reasons not to use DTrace for this, but I'm not sure which of those six goals would be the sticking point other than OS availability. (edit: I also haven't read beyond that yet!)

evmar 11y ago

It's definitely the sort of paper that would get reviewer feedback if it was submitted to a conference, in that it failed to compare any related work. (But it's unfair to grade it by that metric because it reads like more of a "here's what we did" blog post, not a research paper.) From reading it, I believe no other tool is both (1) non-sampled and (2) supports instrumenting all functions simultaneously. (At least none of the other comments in this thread point at tools that do this.)

cthalupa 11y ago
>From reading it, I believe no other tool is both (1) non-sampled and (2) supports instrumenting all functions simultaneously. (At least none of the other comments in this thread point at tools that do this.) Perf supports dynamic tracing of probepoints, and probing multiple points simultaneously. Perhaps I'm misunderstanding things, but I don't see how perf doesn't fit those two requirements. It does a lot more than just sampling
- DannyBee 11y ago
  1. Errr, perf definitely does not support inserting N probepoints, where N is the number of function entry + exit in a program. 2. Perf is 14x slower with tracepoints, and is definitely not that low overhead with tons and tons of probe points. See http://events.linuxfoundation.org/sites/events/files/slides/perf-collabsummit-2015.pdf http://events.linuxfoundation.org/sites/events/files/slides/...
  - cthalupa 11y ago
    Is there something I'm missing in the XRay paper that mentions speed? The only overhead I can find mentioned in the paper is CPU and memory utilization. If XRay is able to trace millions of probe points with less overhead than uprobe + frontends, that's extremely impressive... But I just don't see any numbers so far that have said this is the case?
    - DannyBee 11y ago

DannyBee 11y ago

So, to be fair to the xray people, what they have done meets a very specific set of use cases you can't meet with most other tracing tools (I haven't looked at dtrace hard enough. at a glance, it looks like it would not meet several things, but dtrace is not a sane option for them for a variety of reasons). Past that, the mechanism used to do this, and the system itself, is not uncommon to build. I don't see any claims that say they think they've broken new ground. Just that they built it :)