6 ms·

Sysdig vs DTrace vs Strace: a technical discussion

30 points by craneca0 13y ago

strictfp 13y ago
Dynamic code execution in the kernel - what could possibly go wrong?
- donavanm 13y ago
  Thats how almost every modern tracing facility works. Headaches are generally simplified by no malloc, no backtracking, no preemption and a limited DSL with strict compilation.
donavanm 13y ago
Documentation is sparse in details. Is this built on top of kprobes[1]? If so a comparison with perf, systemtap, et al is probably more appropriate. Do you support utrace/uprobes or similar for userland work? Is data selection/manipulation/summarization available on the kernal side? At high rates extracting enough/iptimal data has caught me with other linux probe based tools. I like the ring buffer and scap concept. Very nice compared to the full dump or summarized stats other tools focus on. [1] https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt
- degio 13y ago
  No, we don't use kprobes at this point, all of our collection is done through tracepoints. Take a look at https://github.com/draios/sysdig/blob/master/driver/main.c https://github.com/draios/sysdig/blob/master/driver/main.c for details. They tend to be very efficient, which is a very important requirement because of our "capture all" approach. And no urace/uprobes support yet. We're considering it as a feature for the future.
deleted 13y ago
[deleted]
brendangregg 12y ago
Isn't it premature to be comparing sysdig to DTrace?
- ylluminate 12y ago
  While dtrace is extremely robust, it is usually, in my case, not pulled up thanks to the level of expertness needed to really dig properly. This seems to really help welcome usage much more frequently and, with the OSX core / kernel team lagging as they have been for years, we really need additional instrumentation like this.
  - brendangregg 12y ago
    If I took a version of DTrace and removed about 95% of it's functionality, you'd have what sysdig currently does. It'd be a lot easier to learn, because there wouldn't be much to learn. The sysdig command line style is promising, and it's exciting to see innovation in this space, but what's there right now is still give and take. Try writing a DTrace aggregation and a sysdig chisel side by side, to see what I mean. In many common cases sysdig is much more laborious to use right now. More functionality can be added to sysdig (aggregations -- or chisel functions, thread local variables, tracepoints, kprobes, uprobes, PMCs, register inspection, kernel stacks, user stacks, user stack helpers, kernel filters and aggregations), allowing it to catch up to what DTrace can do, and solve the problems it can solve. But I think comparing sysdig to DTrace is either wrong or premature. If sysdig isn't going to do those features I mentioned, then it's a different type of tool (nothing wrong with that), but it is misleading and unfair on itself to compare it to DTrace. If it is going to do those features -- and it just hasn't yet -- then why compare it to DTrace now? sysdig will be giving people a poor first impression, and again, it's unfair on itself.
ylluminate 12y ago
Fantastic tool and it feels a lot better than digging around with `dtrace`. Really looking forward to help on the OSX side as it will help us tremendously.
fche 12y ago
A comparison to perf (specifically its use of tracepoints, ring-buffers, user-side scripted postprocessing) would be apropos.
- brendangregg 12y ago
  Sure, perf_events may be a better comparison, but I'd still wait until more functionality was in sysdig before trying to compare. Otherwise I think it's unfair on sysdig. For anyone wanting to compare, who isn't that familiar with perf, here's _some_ of the things perf can do: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html .

30 points by craneca0 13y ago

strictfp 13y ago

Dynamic code execution in the kernel - what could possibly go wrong?

donavanm 13y ago
Thats how almost every modern tracing facility works. Headaches are generally simplified by no malloc, no backtracking, no preemption and a limited DSL with strict compilation.

donavanm 13y ago

Documentation is sparse in details. Is this built on top of kprobes[1]? If so a comparison with perf, systemtap, et al is probably more appropriate. Do you support utrace/uprobes or similar for userland work? Is data selection/manipulation/summarization available on the kernal side? At high rates extracting enough/iptimal data has caught me with other linux probe based tools. I like the ring buffer and scap concept. Very nice compared to the full dump or summarized stats other tools focus on. [1] https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt

degio 13y ago
No, we don't use kprobes at this point, all of our collection is done through tracepoints. Take a look at https://github.com/draios/sysdig/blob/master/driver/main.c https://github.com/draios/sysdig/blob/master/driver/main.c for details. They tend to be very efficient, which is a very important requirement because of our "capture all" approach. And no urace/uprobes support yet. We're considering it as a feature for the future.

deleted 13y ago

[deleted]

brendangregg 12y ago

Isn't it premature to be comparing sysdig to DTrace?

ylluminate 12y ago
While dtrace is extremely robust, it is usually, in my case, not pulled up thanks to the level of expertness needed to really dig properly. This seems to really help welcome usage much more frequently and, with the OSX core / kernel team lagging as they have been for years, we really need additional instrumentation like this.
- brendangregg 12y ago
  If I took a version of DTrace and removed about 95% of it's functionality, you'd have what sysdig currently does. It'd be a lot easier to learn, because there wouldn't be much to learn. The sysdig command line style is promising, and it's exciting to see innovation in this space, but what's there right now is still give and take. Try writing a DTrace aggregation and a sysdig chisel side by side, to see what I mean. In many common cases sysdig is much more laborious to use right now. More functionality can be added to sysdig (aggregations -- or chisel functions, thread local variables, tracepoints, kprobes, uprobes, PMCs, register inspection, kernel stacks, user stacks, user stack helpers, kernel filters and aggregations), allowing it to catch up to what DTrace can do, and solve the problems it can solve. But I think comparing sysdig to DTrace is either wrong or premature. If sysdig isn't going to do those features I mentioned, then it's a different type of tool (nothing wrong with that), but it is misleading and unfair on itself to compare it to DTrace. If it is going to do those features -- and it just hasn't yet -- then why compare it to DTrace now? sysdig will be giving people a poor first impression, and again, it's unfair on itself.

ylluminate 12y ago

Fantastic tool and it feels a lot better than digging around with `dtrace`. Really looking forward to help on the OSX side as it will help us tremendously.

fche 12y ago

A comparison to perf (specifically its use of tracepoints, ring-buffers, user-side scripted postprocessing) would be apropos.

brendangregg 12y ago
Sure, perf_events may be a better comparison, but I'd still wait until more functionality was in sysdig before trying to compare. Otherwise I think it's unfair on sysdig. For anyone wanting to compare, who isn't that familiar with perf, here's _some_ of the things perf can do: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html .