4 ms·

Thanks to BPF devs and system folks like Brendan Gregg Linux tracing on the 4.x kernels can rival Solaris/FreeBSD + Dtrace. So my question is where does Sysdig

Thanks to BPF devs and system folks like Brendan Gregg Linux tracing on the 4.x kernels can rival Solaris/FreeBSD + Dtrace. So my question is where does Sysdig fit in the tracing equation here?

V-eHGsd_ 9y ago
practically, sysdig is how something like carbonblack should be doing its monitoring on linux. so to more directly answer your question; sysdig is the linux equivalent of the kernel module (plus userland grabby bits) needed for the executive pointy-clicky dashboard.
apurvadave 9y ago
I'm not sure if this fully answers your question, but Loris (the sysdig creator) wrote up a technical analysis of sysdig vs dtrace vs strace a while ago. https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discussion/ https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technic... It's worth a read.
deleted 9y ago
[deleted]
viraptor 9y ago
Does Linux perf / bpf tracing allow dealing with userland these days? I wish I could start reading info from live python and ruby like I can with dtrace, but I haven't really seen a way to do that yet.
- aseipp 9y ago
  Yes it does. You can use Userland Statically Defined Trace Points (USDTs) in order to augment arbitrary applications with probe points that can be monitored by tools like BCC 'trace'. The library that provides the API for these trace points is SystemTap (sys/sdt.h), which is -- AFAICS -- the primary method of USDT instrumenting on Linux. Conveniently, SystemTap's USDT API is the exact same as DTrace's API: you can simply use sprinkle `DTRACE_PROBEn` calls around and it will work fine on either, use the 'dtrace' tool to generate code from a provider, etc etc. Their dtrace tool is just a thin script that's compatible with the FreeBSD variant. (The nice part about this is that you should just be able to relax e.g. your autoconf checks from enabling DTRACE_PROBEn calls on FreeBSD/Solaris to include Linux as well, so migration should be smooth) Check out some info here[1]. Alternatively, for "simple" cases where you do not need provider support (e.g. inputs to the trace points), if the executable just has enough symbol information for function names, or the symbol names are exported anyway, you can generally glean some relevant information. For example, the following command uses the 'funclatency' tool to dump a latency histogram of every function call to any function with the word 'poke' in its name, for the application named 'app' in the CWD: sudo /usr/share/bcc/tools/funclatency -F -T \ $(pwd)/app:*poke* \ -p $(pidof app) NB: It is also worth noting that 'perf' on Linux has some form of uprobes support; e.g. it can probe functions based on the symbol name, but only for the events/interfaces it has available, I believe. [1] http://blogs.microsoft.co.il/sasha/2016/03/30/usdt-probe-support-in-bpfbcc/ http://blogs.microsoft.co.il/sasha/2016/03/30/usdt-probe-sup...

Thanks to BPF devs and system folks like Brendan Gregg Linux tracing on the 4.x kernels can rival Solaris/FreeBSD + Dtrace. So my question is where does Sysdig fit in the tracing equation here?

V-eHGsd_ 9y ago

practically, sysdig is how something like carbonblack should be doing its monitoring on linux. so to more directly answer your question; sysdig is the linux equivalent of the kernel module (plus userland grabby bits) needed for the executive pointy-clicky dashboard.

apurvadave 9y ago

I'm not sure if this fully answers your question, but Loris (the sysdig creator) wrote up a technical analysis of sysdig vs dtrace vs strace a while ago. https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discussion/ https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technic... It's worth a read.

deleted 9y ago

[deleted]

viraptor 9y ago

Does Linux perf / bpf tracing allow dealing with userland these days? I wish I could start reading info from live python and ruby like I can with dtrace, but I haven't really seen a way to do that yet.

aseipp 9y ago
Yes it does. You can use Userland Statically Defined Trace Points (USDTs) in order to augment arbitrary applications with probe points that can be monitored by tools like BCC 'trace'. The library that provides the API for these trace points is SystemTap (sys/sdt.h), which is -- AFAICS -- the primary method of USDT instrumenting on Linux. Conveniently, SystemTap's USDT API is the exact same as DTrace's API: you can simply use sprinkle `DTRACE_PROBEn` calls around and it will work fine on either, use the 'dtrace' tool to generate code from a provider, etc etc. Their dtrace tool is just a thin script that's compatible with the FreeBSD variant. (The nice part about this is that you should just be able to relax e.g. your autoconf checks from enabling DTRACE_PROBEn calls on FreeBSD/Solaris to include Linux as well, so migration should be smooth) Check out some info here[1]. Alternatively, for "simple" cases where you do not need provider support (e.g. inputs to the trace points), if the executable just has enough symbol information for function names, or the symbol names are exported anyway, you can generally glean some relevant information. For example, the following command uses the 'funclatency' tool to dump a latency histogram of every function call to any function with the word 'poke' in its name, for the application named 'app' in the CWD: sudo /usr/share/bcc/tools/funclatency -F -T \ $(pwd)/app:*poke* \ -p $(pidof app) NB: It is also worth noting that 'perf' on Linux has some form of uprobes support; e.g. it can probe functions based on the symbol name, but only for the events/interfaces it has available, I believe. [1] http://blogs.microsoft.co.il/sasha/2016/03/30/usdt-probe-support-in-bpfbcc/ http://blogs.microsoft.co.il/sasha/2016/03/30/usdt-probe-sup...