6 ms·

Iosnoop For Linux

169 points by helper 12y ago

hrjet 12y ago
If it's not reliable as claimed by the article, then does it really serve the purpose? There might be a code-path in an innocent little app that calls brk() often and then gets tainted as causing IO. Aside, thanks for reminding me about blktrace. I had forgotten the name last week when I needed the tool.
- brendangregg 12y ago
  You can use it to study block device I/O details, confirming latency outliers, and analyzing device, size, and offset patterns. It's the COMM and PID columns that aren't reliable -- they show what's on-CPU at the time of block device issue, which is usually the origin PID, but sometimes not. This has always been the case for iosnoop (all versions). I'm explaining this caveat better today than I have in the past. As for the origin PID... I usually trace that from a different layer of the stack (vfs, file system, or syscall). But this approach has different caveats: the overhead is much higher (more frequent requests), and if I want to associate it with the block I/O, then many solutions are going to require debuginfo, and/or a different tracer. Also, what if a process buffers minutes of disk I/O then exits, and then that PID is reused by a different process calling disk I/O? It's a remote possibility, but I just wanted to mention another problem in this area.
cokernel_hacker 12y ago
Those of the OS X persuasion can find similar functionality in fs_usage. https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/fs_usage.1.html https://developer.apple.com/library/mac/documentation/Darwin...
- jatoben 12y ago
  fs_usage is a useful tool; the original version of iosnoop(1m) has also been available since Dtrace was added to OS X 10.5.
- frou_dh 12y ago
  There's sc_usage too for cumulative display of all system calls.
- shimshim 12y ago
  He also has a slide about OSX performance: http://www.slideshare.net/brendangregg/analyzing-os-x-systems-performance-with-the-use-method http://www.slideshare.net/brendangregg/analyzing-os-x-system...
rodgerd 12y ago
Very nice. Thanks Brendan!
itamarhaber 12y ago
Looks like a great tool - now I wish I had the storage to test it on...
jdesfossez 12y ago
Great tool ! It is interesting because we are working to extract the same kind of information but with LTTng for low intrusiveness and remote analysis of production servers. If you are interested, you can have a look at the following repository : https://github.com/jdesfossez/lttng-analyses https://github.com/jdesfossez/lttng-analyses
brendangregg 12y ago
I just added an experimental execsnoop to the repo as well: https://github.com/brendangregg/perf-tools https://github.com/brendangregg/perf-tools .
randomguy54 12y ago
All I see in COMM is ksoftirqd, btrfs-transaction or md127_raid. No real programs names :(
templeos2 12y ago
Bill Gates says, "You know assembly language? You're a hacker. <nudge nudge wink wink>" That's like saying to a plumber, "You're a plumber? You're a partisan." Da fuq is yer problem? (pipe bombs)

169 points by helper 12y ago

hrjet 12y ago

If it's not reliable as claimed by the article, then does it really serve the purpose? There might be a code-path in an innocent little app that calls brk() often and then gets tainted as causing IO. Aside, thanks for reminding me about blktrace. I had forgotten the name last week when I needed the tool.

brendangregg 12y ago
You can use it to study block device I/O details, confirming latency outliers, and analyzing device, size, and offset patterns. It's the COMM and PID columns that aren't reliable -- they show what's on-CPU at the time of block device issue, which is usually the origin PID, but sometimes not. This has always been the case for iosnoop (all versions). I'm explaining this caveat better today than I have in the past. As for the origin PID... I usually trace that from a different layer of the stack (vfs, file system, or syscall). But this approach has different caveats: the overhead is much higher (more frequent requests), and if I want to associate it with the block I/O, then many solutions are going to require debuginfo, and/or a different tracer. Also, what if a process buffers minutes of disk I/O then exits, and then that PID is reused by a different process calling disk I/O? It's a remote possibility, but I just wanted to mention another problem in this area.

cokernel_hacker 12y ago

Those of the OS X persuasion can find similar functionality in fs_usage. https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/fs_usage.1.html https://developer.apple.com/library/mac/documentation/Darwin...

jatoben 12y ago
fs_usage is a useful tool; the original version of iosnoop(1m) has also been available since Dtrace was added to OS X 10.5.
frou_dh 12y ago
There's sc_usage too for cumulative display of all system calls.
shimshim 12y ago
He also has a slide about OSX performance: http://www.slideshare.net/brendangregg/analyzing-os-x-systems-performance-with-the-use-method http://www.slideshare.net/brendangregg/analyzing-os-x-system...

rodgerd 12y ago

Very nice. Thanks Brendan!

itamarhaber 12y ago

Looks like a great tool - now I wish I had the storage to test it on...

jdesfossez 12y ago

Great tool ! It is interesting because we are working to extract the same kind of information but with LTTng for low intrusiveness and remote analysis of production servers. If you are interested, you can have a look at the following repository : https://github.com/jdesfossez/lttng-analyses https://github.com/jdesfossez/lttng-analyses

brendangregg 12y ago

I just added an experimental execsnoop to the repo as well: https://github.com/brendangregg/perf-tools https://github.com/brendangregg/perf-tools .

randomguy54 12y ago

All I see in COMM is ksoftirqd, btrfs-transaction or md127_raid. No real programs names :(

templeos2 12y ago

Bill Gates says, "You know assembly language? You're a hacker. <nudge nudge wink wink>" That's like saying to a plumber, "You're a plumber? You're a partisan." Da fuq is yer problem? (pipe bombs)