Sure, there are going to be a few performance sensitive workloads that you can't use it on, but for the vast majority of issues I've used it for, it's been far

Sure, there are going to be a few performance sensitive workloads that you can't use it on, but for the vast majority of issues I've used it for, it's been far more useful than perf or dtruss (on Mac OS X, I've never used dtruss on Solaris so I don't know if it's in better shape). The big advantage is that it already knows how to decode most syscalls, decompose flags into symbolic arguments, decode things like stat return values, and so on. dtruss on Mac OS X can decode some syscalls, but doesn't generally know how to decode structures or pull apart flags, so there's some information you just can't see, and some which you have to laboriously manually decode. perf, as you note, decodes even less than dtruss. When I'm trying to debug an issue, the more information I can gather and see at once, the better. Very few of the occasions I've had to use these tools have been particularly performance sensitive. Your example of the huge difference is because dd is copying fairly small blocks at a time, and thus making a fairly high number of system calls relative to the amount of data being transferred. Usually, I'm just looking for why an operation fails, or why a process is stuck; if it takes a little longer to get to that failure, that's OK, and if the process is stuck then stracing it generally won't affect performance. sysdig looks pretty interesting, thanks for the reference! It's too bad they can't use the existing perf infrastructure in the kernel, and need to add their own kernel module; that makes it that much more cumbersome to deploy and use.