2 ms·

strace can slow the target by up to 200x (http://www.slideshare.net/brendangregg/linux-performance-analysis-and-tools/38 http://www.slideshare.net/brendangregg/

strace can slow the target by up to 200x (http://www.slideshare.net/brendangregg/linux-performance-analysis-and-tools/38 http://www.slideshare.net/brendangregg/linux-performance-ana...). While you can solve plenty of issues with it, you have to be very careful about its use in production. In many cases, it just can't be used. perf added a "trace" subcommand in 3.7, for buffered syscall tracing (http://www.brendangregg.com/perf.html#More http://www.brendangregg.com/perf.html#More). This should eventually be like strace, but with MUCH less performance overhead. sysdig is a newer tool that uses a similar low-overhead interface, and provides an expressive syntax. It's still early days, but sysdig could ultimately replace strace (or perf will via trace).

lambda 12y ago
Sure, there are going to be a few performance sensitive workloads that you can't use it on, but for the vast majority of issues I've used it for, it's been far more useful than perf or dtruss (on Mac OS X, I've never used dtruss on Solaris so I don't know if it's in better shape). The big advantage is that it already knows how to decode most syscalls, decompose flags into symbolic arguments, decode things like stat return values, and so on. dtruss on Mac OS X can decode some syscalls, but doesn't generally know how to decode structures or pull apart flags, so there's some information you just can't see, and some which you have to laboriously manually decode. perf, as you note, decodes even less than dtruss. When I'm trying to debug an issue, the more information I can gather and see at once, the better. Very few of the occasions I've had to use these tools have been particularly performance sensitive. Your example of the huge difference is because dd is copying fairly small blocks at a time, and thus making a fairly high number of system calls relative to the amount of data being transferred. Usually, I'm just looking for why an operation fails, or why a process is stuck; if it takes a little longer to get to that failure, that's OK, and if the process is stuck then stracing it generally won't affect performance. sysdig looks pretty interesting, thanks for the reference! It's too bad they can't use the existing perf infrastructure in the kernel, and need to add their own kernel module; that makes it that much more cumbersome to deploy and use.

lambda 12y ago

Sure, there are going to be a few performance sensitive workloads that you can't use it on, but for the vast majority of issues I've used it for, it's been far more useful than perf or dtruss (on Mac OS X, I've never used dtruss on Solaris so I don't know if it's in better shape). The big advantage is that it already knows how to decode most syscalls, decompose flags into symbolic arguments, decode things like stat return values, and so on. dtruss on Mac OS X can decode some syscalls, but doesn't generally know how to decode structures or pull apart flags, so there's some information you just can't see, and some which you have to laboriously manually decode. perf, as you note, decodes even less than dtruss. When I'm trying to debug an issue, the more information I can gather and see at once, the better. Very few of the occasions I've had to use these tools have been particularly performance sensitive. Your example of the huge difference is because dd is copying fairly small blocks at a time, and thus making a fairly high number of system calls relative to the amount of data being transferred. Usually, I'm just looking for why an operation fails, or why a process is stuck; if it takes a little longer to get to that failure, that's OK, and if the process is stuck then stracing it generally won't affect performance. sysdig looks pretty interesting, thanks for the reference! It's too bad they can't use the existing perf infrastructure in the kernel, and need to add their own kernel module; that makes it that much more cumbersome to deploy and use.