8 ms·
Why use strace in 2023? [video]
- mlyle 3y agostrace is basically my first go-to when a command immediately fails with an obscure message. strace, skim the last 4-5 pages of trace output before the error message is emitted, looking in particular for error return values (it's often not right before the error because the program has started to clean up). Figures out the problem 80% of the time within a couple of minutes.
- iforgotpassword 3y agoAnd then IME, most of the time it's some file missing, or you can't figure out where it's getting its configuration from, so a grep for ^open is a good first try.
- amelius 3y agoIsn't there a flag for getting only the opened filenames? (If not, why not)
- zwp 3y agoYou can use "-e trace=open" to trace only open(2) calls Alternatively "-e trace=%file" to get all file-related system calls (will catch eg failing pre-emptive checks using access(3) -> stat(2)).
- amelius 3y agoOk, but then you will still need to parse the output to get the filenames. That's ok, but since it is something that is used a lot, you'd expect a flag.
- tanelpoder 3y agoCheck out strace -y and strace -yy (--decode-fds)
- zwp 3y agoYou still need to pull out the paths? A sprinkling of grep/perl (awk/sed/ruby/...) is mostly good enough eg: strace -e trace=%file cat /etc/passwd 2>&1 >/dev/null | grep ^open | grep -Po '(?<=").*(?=")'
- eesmith 3y agoIs your example situation really all that common? If so, what format do you expect for the output? If it's one filename-per-line then how do you encode filenames with embedded newlines? How do you encode non-UTF8 characters, or is the file meant to be parsed only in binary mode? I don't know of any generally agreed upon spec for this, so no matter what you think is right, most people are going to have to write a special-purpose parser. In which case you might as well parse the native strace output since one is about as complex as the other.
- c0l0 3y agoFTR, on modern-ish glibc-powered systems (in code that actually does use libc, and does not do its very own syscall-related thing instead), you will not find a single call to open(2) issued, in my experience. That's because the library functions shadowing these syscalls were rewired to use openat(2) under the hood. $ strace -e trace=open cat /dev/null +++ exited with 0 +++ $ strace -e trace=openat cat /dev/null openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/dev/null", O_RDONLY) = 3 +++ exited with 0 +++
- lelanthran 3y agoIf you want to catch both `open` and `openat`, the opensnoop BPF[1] program is pretty nifty, especially if you are trying to figure out file stuff across several different programs ("which #$%^-ing program keeps modifying this file", for example). [1] I've been dipping my toes into BPF recently, and while complicated (best to simply clone the bpftools repo and work off of that) there's a lot that can be done that tools like strace won't be able to match.
- charleshn 3y agostrace's -Z flag is quite useful to filter only syscalls that failed. In a similar way, -k is useful to get tracebacks.
- Two9A 3y agoI should write that down; -Z would've been really useful when my SSH server was falling over in a heap and I was left scrolling through strace to work out what happened. I wrote that experience up last week, for what it's worth: https://imrannazar.com/articles/linux-upgrade-fail https://imrannazar.com/articles/linux-upgrade-fail
- ptsneves 3y agoSame here. I have a theory. The obscure error are due to unexpected Syscall results. Programmers are very aware of the states their programs can be into given the visible logic. They often are not aware of the possible states a syscall can be and their error modes. A pattern I see in python programs is the sys.exit and exception avoidance because they don’t want the user to see a stack trace. So they rely on conditionals and sys.exits, never considering and catching the exceptions that things like open and write can raise, leading ton exception traces anyway. This is one of the reasons strace sometimes is required. I have seen experienced programmers doing this pattern again and again and they are really convinced that it is helpful and consistent. This highlights the what I wrote above programmers often ignore the way external dependencies work, in this case the syscalls. Another common syscall related issue strace helps is with permissions denied. Most scripts just defend against the file existing and not whether the operation is possible, leading to obscure messages like operation not permitted. Strace to the rescue in such cases It is one oft first lines of debugging, even if I have the source code, as sometimes the syscall that fails is buried deep in library code, but with the failed syscall and arguments I can track the line that made the wrong call.
- cwillu 3y ago> I have seen experienced programmers doing this pattern again and again and they are really convinced that it is helpful and consistent. In a nutshell, this is why I hate working in go. While I'm sure your code, dear hn golang fan, doesn't suffer from this, much of the go code I've had to deal with extends the principle of “unaware of many common failure modes and swallow them into the exit code” to every function.
- angra_mainyu 3y agothat's strange, I've been working professionally in Go for years and two things happen: 1) what are exceptions in many languages are plain errors in Go 2) a defer-recover block is usually put at the very top of a risky call This covers 99% of sources of panics. Now, the one that truly does mess you up is when it comes from CGO/calling out to C. That will tear your application and there is nothing you can do about it.
- hawski 3y agoI often use it to see what configuration files the thing reads, it is much more reliable than documentation.
- bboozzoo 3y agoSince you almost always need to open a file in order to do anything useful `strace -vf -e openat` helped solve a lot of issues from simple configuration file lookup to ld.so library loading problems.
- thequux 3y agoIt is often useful to also watch for calls to the stat family of syscalls; some applications stat files that they wish to open rather than depending on the return code from open, so just looking for open will miss files that don't exist.
- agumonkey 3y agosame, and I'm often surprised how many files are touched / read, and also how many times (trying to load local configs from dozens of places).. some unix practices seems very inefficient
- zh3 3y agostrace solved an issue for me today - logging in to a different account on my debian PC using 'su -l guest' would hang for 30s before letting me in. Interestingly, 'strace su -l guest' failed with a permission error during the trace so looked like I was stuck ('sudo' didn't help). Until I opened a 2nd terminal window and did 'while [ 1 ]; do sudo strace -p $(pidof su); done' - that looped until I tried the login again, then attached, followed the calls and showed the delay was due to elogind and dbus; a quick 'apt purge elogind' then fixed the slow login.
- lelandbatey 3y agoTalking about strace per the title without watching; wow is strace ever helpful! You can build an ordered tree of all the programs launched by a program, recursively, using strace. You can do the same for observing files opened. It's an amazing tool when you need to observe some software from the outside; in many ways it'll give you much better understanding of a program than even reading it's source code.
- samsquire 3y agoThis would be an interesting GUI that could be in a web browser to cross reference effects between programs such as communication between programs and Wayland. Specifically the parameters of syscalls create an effect, like write to block. Wireshark for syscalls. I think it already exists. I kind of think of the interactions with the OS as the "effects" of the application like the IO Monad. It's like a protocol too, function application in a certain order causes the effects of the software. The syscalls is a stream. There is also a inner protocol between the buffers of binary protocols inside the syscalls such as a network buffer.
- eesmith 3y agoYou should watch the video then, as it goes into more details about new strace features introduced in the last few years, like print only system calls with a specified return status (successful, failed, etc.). I'm particularly intrigued about what I can do with system call tampering. I like the example of being able to make unlink() be a no-op, giving you easy access to a temporary file which is normally automatically cleaned up.
- eisvogel 3y agoWhy use C in 2024?
- Taniwha 3y agostrace works on anything that makes a system call - written in any language
- amelius 3y agoExcept it doesn't work on programs that call strace.
- Retr0id 3y agoWhy not? A simple "strace strace date" works on my machine.
- methou 3y agoI hated macOS for gating / crippling dtruss with SIP every time I need to find out what’s wrong with a random cmd. Is there any alternatives on Mac?
- loevborg 3y agoYeah the fact that macOS doesn't have strace is a serious shortcoming
- darkr 3y agodtrace is a far more comprehensive tool than strace. The common complaint on MacOS is that SIP by default will not allow you to use it. The same issue occurs on Linux when you're running SELinux or similar.
- GrumpySloth 3y agoSIP doesn’t block dtrace. You can use it just fine to e.g. trace dynamic library function calls. What really is a problem on MacOS is that they stopped shipping kernel syscall symbols. So you can’t trace syscalls anymore in the default setup. I’ve read on some forum that you can download the symbols from some website and it works then, but didn’t bother with it at that point. Ironically, despite DTrace originally being a Unix tool, I’ve had the most success using it on Windows. On Mac I can’t trace syscalls by default, on FreeBSD for some reason it didn’t notice that processes inside jails were opening files (despite DTrace running on the host)... On Windows on the other hand DTrace works perfectly for me.
- deathanatos 3y ago> SIP doesn’t block dtrace. » dtrace echo dtrace: system integrity protection is on, some features will not be available dtrace: failed to initialize dtrace: DTrace requires additional privileges
- GrumpySloth 3y ago
- tanelpoder 3y agoThe safer/non-slowing down alternative for tracing syscalls would be "perf trace" (and eBPF scripts), but I've found in past that perf trace didn't decode enough syscall arguments (and just listed pointers to structs, where strace showed the info inside these structs). But I just ran a few simple "perf trace" commands on RHEL 9, looks like it has caught up somewhat.
- ur-whale 3y agoOh boy, strace is soooo useful, and not just for coding or sysadmin. As a user, when e.g. my browser does funky stuff like freezing for no reason, my knee-jerk reflex is always strace. I was initially a little put down by the russian style moody intro (the longish "why would you use such an antiquated thing as strace in 2023" which made me think he was going to tear down strace), but stick with the video past it, it is actually an ode to strace and it's well worth it. Personnaly, I never realized strace had this many features.
- It_s_me 3y ago> As a user, when e.g. my browser does funky stuff like freezing for no reason, my knee-jerk reflex is always strace. I'm currently trying to figure out why the whole Chrome UI started freezing intermittently last two weeks, when scrolling, switching tabs, etc. What you're suggesting is using dtrace (MacOS) and see what's the output around that given time?
- dwattttt 3y agoFor those curious about the Windows equivalent, procmon is a system-wide strace tool; filtering mandatory.
- ExoticPearTree 3y agoanother useful tool is sysdig with its curses gui csysdig which summarizes operations, you can filter on them etc.
- flanked-evergl 3y agoSysdig is very useful, but it's almost always broken on Fedora when I want to use it. I have recently started using https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace instead, and so far it has covered the same use cases.
- zokier 3y agoSysdig is neat, but csysdig in my experience was extremely fragile and fickle, felt more like a demo than a usable tool. Which is a shame because it had some nice things to it. Also at this point I'd say sysdig in general is just another also-ran together with systemtap and lttng (and others)
- vcdimension 3y agoIn emacs you can use syslog-mode for analyzing strace output: https://github.com/vapniks/syslog-mode https://github.com/vapniks/syslog-mode It allows you to easily navigate, filter, highlight lines, and lookup documentation.
- alecst 3y agoWhat are some options on MacOs? Hard to do much with SIP enabled. Ran into the issue a few days ago and still no solution.
- dygd 3y agoktrace is somewhat similar, at least when it comes to syscall traces.
- INTPenis 3y agoHe mentions all these other ways of doing tracing and I assume he means eBPF, but I've yet to see anything useful I can run from the CLI to use that. So what is there out there? Because I'm a veteran Linux/Unix sysadmin for 25 years now and I love strace when I'm in a bind. So what should I look at now based on eBPF instead?
- tayo42 3y agoBrendan Gregg wrote about it alot https://www.brendangregg.com/ebpf.html https://www.brendangregg.com/ebpf.html Ime it was hard to set up, idk if anything has changed recently. Looks like it's just packages now.
- flanked-evergl 3y agoI have started using https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace for things that I once used sysdig for, it works quite well and is not broken as often as sysdig.
- zbentley 3y agoThe widespread switch to epoll(7)-based I/O systems has taken a big bite out of strace's usefulness for me. Back when most programs did most of their I/O synchronously or with select(2)/poll(2), it was really easy to use strace to track down slow network services. If you wanted to figure out why a process was taking too long, you could strace it, grab the file descriptor numbers that it was spending a long time reading/writing/selecting/polling, feed those into lsof and turn them into network addresses/ports, then shell into those remote hosts and see what it was doing with the slow client's connection (e.g. if the remote host was a MySQL database you could check what queries were running on its connection/session; if it was a RabbitMQ you could see what queues its connection was operating on). Don't get me wrong: strace is still an incredibly useful tool every Linux developer, no matter where they are in the stack, should know (Julia Evans made that case better than I could[1]). I also understand why I/O systems switched to more "file descriptor opaque" tools like epoll/kqueue/io_uring, which makes sense and brings a lot of benefits. I just miss the transparency of the old APIs a little bit, you know? 1: https://jvns.ca/blog/2014/04/20/debug-your-programs-like-theyre-closed-source/ https://jvns.ca/blog/2014/04/20/debug-your-programs-like-the...
- arminiusreturns 3y agoJust be aware that in prod strace can have real impacts and run-away side-effects please!
- frankjr 3y agoSuch as?
- arminiusreturns 3y agoInstead of explaining it poorly, try this admittedly older article from Brendan Gregg https://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html https://www.brendangregg.com/blog/2014-05-11/strace-wow-much... tldr - syscalls and userland / kernel context switching