5 ms·

strace is basically my first go-to when a command immediately fails with an obscure message. strace, skim the last 4-5 pages of trace output before the error m

• iforgotpassword 3y ago
  And then IME, most of the time it's some file missing, or you can't figure out where it's getting its configuration from, so a grep for ^open is a good first try.
  • amelius 3y ago
    Isn't there a flag for getting only the opened filenames? (If not, why not)
    • zwp 3y ago
      You can use "-e trace=open" to trace only open(2) calls Alternatively "-e trace=%file" to get all file-related system calls (will catch eg failing pre-emptive checks using access(3) -> stat(2)).
      • amelius 3y ago
        Ok, but then you will still need to parse the output to get the filenames. That's ok, but since it is something that is used a lot, you'd expect a flag.
        • tanelpoder 3y ago
          Check out strace -y and strace -yy (--decode-fds)
          • zwp 3y ago
            You still need to pull out the paths? A sprinkling of grep/perl (awk/sed/ruby/...) is mostly good enough eg: strace -e trace=%file cat /etc/passwd 2>&1 >/dev/null | grep ^open | grep -Po '(?<=").*(?=")'
        • eesmith 3y ago
          Is your example situation really all that common? If so, what format do you expect for the output? If it's one filename-per-line then how do you encode filenames with embedded newlines? How do you encode non-UTF8 characters, or is the file meant to be parsed only in binary mode? I don't know of any generally agreed upon spec for this, so no matter what you think is right, most people are going to have to write a special-purpose parser. In which case you might as well parse the native strace output since one is about as complex as the other.
          • amelius 3y ago
            It can use the same format as the Unix find utility. This utility has a -print0 flag to separate filenames by NUL characters instead of newlines if desired.
            • eesmith 3y ago
              That is a good point. I still don't see it as common use case.
      • c0l0 3y ago
        FTR, on modern-ish glibc-powered systems (in code that actually does use libc, and does not do its very own syscall-related thing instead), you will not find a single call to open(2) issued, in my experience. That's because the library functions shadowing these syscalls were rewired to use openat(2) under the hood. $ strace -e trace=open cat /dev/null +++ exited with 0 +++ $ strace -e trace=openat cat /dev/null openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/dev/null", O_RDONLY) = 3 +++ exited with 0 +++
        • lelanthran 3y ago
          If you want to catch both `open` and `openat`, the opensnoop BPF[1] program is pretty nifty, especially if you are trying to figure out file stuff across several different programs ("which #$%^-ing program keeps modifying this file", for example). [1] I've been dipping my toes into BPF recently, and while complicated (best to simply clone the bpftools repo and work off of that) there's a lot that can be done that tools like strace won't be able to match.
• charleshn 3y ago
  strace's -Z flag is quite useful to filter only syscalls that failed. In a similar way, -k is useful to get tracebacks.
  • Two9A 3y ago
    I should write that down; -Z would've been really useful when my SSH server was falling over in a heap and I was left scrolling through strace to work out what happened. I wrote that experience up last week, for what it's worth: https://imrannazar.com/articles/linux-upgrade-fail https://imrannazar.com/articles/linux-upgrade-fail
• ptsneves 3y ago
  Same here. I have a theory. The obscure error are due to unexpected Syscall results. Programmers are very aware of the states their programs can be into given the visible logic. They often are not aware of the possible states a syscall can be and their error modes. A pattern I see in python programs is the sys.exit and exception avoidance because they don’t want the user to see a stack trace. So they rely on conditionals and sys.exits, never considering and catching the exceptions that things like open and write can raise, leading ton exception traces anyway. This is one of the reasons strace sometimes is required. I have seen experienced programmers doing this pattern again and again and they are really convinced that it is helpful and consistent. This highlights the what I wrote above programmers often ignore the way external dependencies work, in this case the syscalls. Another common syscall related issue strace helps is with permissions denied. Most scripts just defend against the file existing and not whether the operation is possible, leading to obscure messages like operation not permitted. Strace to the rescue in such cases It is one oft first lines of debugging, even if I have the source code, as sometimes the syscall that fails is buried deep in library code, but with the failed syscall and arguments I can track the line that made the wrong call.
  • cwillu 3y ago
    > I have seen experienced programmers doing this pattern again and again and they are really convinced that it is helpful and consistent. In a nutshell, this is why I hate working in go. While I'm sure your code, dear hn golang fan, doesn't suffer from this, much of the go code I've had to deal with extends the principle of “unaware of many common failure modes and swallow them into the exit code” to every function.
    • angra_mainyu 3y ago
      that's strange, I've been working professionally in Go for years and two things happen: 1) what are exceptions in many languages are plain errors in Go 2) a defer-recover block is usually put at the very top of a risky call This covers 99% of sources of panics. Now, the one that truly does mess you up is when it comes from CGO/calling out to C. That will tear your application and there is nothing you can do about it.
• hawski 3y ago
  I often use it to see what configuration files the thing reads, it is much more reliable than documentation.
  • bboozzoo 3y ago
    Since you almost always need to open a file in order to do anything useful `strace -vf -e openat` helped solve a lot of issues from simple configuration file lookup to ld.so library loading problems.
    • thequux 3y ago
      It is often useful to also watch for calls to the stat family of syscalls; some applications stat files that they wish to open rather than depending on the return code from open, so just looking for open will miss files that don't exist.
  • agumonkey 3y ago
    same, and I'm often surprised how many files are touched / read, and also how many times (trying to load local configs from dozens of places).. some unix practices seems very inefficient
• zh3 3y ago
  strace solved an issue for me today - logging in to a different account on my debian PC using 'su -l guest' would hang for 30s before letting me in. Interestingly, 'strace su -l guest' failed with a permission error during the trace so looked like I was stuck ('sudo' didn't help). Until I opened a 2nd terminal window and did 'while [ 1 ]; do sudo strace -p $(pidof su); done' - that looped until I tried the login again, then attached, followed the calls and showed the delay was due to elogind and dbus; a quick 'apt purge elogind' then fixed the slow login.