11 ms·
Using /proc to get a process' current stack trace
- Annatar 8y agoInstead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process, the article is steeping them in GNU/Linux only way of doing things. This feels exactly like the '90's of the past century, where a lot of people with computer-related careers had no idea that there were other operating systems and other ways of doing things (better): an intel-based PC tin bucket with Windows was the one and only truth for them. Now it's exactly the same except Windows has been replaced with GNU/Linux. 28 years later and the only advancement some people have made is running the proverbial sed 's/Windows/Linux/g'.
- zerkten 8y agoCould you write a blog post in response that describes the better way to achieve this? There are possibly related items which you could throw in for others on achieving better portability.
- someguydave 8y agoUmm, what is the alternative O/S that has complete hardware support?
- dman 8y agoWhat is the non alternative O/S that has complete hardware support? fwiw openbsd / freebsd have been painless to install on every machine I have tried in the last 5 years.
- justinsaccount 8y agoHow's the support for bluetooth on openbsd these days?
- someguydave 8y agoAlso, how many of the aforementioned bsds use drivers that were ported from the Linux kernel tree?
- oneanddone 8y agoBSD user here. Of course there is porting occurring, but this is the nature of open source, is it not? Oftentimes, the BSD devs will take whatever hardware specs they can get and code something from that. OpenBSD frequently writes drivers that are orders of magnitude smaller than comparable Linux drivers, while getting the same functionality. NVIDIA drivers are but one example. Some hardware vendors are generous with hardware specs for OSS, most are not. OpenBSD, as an example, will not have any GPL'd code in the base OS, as it's not truly free. Ports are another issue.
- monocasa 8y ago> will not have any GPL'd code in the base OS, as it's not truly free. "My freedom to lock down someone else's code is more important than my user's freedom to have access to the code."
- TheDong 8y ago> will not have any GPL'd code in the base OS, as it's not truly free That's a disingenuous way to put it. The GPL is a free software license. It is "free" as in libre by all common definitions of "free" in regards to software. The GPL is not compatible with the BSD dev's preferred license, which is much more likely the reason they avoid using such code. By all conventional definitions, the BSD and GPL licenses are "truly free [software licenses]." You're welcome to argue that the BSD license is better (because, for example, it lets sony create derivative playstation OSs without providing that source code to their users, ensuring their users have less freedom than if it were linux) or that the GPL is better (because it would prevent the previous), but they're both free licenses.
- dman 8y agoI dont understand how ports help / hinder usability of drivers. Could you please elaborate?
- Annatar 8y agoAny illumos-based operating system: SmartOS, OmniOS, Tribblix, or any BSD-based one: OpenBSD, NetBSD, FreeBSD.
- monocasa 8y agoThis mainly about grabbing the kernel stack trace. There's not a portable way to do that. Also, doesn't SIGABRT generally kill off the process?
- Annatar 8y agoA kernel should always be compiled with symbols / source code inside of it -- that the Linux kernel doesn't have full support for CTF says more about it than it does about CTF. Yes, a SIGABRT will get you a core file and will kill a process, but if your process is hanging in an endless loop (like the author's was), one already has far bigger problems, and keeping such a process running will not amount to much.
- monocasa 8y agoI can't think of a single OS that gives you kernel stack traces on SIGABRT. Can you disambiguate CTF? I only know that as capture the flag, which doesn't really make sense here. Edit: ok, I figured it out. No theyre not going to give you that raw information because it's a kernel ASLR bypass. You can totally get all the same information with the dwarf symbols, but you're going to have to opt in on a kernel for it to mean anything. Edit2: bitching about how people aren't using portable Unix techniques, and then citing features that are Solaris specific isn't a great look.
- jwilk 8y agoTell us what you figured out…
- monocasa 8y agoHaha sorry. It's a Solaris version of DWARF or pdb style metadata for their kernel. https://docs.oracle.com/cd/E19253-01/816-5041/syntax-20/index.html https://docs.oracle.com/cd/E19253-01/816-5041/syntax-20/inde... For that to be useful, you'd need the addresses in question, which is why it'd be an ASLR bypass. The kernel needs to give higher level information by default so it can sanitize the output. And the debugging symbols exist on Linux anyway, they're just DWARF (which ironically is the more standards compliant way as opposed to CTF).
- pjmlp 8y agoYes, it feels strange to hear millennials talk about UNIX, when they are actually talking about GNU/Linux, and many things don't apply to e.g. OS X, Aix or many other variants.
- monocasa 8y ago...the article doesn't say unix at all with the exception of when the go code at the end is in fd_unix.go in it's source trace.
- mfukar 8y agoUsing SIGABRT gets you a stack trace in the same manner as a burnt house excuses you from sweeping the floors.
- seanhunter 8y agoThat's a very strange response. This is an article about Linux specifically. The author tags the article with Linux and mentions that it's part of a series of articles about Linux. It's not in and of itself a bad idea to have articles that specialise in a specific unix flavour. Secondly, SIGABRT will cause the process to abort and dump core, will it not? That would give you a userspace stack trace (if you load the core into a debugger) whereas this is how you get the kernel-side stack trace of a still-running process. I don't know of a way to get the kernel stack trace of a process in a cross-platform way. Is there such a thing?
- jstanley 8y agoEven if that were the case, a free software hegemony is surely better than a proprietary Microsoft hegemony.
- Annatar 8y agoWrite for yourself -- I've no problem paying Apple Computer for a solution which I plop down on a table, turn on, and start using immediately after that without having to fiddle with it.
- monocasa 8y agoThey regularly fuck up security so much that you aren't getting what you pay for. In the last major release they both allowed anyone to login as root by just not supplying a password, and writing the FDE password on the disk in plaintext as the "hint" instead of the actual hint.
- geofft 8y ago> Instead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process Sending SIGABRT doesn't do what the article is talking about, on any UNIX. Perhaps you would learn something from listening to the kids these days, like the distinction between a kernel stack and a userspace stack.
- Annatar 8y agoMy illumos based kernel begs to differ. And I'll listen to the kids when they actually start understanding what they are doing and why they are doing it, that is, when they learn how to use a computer.
- geofft 8y agoCan you post a transcript of doing so and getting the kernel stack? On not just Illumos but other UNIXes, to demonstrate that it is portable?
- cthalupa 8y agoSIGABRT in most places is not going to both get the stack trace and keep the process running. In fact, I would argue that if you are running something that continues working after it receives a signal telling it to /abort/, it's a bug. What do the word abort mean to you? As for SmartOS, as someone who ran it at home and in production for years: Keep flying that flag, I guess. I liked it. But I also realized that Joyent, even with the Samsung acquisition, does not have the resources to keep it going in any meaningful way for anyone beyond themselves and people who have the exact same usecase as them. Things like lx branded zones are clever. I miss SMF, and am not a fan of systemd. But bpf is better than dtrace. Container management is easier than zone management. I've got less bugs dealing with KVM on Linux than I ever did on SmartOS. I spend less time compiling things from source and having to find random patch files to make things work. I know plenty about Solaris and SmartOS and HPUX and AIX and the BSDs and I don't think anyone is making the incorrect choice in deciding to learn Linux over any other UNIX-like. That ship has sailed, man. And there's no compelling reason that it shouldn't have.
- Annatar 8y ago"Container management is easier than zone management." Linux has no containers, cgroups aren't anything conceptually close to zones. There are 56 different solutions to virtualization on Linux, all competing, mainly because everyone there is still flapping on deployment and lifecycle management. We'll just have to disagree and vehemently at that. As for Joyent having no resources: you debug it and fix it yourself. That's precisely how Linux got to be the hegemony that it is today. Oh, how quickly we forget, how short our memory is...
- sctb 8y agoSounds like a good post for Hacker News! If you find it or write it, please go ahead and submit.
- lixtra 8y agoIf you are running java instead of a c program the proc stacktrace shows you just the virtual machine state. You can still get a stacktrace of your java threads[1]. How about other languages? Python, ruby? [1] https://stackoverflow.com/questions/4876274/kill-3-to-get-java-thread-dump https://stackoverflow.com/questions/4876274/kill-3-to-get-ja...
- monocasa 8y agoThis is about the kernel's stack trace during a system call, not the user space stack trace.
- scott_s 8y agoYes, and if you want to know the user space stack, you may be able to attach gdb to the process: https://stackoverflow.com/questions/2308653/can-i-use-gdb-to-debug-a-running-process https://stackoverflow.com/questions/2308653/can-i-use-gdb-to...
- benfrederickson 8y agoI wrote something that will get you the python interpreter stack from any running cpython process : https://github.com/benfred/py-spy/ https://github.com/benfred/py-spy/ , and rbspy can do the same for ruby https://github.com/rbspy/rbspy https://github.com/rbspy/rbspy
- astral303 8y agoAlso `jstack` from the shell
- scottlamb 8y ago> If you are running java instead of a c program the proc stacktrace shows you just the virtual machine state. No, this is a _kernel_ backtrace: what is happening in kernelspace on behalf of your process. If the work is being done in userspace (that is, in state R; the thread isn't in a syscall or page fault handler), you'll see essentially nothing here. I just tried it on a userspace busylooper and got this: [<0000000000000000>] exit_to_usermode_loop+0x57/0xb0 [<0000000000000000>] prepare_exit_to_usermode+0x20/0x30 [<0000000000000000>] 0xfffffffffffffff Java, Python, C++ nothings all look pretty similar. If you want a userspace stack trace, you need a different tool. If you're using an interpreted (or perhaps JITted) language, yes, you probably want something language-specific. Also note the current stack trace is a per-thread concept, not a per-process one. If you're looking at a multithreaded program, you want to target the thread(s) of interest with "/proc/<PID>/task/<TID>/stack".
- drewg123 8y agoFWIW, the equivalent in FreeBSD is 'procstat -kk $PID' Eg: % procstat -kk 5592 PID TID COMM TDNAME KSTACK 5592 103222 less - mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101 procstat can also do interesting things, like show current rusage state: % procstat -r 5592 PID COMM RESOURCE VALUE 5592 less user time 00:00:00.010805 5592 less system time 00:00:00.002444 5592 less maximum RSS 3172 KB 5592 less integral shared memory 192 KB 5592 less integral unshared data 80 KB 5592 less integral unshared stack 256 KB 5592 less page reclaims 199 5592 less page faults 0 5592 less swaps 0 5592 less block reads 4 5592 less block writes 0 5592 less messages sent 0 5592 less messages received 0 5592 less signals received 0 5592 less voluntary context switches 59 5592 less involuntary context switches 0
- drewg123 8y agoBTW, I totally suck at formatting stuff in various forums. Is there a way to post pre-formatted text here? Eg, like triple back-ticks will do in slack? ``` stuff... ```
- colanderman 8y agoPrepend with four(?) spaces.
- jwilk 8y agoTwo spaces is enough: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- deleted 8y ago[deleted]
- jhallenworld 8y agoI want this capability for embedded ARM systems. I should be able to call a function to have the current stack trace printed: https://communities.mentor.com/thread/16468 https://communities.mentor.com/thread/16468
- monocasa 8y agoI've done it, but it increases binary size by a whole lot. What worked better was a last chance hardware fault handler that wrote a partial core dump out to flash, and a tool to create an ELF core dump file from that that GDB can accept.
- stefan_ 8y agoThe most widely distributed embedded ARM system software in the world, Android, offers this. They use mini debug info (normal debug info sections compressed, IIRC) and then you can signal a daemon in the background to do a stacktrace on your application. Not on production images, though.
- monocasa 8y ago> The most widely distributed embedded ARM system software in the world, Android [citation needed] Also, when talking about embedded systems, people aren't normally talking about full Linux distros.
- _pmf_ 8y agoMy favorite proc hack: https://news.ycombinator.com/item?id=17061499 https://news.ycombinator.com/item?id=17061499
- chaosfox 8y agocheck out progress: https://github.com/Xfennec/progress https://github.com/Xfennec/progress
- AlphaWeaver 8y agoThis is a very well written and formatted article... I found it easy to read!
- indigodaddy 8y agoNicely formatted site. Anyone know if/what static site generator/theme is being used? Couldn't find anything on the footer, site tags, or his GitHub that would reveal that...
- curtis86 8y agoAgreed! It looks like Hugo and possibly this theme: https://themes.gohugo.io//theme/cocoa-eh-hugo-theme/blog/example-article/ https://themes.gohugo.io//theme/cocoa-eh-hugo-theme/blog/exa...
- ktpsns 8y agoNote, the GNU debugger (gdb) can attach to running processes. This should give you a stack trace with readable addresses, cf. https://stackoverflow.com/questions/2308653/can-i-use-gdb-to-debug-a-running-process https://stackoverflow.com/questions/2308653/can-i-use-gdb-to...
- drewg123 8y agoThe /proc interface tells you what the process is doing in the kernel (eg, after a system call), which is orthogonal to getting a user-space stack trace. Both are helpful. The userspace trace is arguably more helpful. The nice thing about the kernel trace is that you're likely to have symbols (or be able to download them); that is unlikely to be true for userspace if you're running a commercial binary, for example.