6 ms·

Kernel debugging for newbies

117 points by alambert 9y ago

Dimi9909 9y ago
why do you need to machines? just one linux box: build kernel code, then update-initramfs and update grub.
cuckcuckspruce 9y ago
Two machines seems like overkill in this case - why not debug using User Mode Linux[1]? Debugging with two machines makes perfect sense if you are debugging a hardware driver, but not here. [1] http://opensourceforu.com/2010/09/user-mode-linux-setup-and-debug/ http://opensourceforu.com/2010/09/user-mode-linux-setup-and-...
- AstralStorm 9y ago
  Because UML cannot interact with most hardware and that is where bugs happen. Even VMs do not provide real hardware or allow you to bring down the machine with it. Instead you can debug a crash using a trace and kexec dump. And also use quite fast ftrace infrastructure which is much better than plain old printf. There are also kprobes, gcov and oprofile.
bcantrill 9y ago
For those of us coming from non-Linux systems (and I'm speaking for illumos/SmartOS here), the effort required to debug the kernel displayed here is galling. That this is so difficult reflects Torvalds' historic disposition against kernel developers[1]: I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program. None of the arguments for a kernel debugger has touched me in the least. And trust me, over the years I've heard quite a lot of them. In the end, they tend to boil down to basically: - it would be so much easier to do development, and we'd be able to add new things faster. And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing. To be fair, this was a long time ago (17 years ago!), but the experience relayed here leaves one believing that Torvalds' historic attitude has cast a long shadow. And if it needs to be said, Torvalds' arguments are themselves deeply confused; he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation -- both of which have proved absolutely essential for kernel development. (Indeed, it is likely that DTrace alone would have allowed the author to debug their problem, as its design center is exactly the kind of non-fatal failure described.) The tutorial will certainly save others pain, but that such pain still exists at all in Linux is deeply unfortunate, and a vivid example of Linux not representing anything close to the state-of-the-art in systems development. [1] https://lwn.net/2000/0914/a/lt-debugger.php3 https://lwn.net/2000/0914/a/lt-debugger.php3
- racer-v 9y ago
  > he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. Where does he do this? The quote is only about single-step debugging. > So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.
  - AstralStorm 9y ago
    Not to mention relocatable kernel and kexec crash dump analysis.
  - bcantrill 9y ago
    No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.
    - richardwhiuk 9y ago
      It's a lot easier to configure using the modern API - the author is intentionally choosing an API which doesn't give much feedback....
      - xenophonf 9y ago
        That's true, but the author has solid reasons for doing so: > I want my application to work on OS X and Linux, so I’m targeting PF_KEYv2 instead of OS-specific APIs.
        makomk 9y ago
        If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.
CalChris 9y ago
The BLIT had a debugger which was named joff. Unix had printf.
chowyuncat 9y ago
Network debugging over VMWare to a 2.6 kernel (e.g. RHEL 6) will not work much of the time; a virtual serial port must be used instead.
lowleveldesign 9y ago
Debugging a syscall or start of the process was a great way for me to learn the system internals. I have some experience with Windows debugging and, after reading the article, I find that configuring the kernel debugging in Windows is quite easy. And I really like the live kernel debugging feature, when you either use windbg (that requires the debug boot flag) or simply run livekd [1] to analyze the running system data (for instance ALPC connections, handles, or loaded drivers data). Is there anything similar available in Linux? I plan to learn Linux internals and would love to use the kernel debugger next to reading the source code and books. Tangential, but if there is anyone interested in Windows debugging (including kernel debugging) have a look at the Inside Windows Debugging book by Tarik Soulami [2] [1] https://docs.microsoft.com/en-us/sysinternals/downloads/livekd https://docs.microsoft.com/en-us/sysinternals/downloads/live... [2] https://www.amazon.com/Inside-Windows-Debugging-Developer-Reference/dp/0735662789 https://www.amazon.com/Inside-Windows-Debugging-Developer-Re...
Timothycquinn 9y ago
Great to see this and I hope I never have to do this on Linux. A few years back, when I was finally getting my personal dev box off of windows, I took a very close look at FreeBSD or derivative. I hit the wall with poor touchpad support which made laptop difficult to use. I tried very hard to debug their touchpad kernel library but had no luck getting the results I needed. Anybody have links on how to do the same kind of remote kernel debugging on FreeBSD like above but using physical box rather than a VM? I would still like to give that can another good kick'n :)

117 points by alambert 9y ago

why do you need to machines? just one linux box: build kernel code, then update-initramfs and update grub.

Two machines seems like overkill in this case - why not debug using User Mode Linux[1]? Debugging with two machines makes perfect sense if you are debugging a hardware driver, but not here. [1] http://opensourceforu.com/2010/09/user-mode-linux-setup-and-debug/ http://opensourceforu.com/2010/09/user-mode-linux-setup-and-...

AstralStorm 9y ago
Because UML cannot interact with most hardware and that is where bugs happen. Even VMs do not provide real hardware or allow you to bring down the machine with it. Instead you can debug a crash using a trace and kexec dump. And also use quite fast ftrace infrastructure which is much better than plain old printf. There are also kprobes, gcov and oprofile.

bcantrill 9y ago

For those of us coming from non-Linux systems (and I'm speaking for illumos/SmartOS here), the effort required to debug the kernel displayed here is galling. That this is so difficult reflects Torvalds' historic disposition against kernel developers[1]: I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program. None of the arguments for a kernel debugger has touched me in the least. And trust me, over the years I've heard quite a lot of them. In the end, they tend to boil down to basically: - it would be so much easier to do development, and we'd be able to add new things faster. And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing. To be fair, this was a long time ago (17 years ago!), but the experience relayed here leaves one believing that Torvalds' historic attitude has cast a long shadow. And if it needs to be said, Torvalds' arguments are themselves deeply confused; he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation -- both of which have proved absolutely essential for kernel development. (Indeed, it is likely that DTrace alone would have allowed the author to debug their problem, as its design center is exactly the kind of non-fatal failure described.) The tutorial will certainly save others pain, but that such pain still exists at all in Linux is deeply unfortunate, and a vivid example of Linux not representing anything close to the state-of-the-art in systems development. [1] https://lwn.net/2000/0914/a/lt-debugger.php3 https://lwn.net/2000/0914/a/lt-debugger.php3

racer-v 9y ago
> he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. Where does he do this? The quote is only about single-step debugging. > So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.
- AstralStorm 9y ago
  Not to mention relocatable kernel and kexec crash dump analysis.
- bcantrill 9y ago
  No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.
  - richardwhiuk 9y ago
    It's a lot easier to configure using the modern API - the author is intentionally choosing an API which doesn't give much feedback....
    - xenophonf 9y ago
      That's true, but the author has solid reasons for doing so: > I want my application to work on OS X and Linux, so I’m targeting PF_KEYv2 instead of OS-specific APIs.
      - makomk 9y ago
        If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.

CalChris 9y ago

The BLIT had a debugger which was named joff. Unix had printf.

chowyuncat 9y ago

Network debugging over VMWare to a 2.6 kernel (e.g. RHEL 6) will not work much of the time; a virtual serial port must be used instead.

lowleveldesign 9y ago

Debugging a syscall or start of the process was a great way for me to learn the system internals. I have some experience with Windows debugging and, after reading the article, I find that configuring the kernel debugging in Windows is quite easy. And I really like the live kernel debugging feature, when you either use windbg (that requires the debug boot flag) or simply run livekd [1] to analyze the running system data (for instance ALPC connections, handles, or loaded drivers data). Is there anything similar available in Linux? I plan to learn Linux internals and would love to use the kernel debugger next to reading the source code and books. Tangential, but if there is anyone interested in Windows debugging (including kernel debugging) have a look at the Inside Windows Debugging book by Tarik Soulami [2] [1] https://docs.microsoft.com/en-us/sysinternals/downloads/livekd https://docs.microsoft.com/en-us/sysinternals/downloads/live... [2] https://www.amazon.com/Inside-Windows-Debugging-Developer-Reference/dp/0735662789 https://www.amazon.com/Inside-Windows-Debugging-Developer-Re...

Timothycquinn 9y ago

Great to see this and I hope I never have to do this on Linux. A few years back, when I was finally getting my personal dev box off of windows, I took a very close look at FreeBSD or derivative. I hit the wall with poor touchpad support which made laptop difficult to use. I tried very hard to debug their touchpad kernel library but had no luck getting the results I needed. Anybody have links on how to do the same kind of remote kernel debugging on FreeBSD like above but using physical box rather than a VM? I would still like to give that can another good kick'n :)