5 ms·

For those of us coming from non-Linux systems (and I'm speaking for illumos/SmartOS here), the effort required to debug the kernel displayed here is galling. T

For those of us coming from non-Linux systems (and I'm speaking for illumos/SmartOS here), the effort required to debug the kernel displayed here is galling. That this is so difficult reflects Torvalds' historic disposition against kernel developers[1]: I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program. None of the arguments for a kernel debugger has touched me in the least. And trust me, over the years I've heard quite a lot of them. In the end, they tend to boil down to basically: - it would be so much easier to do development, and we'd be able to add new things faster. And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing. To be fair, this was a long time ago (17 years ago!), but the experience relayed here leaves one believing that Torvalds' historic attitude has cast a long shadow. And if it needs to be said, Torvalds' arguments are themselves deeply confused; he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation -- both of which have proved absolutely essential for kernel development. (Indeed, it is likely that DTrace alone would have allowed the author to debug their problem, as its design center is exactly the kind of non-fatal failure described.) The tutorial will certainly save others pain, but that such pain still exists at all in Linux is deeply unfortunate, and a vivid example of Linux not representing anything close to the state-of-the-art in systems development. [1] https://lwn.net/2000/0914/a/lt-debugger.php3 https://lwn.net/2000/0914/a/lt-debugger.php3

racer-v 9y ago
> he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. Where does he do this? The quote is only about single-step debugging. > So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.
- AstralStorm 9y ago
  Not to mention relocatable kernel and kexec crash dump analysis.
- bcantrill 9y ago
  No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.
  - richardwhiuk 9y ago
    It's a lot easier to configure using the modern API - the author is intentionally choosing an API which doesn't give much feedback....
    - xenophonf 9y ago
      That's true, but the author has solid reasons for doing so: > I want my application to work on OS X and Linux, so I’m targeting PF_KEYv2 instead of OS-specific APIs.
      - makomk 9y ago
        If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.
        alambert 9y ago
        PF_KEYv2's difficult interface definitely isn't a Linux issue. But PF_KEYv2's limited error reporting means that understanding what's going wrong requires kernel debugging. The difficulty of kernel debugging is a Linux issue. ("I do not condone single-stepping through code to find the bug.") [1] https://news.ycombinator.com/item?id=15953644 https://news.ycombinator.com/item?id=15953644
Twirrim 9y ago
For what it's worth, Microsoft has the same approach to debugging the kernel via WinDBG.
- Avery3R 9y ago
  You can debug the kernel locally using windbg, it's pretty easy to freeze your system doing that though.
deleted 9y ago
[deleted]
rhinoceraptor 9y ago
This is Linux after all, why would you need postmortem debugging if the system never kills itself when it finds itself in an invalid state? :-)
richardwhiuk 9y ago
How do you suggest a kernel debugger should work?
alambert 9y ago
I agree! For Windows, getting symbols was trivial: you just pointed your debugger to the publicly-available symbol server[1]. You can see this attitude continuing in [2]: a beginner is warned away from using a debugger to understand the kernel behavior. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ee416588(v=vs.85).aspx#using_the_microsoft_symbol_server https://msdn.microsoft.com/en-us/library/windows/desktop/ee4... [2] https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016-August/016692.html https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016...
- bcantrill 9y ago
  Wow, that second example is very telling. For whatever it's worth, before we started work on DTrace (ca. 2001), we prioritized a project to add not just symbol information but also debugging information to production kernels. This project -- the Compact C Type Format (CTF)[1][2] -- has proved essential many times over in the years since (and is a major reason why we can consider kernel debugging a part of the core system functionality). It is clear that a project like CTF is unlikely to even be understood let alone prioritized by those displaying such a dismissive attitude towards debugging. [1] http://illumos.org/man/4/ctf http://illumos.org/man/4/ctf [2] http://www.smnd.sk/lovasko/paper.pdf http://www.smnd.sk/lovasko/paper.pdf
  - harry8 9y ago
    Brian, I love your work but seriously. You've mispelled "read" as "understood" there and to write it off as a dismissive attitude toward debugging in toto is not something you should lower yourself to pretend to believe. Deep breath. DTrace is awesome, Oracle is not. Yeah my kissing may not be up to scratch either... ;-)
    - bcantrill 9y ago
      The attitude towards debugging is dismissive -- and I'm not the only one to have drawn that conclusion. Indeed, it's the original author of this article who came across that and drew that inference -- one that I (obviously) share. To flip this around: do you think that the work outlined in the original article is work that should be expected of anyone wishing to debug the kernel?
ploxiln 9y ago
That's a great Torvalds emails which I haven't seen before! Here's the "point" though: > I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_. I think that's really pretty reasonable. Surely you've seen a programmer who ran into a bug and submitted a pull request saying "I fixed it" but it's clearly just a messy work-around which misunderstands the problem and leaves a bunch of analogous problem cases unfixed. I prefer printf-and-inspection debugging myself, I only reach for gdb once a week or so. I'm pretty diligent and my results are quite solid. It's fine that we have different styles, and because of well-designed interfaces, like process boundaries, we can co-exist in peace. Linus is really counter-cultural in today's hip software world, where everyone says everything should be easier and easier. We just can't ever make threads or async or crypto easy enough. Meanwhile the popular applications and frameworks in use seem to get bigger and slower and buggier. Correlation, not causation, but still an annoying trend.

racer-v 9y ago

> he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large. Where does he do this? The quote is only about single-step debugging. > So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.

AstralStorm 9y ago
Not to mention relocatable kernel and kexec crash dump analysis.
bcantrill 9y ago
No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.
- richardwhiuk 9y ago
  It's a lot easier to configure using the modern API - the author is intentionally choosing an API which doesn't give much feedback....
  - xenophonf 9y ago
    That's true, but the author has solid reasons for doing so: > I want my application to work on OS X and Linux, so I’m targeting PF_KEYv2 instead of OS-specific APIs.
    - makomk 9y ago
      If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.
      - alambert 9y ago
        PF_KEYv2's difficult interface definitely isn't a Linux issue. But PF_KEYv2's limited error reporting means that understanding what's going wrong requires kernel debugging. The difficulty of kernel debugging is a Linux issue. ("I do not condone single-stepping through code to find the bug.") [1] https://news.ycombinator.com/item?id=15953644 https://news.ycombinator.com/item?id=15953644

Twirrim 9y ago

For what it's worth, Microsoft has the same approach to debugging the kernel via WinDBG.

Avery3R 9y ago
You can debug the kernel locally using windbg, it's pretty easy to freeze your system doing that though.

deleted 9y ago

[deleted]

rhinoceraptor 9y ago

This is Linux after all, why would you need postmortem debugging if the system never kills itself when it finds itself in an invalid state? :-)

richardwhiuk 9y ago

How do you suggest a kernel debugger should work?

alambert 9y ago

I agree! For Windows, getting symbols was trivial: you just pointed your debugger to the publicly-available symbol server[1]. You can see this attitude continuing in [2]: a beginner is warned away from using a debugger to understand the kernel behavior. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ee416588(v=vs.85).aspx#using_the_microsoft_symbol_server https://msdn.microsoft.com/en-us/library/windows/desktop/ee4... [2] https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016-August/016692.html https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016...

bcantrill 9y ago
Wow, that second example is very telling. For whatever it's worth, before we started work on DTrace (ca. 2001), we prioritized a project to add not just symbol information but also debugging information to production kernels. This project -- the Compact C Type Format (CTF)[1][2] -- has proved essential many times over in the years since (and is a major reason why we can consider kernel debugging a part of the core system functionality). It is clear that a project like CTF is unlikely to even be understood let alone prioritized by those displaying such a dismissive attitude towards debugging. [1] http://illumos.org/man/4/ctf http://illumos.org/man/4/ctf [2] http://www.smnd.sk/lovasko/paper.pdf http://www.smnd.sk/lovasko/paper.pdf
- harry8 9y ago
  Brian, I love your work but seriously. You've mispelled "read" as "understood" there and to write it off as a dismissive attitude toward debugging in toto is not something you should lower yourself to pretend to believe. Deep breath. DTrace is awesome, Oracle is not. Yeah my kissing may not be up to scratch either... ;-)
  - bcantrill 9y ago
    The attitude towards debugging is dismissive -- and I'm not the only one to have drawn that conclusion. Indeed, it's the original author of this article who came across that and drew that inference -- one that I (obviously) share. To flip this around: do you think that the work outlined in the original article is work that should be expected of anyone wishing to debug the kernel?

ploxiln 9y ago

That's a great Torvalds emails which I haven't seen before! Here's the "point" though: > I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_. I think that's really pretty reasonable. Surely you've seen a programmer who ran into a bug and submitted a pull request saying "I fixed it" but it's clearly just a messy work-around which misunderstands the problem and leaves a bunch of analogous problem cases unfixed. I prefer printf-and-inspection debugging myself, I only reach for gdb once a week or so. I'm pretty diligent and my results are quite solid. It's fine that we have different styles, and because of well-designed interfaces, like process boundaries, we can co-exist in peace. Linus is really counter-cultural in today's hip software world, where everyone says everything should be easier and easier. We just can't ever make threads or async or crypto easy enough. Meanwhile the popular applications and frameworks in use seem to get bigger and slower and buggier. Correlation, not causation, but still an annoying trend.