6 ms·

Linux-Kernel: Printk() Indexing

80 points by signa11 5y ago

freehrtradical 5y ago
The article mentions debugfs which I've never heard about and is pretty cool: https://www.kernel.org/doc/html/latest/filesystems/debugfs.html https://www.kernel.org/doc/html/latest/filesystems/debugfs.h...
- inshadows 5y ago
  Debugfs is just a dumping ground for various debug functions. For instance ftrace subsystem is there. Of course, it's 700 root by default, because you can find a lot there.
CalChris 5y ago
I mean this semi-seriously but doesn't the hypothetical example "CPU on fire" vs "CPU in flames" violate We Do Not Break User Space? As someone who got a useful (IMHO) one-line patch with less change dinged for WDNBUS, I would say yes.
- dcminter 5y ago
  According to the article "those messages have never really been considered to be a part of the kernel's ABI"
  - mjw1007 5y ago
    Right, but the We Do Not Break User Space rule has often been explained (somewhat emphatically on occasion) as depending on whether anyone's code actually gets broken, not on whether the thing they were relying on was intended to be, or advertised as, a stable interface.
    - fctorial 5y ago
      I think WDNBUS is only respected for the syscall api.
      - dcminter 5y ago
        What about /proc ?
        touisteur 5y ago
        /proc has surely killed some user apps. If you were expected to see a fixed number of lines, a fixed number of columns or the fields to mean the same thing, you're in for suffering. At least I know of meminfo changes, stuff in /proc/net... But to be honest, so have ifconfig, hostname, df... At least in procfs they add things (columns, lines) so apart from meminfo I don't know of changing the meaning of fields.
      - mjw1007 5y ago
        Here's a relatively mild rant where the breaking change in question was in /proc/cpuinfo: «Really. Shut up. The whole "no regressions" thing is very much about the fact that we don't waste users time.» https://linux.kernel.narkive.com/7hMPh8zX/patch-revert-9fc2105aeaaf56b0cf75296a84702d0f9e64437b-to-fix-pyaudio-and-probably-more#post12 https://linux.kernel.narkive.com/7hMPh8zX/patch-revert-9fc21... Of course in an ideal world the extent of the intended promise would be clearly written down in some obvious place. But we don't live in an ideal world.
        dcminter 5y ago
        I found this statement from Linus in that thread; "It's not that the kernel ABI (with all the system calls, all the /proc files, all the ioctl's, etc) is set in stone and "sacred". Absolutely anything can be changed, wildly. But if it turns out that applications (or hardware) that people use end up breaking noticeably, then that is a regression."
  - tinus_hn 5y ago
    Linus has been pretty strict in not unnecessarily breaking userspace even if something is not strictly ‘part of the ABI’. It’s a judgement call though.
scrps 5y ago
>The 2011 Kernel Summit featured a combative session around a proposal to add a 128-bit binary tag to every kernel message. That idea, like all others before and after, failed to get anywhere near the mainline. The kernel community has, quite simply, never felt the need to impose any structure on the logging information it puts out via printk(); the additional effort that would be required does not seem to be worth the cost. It seems like that was the more elegant solution vs enabling debugfs, running through the list of error messages to ensure nothing changed. How many distros even ship with debugfs enabled by default? Edit: formatting
- nexuist 5y ago
  I think the problem then becomes what message gets what tag and you have to bikeshed over if smaller or larger numbers mean bad/urgent messages. It's like the same problem of port allocation that IANA was created to solve. Do we really expect every kernel module to adhere to e.g. registering a unique tag with some authority for their messages? What if two separate modules print the same tag? And so and so on, and all this time of arguing over what is the "Linuxy" way to match numbers with format strings would be better used reviewing commits that could actually improve the kernel. I think the better morale of the story is to not rely on third party log output to troubleshoot problems. If you want to be notified when the CPU catches fire then you should have a daemon watching CPU temp waiting to trigger the event, not relying on some black box to log the correct line at the right time and some other black box to match the line against some ancient regex that nobody understands anymore. At the very least your temperature monitoring daemon should `printk` a message you decided upon instead of someone else.
- elurg 5y ago
  > How many distros even ship with debugfs enabled by default? Almost all of them, try `mount | grep ^debugs` or at least `/proc/filesystems |grep debugfs`. It's root-only though.
eqvinox 5y ago
If anyone wants to steal a done-tested-and-working userspace variant (for C, and with limitations C++) of this: https://github.com/FRRouting/frr/pull/6766 https://github.com/FRRouting/frr/pull/6766 It's pretty much the same thing, all occurences of "zlog_debug" and the like create an entry in a separate ELF section. The section can be accessed both at runtime by the program itself as well as offline by reading the appropriate ELF headers (there's an extraction tool in the codebase that does that, it even works cross-OS cross-CPU cross-endian). disclaimer: it's my code ;) disclaimer[2]: this will not work without major overhaul on OSX or Windows. It relies on the binary format being ELF. disclaimer[3]: the PR linked above is only the "core" of it, you may have to fish out other pieces elsewhere.

80 points by signa11 5y ago

The article mentions debugfs which I've never heard about and is pretty cool: https://www.kernel.org/doc/html/latest/filesystems/debugfs.html https://www.kernel.org/doc/html/latest/filesystems/debugfs.h...

inshadows 5y ago
Debugfs is just a dumping ground for various debug functions. For instance ftrace subsystem is there. Of course, it's 700 root by default, because you can find a lot there.

CalChris 5y ago

I mean this semi-seriously but doesn't the hypothetical example "CPU on fire" vs "CPU in flames" violate We Do Not Break User Space? As someone who got a useful (IMHO) one-line patch with less change dinged for WDNBUS, I would say yes.

dcminter 5y ago
According to the article "those messages have never really been considered to be a part of the kernel's ABI"
- mjw1007 5y ago
  Right, but the We Do Not Break User Space rule has often been explained (somewhat emphatically on occasion) as depending on whether anyone's code actually gets broken, not on whether the thing they were relying on was intended to be, or advertised as, a stable interface.
  - fctorial 5y ago
    I think WDNBUS is only respected for the syscall api.
    - dcminter 5y ago
      What about /proc ?
      - touisteur 5y ago
        /proc has surely killed some user apps. If you were expected to see a fixed number of lines, a fixed number of columns or the fields to mean the same thing, you're in for suffering. At least I know of meminfo changes, stuff in /proc/net... But to be honest, so have ifconfig, hostname, df... At least in procfs they add things (columns, lines) so apart from meminfo I don't know of changing the meaning of fields.
    - mjw1007 5y ago
      Here's a relatively mild rant where the breaking change in question was in /proc/cpuinfo: «Really. Shut up. The whole "no regressions" thing is very much about the fact that we don't waste users time.» https://linux.kernel.narkive.com/7hMPh8zX/patch-revert-9fc2105aeaaf56b0cf75296a84702d0f9e64437b-to-fix-pyaudio-and-probably-more#post12 https://linux.kernel.narkive.com/7hMPh8zX/patch-revert-9fc21... Of course in an ideal world the extent of the intended promise would be clearly written down in some obvious place. But we don't live in an ideal world.
      - dcminter 5y ago
        I found this statement from Linus in that thread; "It's not that the kernel ABI (with all the system calls, all the /proc files, all the ioctl's, etc) is set in stone and "sacred". Absolutely anything can be changed, wildly. But if it turns out that applications (or hardware) that people use end up breaking noticeably, then that is a regression."
- tinus_hn 5y ago
  Linus has been pretty strict in not unnecessarily breaking userspace even if something is not strictly ‘part of the ABI’. It’s a judgement call though.

scrps 5y ago

>The 2011 Kernel Summit featured a combative session around a proposal to add a 128-bit binary tag to every kernel message. That idea, like all others before and after, failed to get anywhere near the mainline. The kernel community has, quite simply, never felt the need to impose any structure on the logging information it puts out via printk(); the additional effort that would be required does not seem to be worth the cost. It seems like that was the more elegant solution vs enabling debugfs, running through the list of error messages to ensure nothing changed. How many distros even ship with debugfs enabled by default? Edit: formatting

nexuist 5y ago
I think the problem then becomes what message gets what tag and you have to bikeshed over if smaller or larger numbers mean bad/urgent messages. It's like the same problem of port allocation that IANA was created to solve. Do we really expect every kernel module to adhere to e.g. registering a unique tag with some authority for their messages? What if two separate modules print the same tag? And so and so on, and all this time of arguing over what is the "Linuxy" way to match numbers with format strings would be better used reviewing commits that could actually improve the kernel. I think the better morale of the story is to not rely on third party log output to troubleshoot problems. If you want to be notified when the CPU catches fire then you should have a daemon watching CPU temp waiting to trigger the event, not relying on some black box to log the correct line at the right time and some other black box to match the line against some ancient regex that nobody understands anymore. At the very least your temperature monitoring daemon should `printk` a message you decided upon instead of someone else.
elurg 5y ago
> How many distros even ship with debugfs enabled by default? Almost all of them, try `mount | grep ^debugs` or at least `/proc/filesystems |grep debugfs`. It's root-only though.

eqvinox 5y ago

If anyone wants to steal a done-tested-and-working userspace variant (for C, and with limitations C++) of this: https://github.com/FRRouting/frr/pull/6766 https://github.com/FRRouting/frr/pull/6766 It's pretty much the same thing, all occurences of "zlog_debug" and the like create an entry in a separate ELF section. The section can be accessed both at runtime by the program itself as well as offline by reading the appropriate ELF headers (there's an extraction tool in the codebase that does that, it even works cross-OS cross-CPU cross-endian). disclaimer: it's my code ;) disclaimer[2]: this will not work without major overhaul on OSX or Windows. It relies on the binary format being ELF. disclaimer[3]: the PR linked above is only the "core" of it, you may have to fish out other pieces elsewhere.