3 ms·

One tactic is actually to keep the log buffer in RAM and then you can dump it from the core file. You can often have a RAM only buffer which has lower level log

One tactic is actually to keep the log buffer in RAM and then you can dump it from the core file. You can often have a RAM only buffer which has lower level logs than that which you can log to disk.

tonyarkles 8y ago
I absolutely love this approach and have used it successfully in the past, but there's a caveat: I have only encountered a handful of people in my career who know or care what a core file is, and even fewer who know how extract data out of one. If you're building something that's going to primarily have strong systems people working on it, it's a great approach, especially for ephemeral state logs and debugging. If the "general public" is going to be trying to extract logs out of it, it's probably not very ergonomic for them. Edit: taking it a level deeper, this is used in the Linux kernel during the early stages of booting. When I was doing bring-up on a board, I wasn't getting any console output when trying to boot the kernel. What I could do is let it sit for a bit, reset, and dump the kmesg buffer from u-boot to see what the kernel had been doing. Absolutely beautiful backdoor when trying to debug broken serial output.
- anitil 8y ago
  There's an article called something like "The logging system you can't have" which talks about using a similar thing. In the case of a panic, the device would restart and find the buffer in memory, allowing you to recover crash data across boots. From memory they then pointed all logs as /dev/kmsg rather than /var/log/syslog or equivalent logger, so that all userspace logs were buffered immediately into memory. I think that the system logger then read from the buffer? Wish I could find the article.
- richardwhiuk 8y ago
  It's fairly easy to write a tool to collect the core file once it's dumped, and also fairly easy to write a tool which wraps gdb and prints out the buffer.

One tactic is actually to keep the log buffer in RAM and then you can dump it from the core file. You can often have a RAM only buffer which has lower level logs than that which you can log to disk.

tonyarkles 8y ago

I absolutely love this approach and have used it successfully in the past, but there's a caveat: I have only encountered a handful of people in my career who know or care what a core file is, and even fewer who know how extract data out of one. If you're building something that's going to primarily have strong systems people working on it, it's a great approach, especially for ephemeral state logs and debugging. If the "general public" is going to be trying to extract logs out of it, it's probably not very ergonomic for them. Edit: taking it a level deeper, this is used in the Linux kernel during the early stages of booting. When I was doing bring-up on a board, I wasn't getting any console output when trying to boot the kernel. What I could do is let it sit for a bit, reset, and dump the kmesg buffer from u-boot to see what the kernel had been doing. Absolutely beautiful backdoor when trying to debug broken serial output.

anitil 8y ago
There's an article called something like "The logging system you can't have" which talks about using a similar thing. In the case of a panic, the device would restart and find the buffer in memory, allowing you to recover crash data across boots. From memory they then pointed all logs as /dev/kmsg rather than /var/log/syslog or equivalent logger, so that all userspace logs were buffered immediately into memory. I think that the system logger then read from the buffer? Wish I could find the article.
richardwhiuk 8y ago
It's fairly easy to write a tool to collect the core file once it's dumped, and also fairly easy to write a tool which wraps gdb and prints out the buffer.