4 ms·

Pernosco currently targets C, C++, Ada, Rust and V8 JS (e.g. node.js) applications that run on x86 Linux and work with rr. Cortex-M developer here. Let me kno

Pernosco currently targets C, C++, Ada, Rust and V8 JS (e.g. node.js) applications that run on x86 Linux and work with rr. Cortex-M developer here. Let me know when you support bare metal.

db48x 6y ago
That’s an interesting point. The way RR works is that it intercepts syscalls, records the execution point of the program that generated the syscall, records the arguments to the syscall, and the kernel’s response. During playback, it intercepts those syscalls, but instead of sending them to the kernel it just supplies the recorded response instead. It also records several hardware performance counters that were added to Intel CPUs over a decade ago to ensure that it can stop the playback of the program at the correct points reliably. It could also do this purely by single–stepping the program, but that would be horribly slow. For an embedded system, or for debugging the Linux kernel itself, this doesn’t work. The kernel doesn’t make syscalls, and your embedded system might not even have a kernel to make syscalls to. And your hardware probably doesn’t have the right performance counters; AMD cpus only got equivalent capabilities quite recently. On the other hand, perhaps you could capture and record interrupts instead of syscalls. Maybe the performance counters on the Cortex–M cpu could be made to work for RR’s purposes (I’m just assuming it has some performance counters; I haven’t checked). But that would take a lot of work, and most people just want to be notified when someone else has completed the work.
- rcxdude 6y ago
  Cortex processors have a tracing functionality which allows capturing the entire instruction and data stream of the processor, allowing rr-style debugging. The main problem is this is almost always a very high-bandwidth stream which required dedicated pins on the microcontroller and a specialised (and usually expensive) adapter to collect. I've never actually seen it used, I don't know how many embedded developers actually use it (the only place I've heard tell of it being used is inside ARM). It's possible there's some configuration which only records the minimal trace needed to reconstruct the trace later in a way that is accessible to the software running on the chip (allowing extracting the data without needing a debugger attached), but I've not seen it, and the details of this functionality seem to vary substantially across different devices.
  - joezydeco 6y ago
    The go-to companies for these devices are Segger and Lauterbach. I'm a Segger user for JTAG debugging, although it's rare that the shops I work in will have a J-TRACE device on the shelf. J-TRACE is the thing you buy when it's 5 minutes to production and your hair is on fire and you can't figure out what the problem is. Which is why Segger prices it at $2,000.
  - db48x 6y ago
    Yea, that’s how program recording is done natively by GDB as well; it inserts a breakpoint after every single instruction, and records the the state of the program after every one. It’s mind–bogglingly slow, and the log files are gigantic. You generally have to limit it to record only a few instructions, generally tens of thousands rather than billions, in order to get any use out of it. At least if you can do it in hardware then it doesn’t have to be slow, even if the recorded state is still huge.

Pernosco currently targets C, C++, Ada, Rust and V8 JS (e.g. node.js) applications that run on x86 Linux and work with rr. Cortex-M developer here. Let me know when you support bare metal.

db48x 6y ago

That’s an interesting point. The way RR works is that it intercepts syscalls, records the execution point of the program that generated the syscall, records the arguments to the syscall, and the kernel’s response. During playback, it intercepts those syscalls, but instead of sending them to the kernel it just supplies the recorded response instead. It also records several hardware performance counters that were added to Intel CPUs over a decade ago to ensure that it can stop the playback of the program at the correct points reliably. It could also do this purely by single–stepping the program, but that would be horribly slow. For an embedded system, or for debugging the Linux kernel itself, this doesn’t work. The kernel doesn’t make syscalls, and your embedded system might not even have a kernel to make syscalls to. And your hardware probably doesn’t have the right performance counters; AMD cpus only got equivalent capabilities quite recently. On the other hand, perhaps you could capture and record interrupts instead of syscalls. Maybe the performance counters on the Cortex–M cpu could be made to work for RR’s purposes (I’m just assuming it has some performance counters; I haven’t checked). But that would take a lot of work, and most people just want to be notified when someone else has completed the work.

rcxdude 6y ago
Cortex processors have a tracing functionality which allows capturing the entire instruction and data stream of the processor, allowing rr-style debugging. The main problem is this is almost always a very high-bandwidth stream which required dedicated pins on the microcontroller and a specialised (and usually expensive) adapter to collect. I've never actually seen it used, I don't know how many embedded developers actually use it (the only place I've heard tell of it being used is inside ARM). It's possible there's some configuration which only records the minimal trace needed to reconstruct the trace later in a way that is accessible to the software running on the chip (allowing extracting the data without needing a debugger attached), but I've not seen it, and the details of this functionality seem to vary substantially across different devices.
- joezydeco 6y ago
  The go-to companies for these devices are Segger and Lauterbach. I'm a Segger user for JTAG debugging, although it's rare that the shops I work in will have a J-TRACE device on the shelf. J-TRACE is the thing you buy when it's 5 minutes to production and your hair is on fire and you can't figure out what the problem is. Which is why Segger prices it at $2,000.
- db48x 6y ago
  Yea, that’s how program recording is done natively by GDB as well; it inserts a breakpoint after every single instruction, and records the the state of the program after every one. It’s mind–bogglingly slow, and the log files are gigantic. You generally have to limit it to record only a few instructions, generally tens of thousands rather than billions, in order to get any use out of it. At least if you can do it in hardware then it doesn’t have to be slow, even if the recorded state is still huge.