4 ms·

Computer science-y question. Initially this question was going to be "can we log executed instructions" but I rapidly realized that not even DDR5 could keep up

• dom0 9y ago
  > Initially this question was going to be "can we log executed instructions" but I rapidly realized that not even DDR5 could keep up with such a logging system - it would slow things down too much and likely mask the bug (not to mention the TBs of space that would be needed). Yes, the instruction fetching is quite a big part of all memory/cache reads :)
• todd8 9y ago
  From my experience, tracing the execution paths is possible, but it isn't really logging every instruction. To isolate the fault, one can start use a kind of binary search on the program containing the fault. By putting in one very light weight tracing instruction that records being executed one can look for the fault happening before the tracing instruction is executed or if the fault happens after the tracing instruction. The tracing instruction can be moved to the half of the code where the fault happened. By repeatedly dividing the code into smaller regions one can eventually narrow the location down to a small sequence of instructions that might contain the problem. Of course, doing this for a problem like this involves overcoming a number of large obstacles. First, the fault the OP is talking about appears to be somewhat unpredictable. This means that we will have to keep records of the execution of the tracing instruction and need to have multiple tracing instructions in the code to see where the processor was really executing instructions when the fault happens. A good understanding of the code's organization as basic blocks (roughly sequences of machine instructions without branches) and some way of analyzing in a programmatic way the location where the fault must have occurred by looking at the counts of times that all of the different tracing instructions were executed will help in narrowing down the region where the fault happens. Compilers, like GCC, can be used to systematically instrument the code. How can tracing instructions be of such low impact that they don't interfere with the fault being searched for? There is no guarantee that the attempt to measure or detect faults won't hide them, but light weight tracing can be done with pretty simple tracing hardware (simple that is for companies that make computers, like IBM or HP). Basically a device/card is plugged into one of the addressable buses (memory bus or I/O bus like PCI). The tracing hardware simple looks for bus addresses in a range of addresses that are reserved for tracing, say 8k of addresses allowing up to 8k tracing instructions to be scattered around the code. The tracing hardware can then record in its own separate memory the last few million of these addresses that appear on the bus in this unused range of address. The tracing instructions inserted into the program under test (in this case, say bash) will depend on the hardware, in this case, amd64. I'm not familiar enough with all of the new instructions available on the latest processors, but an instruction like set-memory-to-zero would work. The instruction doesn't really matter to the tracing hardware, it ignores the instruction it just looks for an address in the special range on the bus. Even this fancy tracing hardware is too slow to use in the middle of loops running in registers or the cache, but by tracing the entrance and exit from such sequences the hardware/software causing the problems can isolated. The same techniques are used to do debugging and performance tuning of operating systems, special hardware traces the operation of say the disk scheduler and a careful study of the relationship between the code responsible for scheduling operations on the disk drive and what shows up in the tracing is used to reduce the inefficiencies or problems in the low level drivers.
  • i336_ 9y ago
    Thanks so much for this answer, I just learned a lot. > There is no guarantee that the attempt to measure or detect faults won't hide them... I got completely stuck on this in my original ponderings. I totally didn't think of sprinkling instrumentation instructions into the code and seeing if the bug still fired. If it did, the approach you described would certainly work very well (and, indeed, you describe it being widely used). Major TIL with the address-based tracing hardware idea. That's an awesome approach, to do it that way... wow. :) Building something like this would actually be a really cool challenge in designing a really fast piece of hardware. Considering the kind of access speed needed, though (particularly with the memory bus approach)... would a custom ASIC be required? :/ Or could I get away with using a (perhaps decent/pricey) FPGA? I say this because it would be awesome to make something like this inexpensively available for people to put in their workstations. I can totally see a device like this also having some fast, nonvolatile* memory-mapped storage for things like infinite logging, as well. For example, the way Linux handles crashes is to kexec into a new kernel that hopefully fishes the log out of RAM and saves it. Very clunky. This approach also does not handle early kernel bringup - or even BIOS/EFI bringup, the libreboot folks would probably love something like this. (*By "fast, nonvolatile" I mean something that writes straight to a large DDR-backed buffer and is then quickly yanked onto something like an NVMe disk.)
• posterboy 9y ago
  Since programms are non-deterministic, knowing the initial parameters should be enough, theoretically, but I am not sure if the internal translation of microcode is still deterministic. Considering the initial conditions for the kernel and every other part of the system would have to be reset in hardware and rebooted for every run until the bug is triggered, I'm not sure how feasible this approach would be. Edit: My comment doesn't even get to your question. I mean, bug reproduction is quite hard before recording.