4 ms·

I had one of these back in the 90s that turned out to be a compiler bug. It was code that ran a mobile robot with an arm. Exact same code running on a Sun wor

by lisper 4y ago

I had one of these back in the 90s that turned out to be a compiler bug. It was code that ran a mobile robot with an arm. Exact same code running on a Sun workstation never failed, but running on an embedded system running vxWorks crashed intermittently, but only when the arm was moving. Entire heap was corrupted, so by the time the crash occurred there was no hope of getting a stack trace or any hint of what went wrong upstream. Turned out to be two mis-ordered instructions that accessed a value on the stack after the stack pointer had been popped. On vxWorks, interrupts used the same stack as the currently running process, so if an interrupt occurred exactly between these two instructions it would clobber that value, and chaos ensued. Took a full year to figure it out. Good times.

aw1621107 4y ago
How did you end up piecing together what happened?
- lisper 4y ago
  Long story but the tldr is that it happened in two stages. First someone figured out a way to reliably reproduce the problem. And then I spent a very long time single stepping through machine instructions until I had a eureka moment.
  - dekhn 4y ago
    And the compiler was emitting the two instructions in the wrong order?
    - lisper 4y ago
      Yes.
      - dylan604 4y ago
        I hope someone bought you a beer
        lisper 4y ago
        Discovery is its own reward ;-) Actually, I remember reporting the bug to the compiler authors and being stunned when they told me that they were not going to issue a new version with the bug fix because the project was no longer being funded. (This was the T dialect of Lisp in case you're wondering.)
  - aw1621107 4y ago
    Sounds like quite the arduous process! What was done in the time period between discovering the bug and its cause?
    - lisper 4y ago
      A lot of rebooting and cursing. Fortunately, it only happened when the robot's arm was moving, and we were mostly doing mobility research so we were able to be productive simply by not using the arm.

by lisper 4y ago

aw1621107 4y ago

How did you end up piecing together what happened?

lisper 4y ago
Long story but the tldr is that it happened in two stages. First someone figured out a way to reliably reproduce the problem. And then I spent a very long time single stepping through machine instructions until I had a eureka moment.
- dekhn 4y ago
  And the compiler was emitting the two instructions in the wrong order?
  - lisper 4y ago
    Yes.
    - dylan604 4y ago
      I hope someone bought you a beer
      - lisper 4y ago
        Discovery is its own reward ;-) Actually, I remember reporting the bug to the compiler authors and being stunned when they told me that they were not going to issue a new version with the bug fix because the project was no longer being funded. (This was the T dialect of Lisp in case you're wondering.)
- aw1621107 4y ago
  Sounds like quite the arduous process! What was done in the time period between discovering the bug and its cause?
  - lisper 4y ago
    A lot of rebooting and cursing. Fortunately, it only happened when the robot's arm was moving, and we were mostly doing mobility research so we were able to be productive simply by not using the arm.