4 ms·

To check my understanding: The bug looks a bit like these two threads running in parallel: A: x := N A: y := N A: done := true B: if(done) { B:

To check my understanding: The bug looks a bit like these two threads running in parallel: A: x := N A: y := N A: done := true B: if(done) { B: assert(x == y) B: } else { retry } Where the assert in thread B replaces some other code which had x == y as a precondition. The problem was thought to be that, due to weak memory ordering on the processor, B would sometimes observe done to be true but x != y. The fix would be to insert appropriate memory barriers to get the threads to see consistent data. The problem is to try to determine whether the hypothesis was correct. The bug is rare and the proposed fix might make it more rare. It would be bad if the fix just made another bug harder to discover. The idea was then that to test this, you hit the edge case and test if the memory fence is sufficient to cause the assertion to succeed. But instead of rewriting the code to something like the following, you can just rely on invoking the debugger being sufficiently invasive to have the effect of the fence and then rerun the assertion. B: if(done) { B: if(x != y) { B: fence(); B: assert(x == y); B: printf("Gotcha!\n"); B: } B: // Now use x, y as before… B: } else { retry } One issue is that while this finds evidence that missing memory barriers were sometimes the cause, protecting from the case where the cpu is just having a good day while you’re testing and the bug-triggering case doesn’t happen, it doesn’t really prove that the only cause of x != y issue (this example is presumably distilled from a single potential cause of something more complicated) is caused by the memory barrier problem (eg maybe a rare buffer overflow sets done to true too early). Is that a fair summary or did I miss something?

anyfoo 4y ago
I think you understood it. To your last point: > it doesn’t really prove that the only cause of x != y issue [...] is caused by the memory barrier problem (eg maybe a rare buffer overflow sets done to true too early). I mean, you can never prove that, and that goes for pretty much any bug. Hypothetically, a hardware issue could be in the mix. Or a cosmic ray may hit the device (that's a real thing). You can't prove a negative after all. But while this does not prove that there's no other cause with the same effect, this at least makes it extremely likely that the missing barrier was an issue: Breaking with the debugger suddenly makes it work without any other intervention, and also they've just looked at the machine code in the first place, with the knowledge of how CPUs work. The likelihood that there's also a rare buffer overflow consistently reproducing the same issue is negligibly small. If there is, that's a problem you can look at once it actually happens again. And then stare at it in bewildered disbelief.

anyfoo 4y ago

I think you understood it. To your last point: > it doesn’t really prove that the only cause of x != y issue [...] is caused by the memory barrier problem (eg maybe a rare buffer overflow sets done to true too early). I mean, you can never prove that, and that goes for pretty much any bug. Hypothetically, a hardware issue could be in the mix. Or a cosmic ray may hit the device (that's a real thing). You can't prove a negative after all. But while this does not prove that there's no other cause with the same effect, this at least makes it extremely likely that the missing barrier was an issue: Breaking with the debugger suddenly makes it work without any other intervention, and also they've just looked at the machine code in the first place, with the knowledge of how CPUs work. The likelihood that there's also a rare buffer overflow consistently reproducing the same issue is negligibly small. If there is, that's a problem you can look at once it actually happens again. And then stare at it in bewildered disbelief.