We had a function that looked like void f() { bool run = true; while (run) { g(); } } This function was exiting, and not because s

We had a function that looked like void f() { bool run = true; while (run) { g(); } } This function was exiting, and not because something threw. So the loop was terminating. And when it did, run was false. The obvious answer is that g() was smashing the stack. But I ruled that out, because g() was returning to f(), and if g() was smashing the stack, I would expect it to destroy the return address before it destroyed a variable in f(). I tried to solve it for a month, off and on. Every time I tried to get more information, the problem disappeared. Finally I got desperate enough to look at the assembly output of the compiler, and light dawned. (This was g++ on an ARM, by the way.) run, the bool in f(), had no address. It lived in register R12. When f() called g(), it pushed the return address. In the implementation of g(), the first thing it did is push R12 so that it could place its own variable in the scratchpad register. So f()'s local variable wound up in g()'s stack frame... And g() was smashing the stack. Duh. In particular, it called msgrcv(int msqid, const void msgp, size_t msgsz, int msgflg), which has a highly misleading API. (For those not in the know, this is a POSIX message queue implementation.) It expects msgp to point to a structure like struct msgbuf { long mtype; /* message type, must be > 0 */ char mtext[1]; /* message data */ }; and msgsize is the size of msgbuf.mtext array, not* the size of msgbuf. The contractors who wrote this code used the size of msgbuf, which is 4 bytes too high, so they were writing four bytes too many, which happened to overwrite the pushed value of R12, which was f()'s run variable. (The queue did not get out of sync, because they made the same mistake on the other side, and wrote four bytes too many as well.) One more twist: The message queue was actually wrapping communication with another CPU. So whenever an unrelated four bytes on a different CPU were zero, then the loop would exit and f() would terminate.