3 ms·

> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very

> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. That sounds like an interesting detective story. Did they write it up anywhere?

Negitivefrags 4y ago
They didn’t, but it was one hell of a pain in the ass. We only ever saw it on the production databases and the crash, which was the main thing we were debugging, would only happen every week or so. Core dumps didn’t show anything useful because the crashes were just that a pointer was mysteriously pointing into a bad place. But how did it get corrupted? We were doing increasingly insane things to reproduce the problem locally. An interesting idea I had to reproduce it was using a database replica, which did crash sometimes. We used TCPDump to capture the replication stream, then waited a week for the replica to crash. We could start up a copy of the replica in the state it was at the beginning and replay the week of replication data into the socket to reproduce the problem. My theory was that since a DB replica doesn’t do anything other than the sequence of operations that come to it via the replication stream, that this should deterministicly reproduce the problem. Turns out that it didn’t. Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though.
- guerrilla 4y ago
  > Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though. Ahhh I could see how that could be! Fascinating. That's exactly the kind of heisenbug I used to love. I always liked the ones that seemed literally impossible but had a nice explanation.
  - iquerno 4y ago
    schrödinbug*

Negitivefrags 4y ago

They didn’t, but it was one hell of a pain in the ass. We only ever saw it on the production databases and the crash, which was the main thing we were debugging, would only happen every week or so. Core dumps didn’t show anything useful because the crashes were just that a pointer was mysteriously pointing into a bad place. But how did it get corrupted? We were doing increasingly insane things to reproduce the problem locally. An interesting idea I had to reproduce it was using a database replica, which did crash sometimes. We used TCPDump to capture the replication stream, then waited a week for the replica to crash. We could start up a copy of the replica in the state it was at the beginning and replay the week of replication data into the socket to reproduce the problem. My theory was that since a DB replica doesn’t do anything other than the sequence of operations that come to it via the replication stream, that this should deterministicly reproduce the problem. Turns out that it didn’t. Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though.

guerrilla 4y ago
> Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though. Ahhh I could see how that could be! Fascinating. That's exactly the kind of heisenbug I used to love. I always liked the ones that seemed literally impossible but had a nice explanation.
- iquerno 4y ago
  schrödinbug*