4 ms·

If you think that is bad, then you might think that it’s even worse that memcpy() was broken for years when running a 32 bit process on a 64 bit linux system. A

If you think that is bad, then you might think that it’s even worse that memcpy() was broken for years when running a 32 bit process on a 64 bit linux system. Also due to a hand optimised assembly function. We had an issue for years where occasionally our backend database would fall over and we were just seeing strange corruption in the stored data sometimes. Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. What blew me away is the idea that there can be a bug in something that is used literally everywhere all the time. In addition, there was a pull request with a fix that someone had written that got ignored for years that we found. We just applied the diff in the pull request and it fixed our issues. The fix was eventually applied upstream a few years later.

guerrilla 4y ago
> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. That sounds like an interesting detective story. Did they write it up anywhere?
- Negitivefrags 4y ago
  They didn’t, but it was one hell of a pain in the ass. We only ever saw it on the production databases and the crash, which was the main thing we were debugging, would only happen every week or so. Core dumps didn’t show anything useful because the crashes were just that a pointer was mysteriously pointing into a bad place. But how did it get corrupted? We were doing increasingly insane things to reproduce the problem locally. An interesting idea I had to reproduce it was using a database replica, which did crash sometimes. We used TCPDump to capture the replication stream, then waited a week for the replica to crash. We could start up a copy of the replica in the state it was at the beginning and replay the week of replication data into the socket to reproduce the problem. My theory was that since a DB replica doesn’t do anything other than the sequence of operations that come to it via the replication stream, that this should deterministicly reproduce the problem. Turns out that it didn’t. Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though.
  - guerrilla 4y ago
    > Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though. Ahhh I could see how that could be! Fascinating. That's exactly the kind of heisenbug I used to love. I always liked the ones that seemed literally impossible but had a nice explanation.
    - iquerno 4y ago
      schrödinbug*
butlerm 4y ago
> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. That is surprising outside of the kernel. There are a large number of programs that would fail very hard very fast with that sort of artificial restriction.
ketralnis 4y ago
> What blew me away is the idea that there can be a bug in something that is used literally everywhere all the time. We've become so accustomed to computers being generally unreliable that I guess people see something weird and just reboot. We checksum, run reductant servers, and just generally tolerate and work around way more failures and weirdness than we should. It's funny because we're theoretically an industry that could be full of maths and proofs but ask anybody outside of it and you'll get "yeah computers just don't work sometimes I guess that's how it is"
- ratorx 4y ago
  The problem is that even a perfectly implemented computer will fail randomly, due to inherent unreliability in the underlying hardware (cosmic rays, bad silicon etc.). And once you start designing around that unreliability, it is harder to justify chasing esoteric bugs rather than folding that alongside the probability model of random hardware failure. It’s hard for any system dealing with the real world to be completely “maths and proofs”. There is plenty of it that’s applied even in the real world systems we operate today. At the end of the day, it’s Engineering and not a pure science. And I guess the other problem is that quite often you don’t need to quantify the failure modes and do “proper engineering”, because a lot of the systems we build and use just don’t need the level of reliability that e.g. a bridge needs (yet). There are definitely industries where software reliability is considered paramount, but that’s not the majority of the market.
  - ketralnis 4y ago
    > a lot of the systems we build and use just don’t need the level of reliability that e.g. a bridge needs Sure. As the saying goes, "Anybody can build a bridge that never falls down. But it takes an engineer to build a bridge that _almost_ doesn't fall down" (referring to being able to build it in a cost-conscious way). There's a spectrum there for sure, but as an industry we're on the other side of it from where I personally wish we were. We should be embarrassed that "it must have been a computer glitch" can be used to explain away nearly any problem. Because it's usually right, and it's usually our fault rather than the cosmic rays.
CJefferson 4y ago
Similar issue, https://sourceware.org/bugzilla/show_bug.cgi?id=22644 https://sourceware.org/bugzilla/show_bug.cgi?id=22644 , which was found by a friend of mine and took is ages to track down. memmove was broken on 32-bit executables on systems with SSE2 whenever the move occurred over the 2GB memory boundary due to an issue with signed vs unsigned integers.
- benibela 4y ago
  I might have seen my 32-bit program crash from that. Either that or there was yet another bug in memmove/memcpy. I did not investigate it.
actionfromafar 4y ago
And people got funny looks for wanting to run 32 bit workloads in their own 32 bit VMs...
ryao 4y ago
That is not the first `memcpy()` issue that I have heard. There was the reverse `memcpy()` in glibc 2.13 that broke plenty of software using `memcpy()` in places where `memmove()` should have been used.
benibela 4y ago
I cross compile my side-project program from 64 to 32-bit and then I noticed, it would crash when testing after updating linux The stacktrace said the crash came from SSE memcpy. I did not bother investigating it, since I have no users anyways. I had an really old CPU (like i5-520m). Perhaps the code required some SSE feature, it did not have That was libc. I also use Pascal, which has its own memcpy implementation. A while ago, they discovered it does not handle alignment correctly. (like it would adjust by 4-i when it should have adjusted by i) I thought that is what I get when using Pascal rather than a more commonly used language, but apparently nothing works

guerrilla 4y ago

> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. That sounds like an interesting detective story. Did they write it up anywhere?

Negitivefrags 4y ago
They didn’t, but it was one hell of a pain in the ass. We only ever saw it on the production databases and the crash, which was the main thing we were debugging, would only happen every week or so. Core dumps didn’t show anything useful because the crashes were just that a pointer was mysteriously pointing into a bad place. But how did it get corrupted? We were doing increasingly insane things to reproduce the problem locally. An interesting idea I had to reproduce it was using a database replica, which did crash sometimes. We used TCPDump to capture the replication stream, then waited a week for the replica to crash. We could start up a copy of the replica in the state it was at the beginning and replay the week of replication data into the socket to reproduce the problem. My theory was that since a DB replica doesn’t do anything other than the sequence of operations that come to it via the replication stream, that this should deterministicly reproduce the problem. Turns out that it didn’t. Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though.
- guerrilla 4y ago
  > Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though. Ahhh I could see how that could be! Fascinating. That's exactly the kind of heisenbug I used to love. I always liked the ones that seemed literally impossible but had a nice explanation.
  - iquerno 4y ago
    schrödinbug*

butlerm 4y ago

> Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it. That is surprising outside of the kernel. There are a large number of programs that would fail very hard very fast with that sort of artificial restriction.

ketralnis 4y ago

> What blew me away is the idea that there can be a bug in something that is used literally everywhere all the time. We've become so accustomed to computers being generally unreliable that I guess people see something weird and just reboot. We checksum, run reductant servers, and just generally tolerate and work around way more failures and weirdness than we should. It's funny because we're theoretically an industry that could be full of maths and proofs but ask anybody outside of it and you'll get "yeah computers just don't work sometimes I guess that's how it is"

ratorx 4y ago
The problem is that even a perfectly implemented computer will fail randomly, due to inherent unreliability in the underlying hardware (cosmic rays, bad silicon etc.). And once you start designing around that unreliability, it is harder to justify chasing esoteric bugs rather than folding that alongside the probability model of random hardware failure. It’s hard for any system dealing with the real world to be completely “maths and proofs”. There is plenty of it that’s applied even in the real world systems we operate today. At the end of the day, it’s Engineering and not a pure science. And I guess the other problem is that quite often you don’t need to quantify the failure modes and do “proper engineering”, because a lot of the systems we build and use just don’t need the level of reliability that e.g. a bridge needs (yet). There are definitely industries where software reliability is considered paramount, but that’s not the majority of the market.
- ketralnis 4y ago
  > a lot of the systems we build and use just don’t need the level of reliability that e.g. a bridge needs Sure. As the saying goes, "Anybody can build a bridge that never falls down. But it takes an engineer to build a bridge that _almost_ doesn't fall down" (referring to being able to build it in a cost-conscious way). There's a spectrum there for sure, but as an industry we're on the other side of it from where I personally wish we were. We should be embarrassed that "it must have been a computer glitch" can be used to explain away nearly any problem. Because it's usually right, and it's usually our fault rather than the cosmic rays.

CJefferson 4y ago

Similar issue, https://sourceware.org/bugzilla/show_bug.cgi?id=22644 https://sourceware.org/bugzilla/show_bug.cgi?id=22644 , which was found by a friend of mine and took is ages to track down. memmove was broken on 32-bit executables on systems with SSE2 whenever the move occurred over the 2GB memory boundary due to an issue with signed vs unsigned integers.

benibela 4y ago
I might have seen my 32-bit program crash from that. Either that or there was yet another bug in memmove/memcpy. I did not investigate it.

actionfromafar 4y ago

And people got funny looks for wanting to run 32 bit workloads in their own 32 bit VMs...

ryao 4y ago

That is not the first `memcpy()` issue that I have heard. There was the reverse `memcpy()` in glibc 2.13 that broke plenty of software using `memcpy()` in places where `memmove()` should have been used.

benibela 4y ago

I cross compile my side-project program from 64 to 32-bit and then I noticed, it would crash when testing after updating linux The stacktrace said the crash came from SSE memcpy. I did not bother investigating it, since I have no users anyways. I had an really old CPU (like i5-520m). Perhaps the code required some SSE feature, it did not have That was libc. I also use Pascal, which has its own memcpy implementation. A while ago, they discovered it does not handle alignment correctly. (like it would adjust by 4-i when it should have adjusted by i) I thought that is what I get when using Pascal rather than a more commonly used language, but apparently nothing works