5 ms·

My first compiler bug was in my first year at Google. I'd just introduced a new system for the animation that updates your position while driving in Google Maps

My first compiler bug was in my first year at Google. I'd just introduced a new system for the animation that updates your position while driving in Google Maps. It was perfectly buttery smooth as planned, except on my manager's commute the next day, where it constantly lurched back and forth. The others on the team were convinced that it had to be something with my code, but I didn't think it could be, because my code had no conditional statements and should either always be right or always be wrong. It kind of looked like it was being fed nonsense speed values, so I got the GPS log from my manager and checked - but no weird speed values, actually a remarkably clean GPS log. Replayed his GPS on my phone - worked perfectly fine, buttery smooth. Eventually it came out that it only happened on my manager's phone. Borrowing said phone and narrowing things down with printf, I showed that my core animation function was being called with the correct values (a, b, c, d) but was being run with the wrong ones (a, a, c d). This is when my manager thought to mention that he was running the latest internal alpha preview of Android. Searching Android's bug tracker for JIT bugs, I found that they had a known register aliasing bug. Honestly I have no idea how it ran well enough to get to my code in the first place. But I tagged my weird animation bug as related to that (they didn't really believe me) and ignored it until they fixed their thing, at which point it went away.

sebazzz 6y ago
JIT bugs must be horrible to debug because they happen in full running programs and might not always result in crashes. RyuJIT of dotnet had a bug a few years ago which caused certain calculations on certain specific conditions to go wrong.
- cblum 6y ago
  I caught a JIT bug in .NET back when I worked at Microsoft. Unfortunately I don't remember the specifics, but it was doing something wrong when trying to optimize a bit of code using AVX instructions. I was immediately sure it was a compiler bug when I realized I could make the code work correctly if I changed the order of variable declarations. It would happen on something like: int a; int b; float c; (again, that's only illustrative - too long ago for me to remember the specifics) But not with: int a; float c; int b; Folks in my team didn't want to believe it at first :) Edit: found it here - https://github.com/dotnet/runtime/issues/17395 https://github.com/dotnet/runtime/issues/17395 Not quite as I described but close.
  - pfdietz 6y ago
    That not only shows it's a compiler bug (assuming the program had no undefined behavior), it gives a general technique for generating compiler tests. Take any program, randomly change the order of declarations in a way that doesn't alter the meaning of the program, and see if both versions have the same behavior on some set of inputs. This is an example of "metamorphic testing".
- jtolmar 6y ago
  I ran into several other JIT bugs working on Google Maps. Since I worked on location, my code was the first complex thing that ran under most flows, and rather prone to being the first thing to crash when the system behaved in a very dumb way. About once a year we'd get a spike of crash reports from some ancient version of Android where a third party had built a custom JIT to get the system to run on very cheap hardware. Then the crash would be something absurd, like a null pointer on a value that was never assigned null, or an ArrayIndexOutOfBounds on a statically sized array that was only ever accessed with constants. Sometimes someone would actually try to solve these, but I preferred the times where someone just changed the Proguard config and hope it shuffled things around enough that it didn't trigger whatever bug the JIT had.
  - exikyut 6y ago
    > spike of crash reports from some ancient version of Android Quasi-related question: app installs/updates have been very very consistently SIGBUSing system_server with BUS_ADRALN on my ancient Galaxy Note 3 for about a year, but downgrading the Play Store app to the factory version (5.5.12 :D) makes the problem go away completely. I've tentatively considered writing something up for issuetracker.google.com, but besides tons of logcats I'm not sure what info to provide, and I also wonder if the info would be considered useful due to the age of the device. Any advice on whether/how to proceed would be appreciated!
    - jtolmar 6y ago
      I think the biggest problem you're going to run into is that there are very few people assigned to maintaining old systems, and it's quite hard to get these issues brought to anyone's attention. The ones I mentioned only got fixed because they crept up to the top most common causes of crashes. A bug report would be the first thing I'd want for something like this - in addition to logcat info it includes a bunch of process information, and apps can add custom data to the report. It'd also be nice to know anything to narrow down when the problem started - whatever bounds you can provide on when you were last absolutely certain it was up to date and working, plus your best guess. Someone's going to have to dig a Galaxy Note 3 out of a drawer somewhere, go into the archives of old Play Store versions, and try installing them until they find the version that broke it, so anything that makes that binary search process less terrible would be a start.
- cpeterso 6y ago
  Mozilla engineer Yulia Startsev has a series of live-coding YouTube videos called "Compiler Compiler" about developing and debugging Firefox's SpiderMonkey JS interpreter and JIT: https://www.youtube.com/results?search_query=Mozilla+Hacks+Compiler+Compiler https://www.youtube.com/results?search_query=Mozilla+Hacks+C...
- samatman 6y ago
  Yep, JIT bugs are a bear. We use LuaJIT pretty extensively and some codepaths trigger a crash-on-assert which we've confirmed is a JIT bug. But not consistently, the weather has to be just right. 95% of the time, randomly inserting `assert(true)` into the code un-breaks it. We're using the beta, so this is an expected sort of thing. I'd love to be the guy who squashes it, but that's way outside my wheelhouse. I'm left crossing my fingers and hoping one of the updates makes it go away. Globally disabling the JIT is a nice way to confirm that using LuaJIT was a good idea, though. Instant code starts taking seconds to complete.
- ShroudedNight 6y ago
  Back when I worked on IBM's J9, the nightmare scenario was either a dropped memory fence, or the map keeping track of links between register / stack references and GC heap objects getting screwed up. Things could go on a very long time before those errors manifest visibly.

sebazzz 6y ago

JIT bugs must be horrible to debug because they happen in full running programs and might not always result in crashes. RyuJIT of dotnet had a bug a few years ago which caused certain calculations on certain specific conditions to go wrong.

cblum 6y ago
I caught a JIT bug in .NET back when I worked at Microsoft. Unfortunately I don't remember the specifics, but it was doing something wrong when trying to optimize a bit of code using AVX instructions. I was immediately sure it was a compiler bug when I realized I could make the code work correctly if I changed the order of variable declarations. It would happen on something like: int a; int b; float c; (again, that's only illustrative - too long ago for me to remember the specifics) But not with: int a; float c; int b; Folks in my team didn't want to believe it at first :) Edit: found it here - https://github.com/dotnet/runtime/issues/17395 https://github.com/dotnet/runtime/issues/17395 Not quite as I described but close.
- pfdietz 6y ago
  That not only shows it's a compiler bug (assuming the program had no undefined behavior), it gives a general technique for generating compiler tests. Take any program, randomly change the order of declarations in a way that doesn't alter the meaning of the program, and see if both versions have the same behavior on some set of inputs. This is an example of "metamorphic testing".
jtolmar 6y ago
I ran into several other JIT bugs working on Google Maps. Since I worked on location, my code was the first complex thing that ran under most flows, and rather prone to being the first thing to crash when the system behaved in a very dumb way. About once a year we'd get a spike of crash reports from some ancient version of Android where a third party had built a custom JIT to get the system to run on very cheap hardware. Then the crash would be something absurd, like a null pointer on a value that was never assigned null, or an ArrayIndexOutOfBounds on a statically sized array that was only ever accessed with constants. Sometimes someone would actually try to solve these, but I preferred the times where someone just changed the Proguard config and hope it shuffled things around enough that it didn't trigger whatever bug the JIT had.
- exikyut 6y ago
  > spike of crash reports from some ancient version of Android Quasi-related question: app installs/updates have been very very consistently SIGBUSing system_server with BUS_ADRALN on my ancient Galaxy Note 3 for about a year, but downgrading the Play Store app to the factory version (5.5.12 :D) makes the problem go away completely. I've tentatively considered writing something up for issuetracker.google.com, but besides tons of logcats I'm not sure what info to provide, and I also wonder if the info would be considered useful due to the age of the device. Any advice on whether/how to proceed would be appreciated!
  - jtolmar 6y ago
    I think the biggest problem you're going to run into is that there are very few people assigned to maintaining old systems, and it's quite hard to get these issues brought to anyone's attention. The ones I mentioned only got fixed because they crept up to the top most common causes of crashes. A bug report would be the first thing I'd want for something like this - in addition to logcat info it includes a bunch of process information, and apps can add custom data to the report. It'd also be nice to know anything to narrow down when the problem started - whatever bounds you can provide on when you were last absolutely certain it was up to date and working, plus your best guess. Someone's going to have to dig a Galaxy Note 3 out of a drawer somewhere, go into the archives of old Play Store versions, and try installing them until they find the version that broke it, so anything that makes that binary search process less terrible would be a start.
cpeterso 6y ago
Mozilla engineer Yulia Startsev has a series of live-coding YouTube videos called "Compiler Compiler" about developing and debugging Firefox's SpiderMonkey JS interpreter and JIT: https://www.youtube.com/results?search_query=Mozilla+Hacks+Compiler+Compiler https://www.youtube.com/results?search_query=Mozilla+Hacks+C...
samatman 6y ago
Yep, JIT bugs are a bear. We use LuaJIT pretty extensively and some codepaths trigger a crash-on-assert which we've confirmed is a JIT bug. But not consistently, the weather has to be just right. 95% of the time, randomly inserting `assert(true)` into the code un-breaks it. We're using the beta, so this is an expected sort of thing. I'd love to be the guy who squashes it, but that's way outside my wheelhouse. I'm left crossing my fingers and hoping one of the updates makes it go away. Globally disabling the JIT is a nice way to confirm that using LuaJIT was a good idea, though. Instant code starts taking seconds to complete.
ShroudedNight 6y ago
Back when I worked on IBM's J9, the nightmare scenario was either a dropped memory fence, or the map keeping track of links between register / stack references and GC heap objects getting screwed up. Things could go on a very long time before those errors manifest visibly.