7 ms·

The Infinite Loop That Wasn't

275 points by tambre 7y ago

techdragon 7y ago
I love reading these kinds of reverse engineering discovery tales, I find them technically educative, and overall enlightening... they help put my own struggles debugging thorny things into sharp perspective, making me even more appreciative of the level of control I have over my own code and it’s runtime environments. The level of effort put into reverse engineering for emulating classic hardware of all kinds is just remarkable. The Dolphin EMU team has a long history of these kind of blog posts for people looking for more of this kind of writing https://dolphin-emu.org/blog/ https://dolphin-emu.org/blog/
hyperman1 7y ago
I've always had the attitude that if can reproduce a bug consistently, I can debug it, and I can fix it. Now here's a nail in the coffin of that theory.
- aidenn0 7y ago
  A rule of thumb I use is that if I can reproduce a bug in approximately 0 time, I can fix it within 1 hour 95% of the time.
- masswerk 7y ago
  In defense of the coffin, I'd say this is not so much about a bug, but about reverse engineering real world effects of undefined behavior (as caused by a bug in software).
  - deleted 7y ago
    [deleted]
  - aidenn0 7y ago
    If the goal of mgba is to reproduce the GBA hardware faithfully, then this is a pure software bug in mgba.
    - NieDzejkob 7y ago
      You are trying to argue about the meaning of the words being used. This is not productive, as what matters is what the author of the argument meant when they used that word.
      - aidenn0 7y ago
        I don't think that's the case. When talking about emulation and virtualization and such, it is easy to get terms backwards. For example, a bug in the implementation of a virtualized device looks like a hardware bug to the client and like a software bug to the host.
        masswerk 7y ago
        To be honest, I think it's about crossing the lines of emulation and simulation. In emulation, we mostly consider instruction level behavior as good enough, but here, we'll have to venture below cycle level and integrated hardware effects to copy the behavior.
    - pingyong 7y ago
      That is not the goal of mGBA (or any GBA emulator for that matter), btw. If you wanted to reproduce it faithfully, the emulator would be unusably slow. https://emulation.gametechwiki.com/index.php/Emulation_Accuracy https://emulation.gametechwiki.com/index.php/Emulation_Accur...
      - aidenn0 7y ago
        s/faithfully/faithfully enough/ Prior to the bug being fixed this was clearly not faithfully enough for at least 2 games.
- rgoulter 7y ago
  Most of the bugs I've dealt with, the problem was usually down to something simple that can be explained quickly. (e.g. syntactically valid typo, config error, simple logic error etc.). Seems like "Holy grail" bugs like this happen to need a thorough understanding of complex systems (which I usually like to hide under layers of abstraction).
- umvi 7y ago
  Only if the problem is purely software. I've had reproducible problems that were impossible to solve by myself. The process went like this: - Huh, getting USB disconnects randomly, need to fix this - Hmm, device driver seems to be fine - Hmm usbnet driver seems fine - Hmm xhci driver seem fine - $5000 and one USB 3.0 protocol analyzer later... - What the heck is this host controller doing? At this point it was looking like a hardware problem but seeing as I didn't have the schematics or verilog of the host controller I could go no further (nor did I want too... I never imagined USB could be such a deep rabbit hole).
Out_of_Characte 7y ago
Amazing story. Just shows you how complex hardware can truly be.
- wkearney99 7y ago
  Correct. When you're coding really close to the metal you discover (and perhaps depend upon) all kinds of shenanigans that a higher level language compiler might work around. The graphics on Atari 2600 had to be fed line-by-line during the vertical blanking interval. Made for some really tricky assembly coding, trying to get enough done to make the interval without wasting VERY limited ROM resources (aka can't cheap with NOP instructions).
  - abbeyj 7y ago
    I'm assuming you meant "horizontal blanking interval" instead of "vertical blanking interval". But I don't think that's entirely correct either. Some games will change the registers while the beam is actively drawing so that, e.g., the left half of the line will be drawn with one set of settings and the right half drawn with another.
unwind 7y ago
So uh I didn't see it mentioned, but the question of why this code is in the games arises. Is it buggy code, or something intentional like anti-emulator sneakery?
- hrydgard 7y ago
  It's pretty likely that it's just buggy code that happened to work on hardware.
  - deleted 7y ago
    [deleted]
  - near 7y ago
    Definitely just working by chance in this case. This comes up a lot, unfortunately. Small budgets, tight deadlines, games not validated for correctness under software emulation for detecting these types of edge cases prior to shipping. Another example: there are several GBA games that dereference null pointers, but it works because there is no memory protection fault hardware in the GBA. Since the BIOS is mapped to address 0 and yet locked out from reading post-boot (in a failed anti-copying mechanism), you have to emulate the open bus behavior to run these games correctly as a result. There are also several anti-emulator routines in the GBA library, such as lying about the type of save memory (flash, EEPROM, or SRAM) that a given game actually has (by including strings in the ROM from the Nintendo SDK used to access them.)
    - Wowfunhappy 7y ago
      Still, in the case of Hello Kitty, there's got to be a reason the developer added that loop at startup, right? They might have not understood the intricacies of how or why it worked, but presumably they were trying to do something.
      - Izkata 7y ago
        > Then I noticed a difference between my test ROM and Pokémon Emerald when they hang. There was music playing in Pokémon. There was also music playing in Sonic Pinball Party. There wasn’t music playing in Hello Kitty, but this gave me an idea. Maybe there was supposed to be music playing, this was for timing it, and it was removed, broken, or just forgotten about?
brailsafe 7y ago
Mental.
libraryofbabel 7y ago
This reminds me of the wonderful novel "The Bug" by Ellen Ullman. A beautifully-written fictionalized account of the quest for a "Holy Grail" bug.
DubiousPusher 7y ago
Interesting story. Goes to show you the falacy of writing fault tolerant systems as opposed to writing noisy systems. Had the GBA architects made the device halt on invalid memory rather than return the last thing on the bus and hope everything would be ok this entire class of bug wouldn't exist and the authors of that code likely would've found that bug during production.
- Wowfunhappy 7y ago
  > Had the GBA architects made the device halt on invalid memory rather than return the last thing on the bus and hope everything would be ok this entire class of bug wouldn't exist and the authors of that code likely would've found that bug during production. Of course, would it matter if these bugs were found? The games ran fine on real hardware. There’s no maintenance burden either, because updating a gba game is literally impossible.
  - ranger207 7y ago
    >updating a gba game is literally impossible. Not quite. The Pokémon Ruby/Sapphire Berry Glitch bug was fixed by connecting to games that had a patch program, such as Pokémon Emerald, the 3rd in the Ruby/Sapphire/Emerald series. https://bulbapedia.bulbagarden.net/wiki/Berry_glitch https://bulbapedia.bulbagarden.net/wiki/Berry_glitch
    - Wowfunhappy 7y ago
      Huh, cool. Does anyone know how this worked? I know the carts themselves are mostly write-once so the bug and patch must have been related to the save portion or some hardware specific to this cartridge, right?
      - wizzwizz4 7y ago
        I think it just used an arbitrary code execution thing to change the system clock, so that the faulty code was still there, but never failed.
matheusmoreira 7y ago
These test ROMs are interesting. Since they are designed to run on real hardware that lacks debugging capabilities, the user must be able to operate the test suite and check the results manually. They aren't like modern software test suites. According to the nesdev wiki, automation consists of playing back controller input, waiting for the tests to run, taking a screen shot of the results screen, hashing it and comparing the result to the hash of the expected results screen. This method also allows testing real games to check for reproducible bugs. https://wiki.nesdev.com/w/index.php/Emulator_tests#Automated_testing https://wiki.nesdev.com/w/index.php/Emulator_tests#Automated... Pretty nice. I expected the test ROM to write the results to some special location in memory and have the emulator report those results.

275 points by tambre 7y ago

I love reading these kinds of reverse engineering discovery tales, I find them technically educative, and overall enlightening... they help put my own struggles debugging thorny things into sharp perspective, making me even more appreciative of the level of control I have over my own code and it’s runtime environments. The level of effort put into reverse engineering for emulating classic hardware of all kinds is just remarkable. The Dolphin EMU team has a long history of these kind of blog posts for people looking for more of this kind of writing https://dolphin-emu.org/blog/ https://dolphin-emu.org/blog/

hyperman1 7y ago

I've always had the attitude that if can reproduce a bug consistently, I can debug it, and I can fix it. Now here's a nail in the coffin of that theory.

aidenn0 7y ago
A rule of thumb I use is that if I can reproduce a bug in approximately 0 time, I can fix it within 1 hour 95% of the time.
masswerk 7y ago
In defense of the coffin, I'd say this is not so much about a bug, but about reverse engineering real world effects of undefined behavior (as caused by a bug in software).
- deleted 7y ago
  [deleted]
- aidenn0 7y ago
  If the goal of mgba is to reproduce the GBA hardware faithfully, then this is a pure software bug in mgba.
  - NieDzejkob 7y ago
    You are trying to argue about the meaning of the words being used. This is not productive, as what matters is what the author of the argument meant when they used that word.
    - aidenn0 7y ago
      I don't think that's the case. When talking about emulation and virtualization and such, it is easy to get terms backwards. For example, a bug in the implementation of a virtualized device looks like a hardware bug to the client and like a software bug to the host.
      - masswerk 7y ago
        To be honest, I think it's about crossing the lines of emulation and simulation. In emulation, we mostly consider instruction level behavior as good enough, but here, we'll have to venture below cycle level and integrated hardware effects to copy the behavior.
  - pingyong 7y ago
    That is not the goal of mGBA (or any GBA emulator for that matter), btw. If you wanted to reproduce it faithfully, the emulator would be unusably slow. https://emulation.gametechwiki.com/index.php/Emulation_Accuracy https://emulation.gametechwiki.com/index.php/Emulation_Accur...
    - aidenn0 7y ago
      s/faithfully/faithfully enough/ Prior to the bug being fixed this was clearly not faithfully enough for at least 2 games.
rgoulter 7y ago
Most of the bugs I've dealt with, the problem was usually down to something simple that can be explained quickly. (e.g. syntactically valid typo, config error, simple logic error etc.). Seems like "Holy grail" bugs like this happen to need a thorough understanding of complex systems (which I usually like to hide under layers of abstraction).
umvi 7y ago
Only if the problem is purely software. I've had reproducible problems that were impossible to solve by myself. The process went like this: - Huh, getting USB disconnects randomly, need to fix this - Hmm, device driver seems to be fine - Hmm usbnet driver seems fine - Hmm xhci driver seem fine - $5000 and one USB 3.0 protocol analyzer later... - What the heck is this host controller doing? At this point it was looking like a hardware problem but seeing as I didn't have the schematics or verilog of the host controller I could go no further (nor did I want too... I never imagined USB could be such a deep rabbit hole).

Out_of_Characte 7y ago

Amazing story. Just shows you how complex hardware can truly be.

wkearney99 7y ago
Correct. When you're coding really close to the metal you discover (and perhaps depend upon) all kinds of shenanigans that a higher level language compiler might work around. The graphics on Atari 2600 had to be fed line-by-line during the vertical blanking interval. Made for some really tricky assembly coding, trying to get enough done to make the interval without wasting VERY limited ROM resources (aka can't cheap with NOP instructions).
- abbeyj 7y ago
  I'm assuming you meant "horizontal blanking interval" instead of "vertical blanking interval". But I don't think that's entirely correct either. Some games will change the registers while the beam is actively drawing so that, e.g., the left half of the line will be drawn with one set of settings and the right half drawn with another.

unwind 7y ago

So uh I didn't see it mentioned, but the question of why this code is in the games arises. Is it buggy code, or something intentional like anti-emulator sneakery?

hrydgard 7y ago
It's pretty likely that it's just buggy code that happened to work on hardware.
- deleted 7y ago
  [deleted]
- near 7y ago
  Definitely just working by chance in this case. This comes up a lot, unfortunately. Small budgets, tight deadlines, games not validated for correctness under software emulation for detecting these types of edge cases prior to shipping. Another example: there are several GBA games that dereference null pointers, but it works because there is no memory protection fault hardware in the GBA. Since the BIOS is mapped to address 0 and yet locked out from reading post-boot (in a failed anti-copying mechanism), you have to emulate the open bus behavior to run these games correctly as a result. There are also several anti-emulator routines in the GBA library, such as lying about the type of save memory (flash, EEPROM, or SRAM) that a given game actually has (by including strings in the ROM from the Nintendo SDK used to access them.)
  - Wowfunhappy 7y ago
    Still, in the case of Hello Kitty, there's got to be a reason the developer added that loop at startup, right? They might have not understood the intricacies of how or why it worked, but presumably they were trying to do something.
    - Izkata 7y ago
      > Then I noticed a difference between my test ROM and Pokémon Emerald when they hang. There was music playing in Pokémon. There was also music playing in Sonic Pinball Party. There wasn’t music playing in Hello Kitty, but this gave me an idea. Maybe there was supposed to be music playing, this was for timing it, and it was removed, broken, or just forgotten about?

brailsafe 7y ago

Mental.

libraryofbabel 7y ago

This reminds me of the wonderful novel "The Bug" by Ellen Ullman. A beautifully-written fictionalized account of the quest for a "Holy Grail" bug.

DubiousPusher 7y ago

Interesting story. Goes to show you the falacy of writing fault tolerant systems as opposed to writing noisy systems. Had the GBA architects made the device halt on invalid memory rather than return the last thing on the bus and hope everything would be ok this entire class of bug wouldn't exist and the authors of that code likely would've found that bug during production.

Wowfunhappy 7y ago
> Had the GBA architects made the device halt on invalid memory rather than return the last thing on the bus and hope everything would be ok this entire class of bug wouldn't exist and the authors of that code likely would've found that bug during production. Of course, would it matter if these bugs were found? The games ran fine on real hardware. There’s no maintenance burden either, because updating a gba game is literally impossible.
- ranger207 7y ago
  >updating a gba game is literally impossible. Not quite. The Pokémon Ruby/Sapphire Berry Glitch bug was fixed by connecting to games that had a patch program, such as Pokémon Emerald, the 3rd in the Ruby/Sapphire/Emerald series. https://bulbapedia.bulbagarden.net/wiki/Berry_glitch https://bulbapedia.bulbagarden.net/wiki/Berry_glitch
  - Wowfunhappy 7y ago
    Huh, cool. Does anyone know how this worked? I know the carts themselves are mostly write-once so the bug and patch must have been related to the save portion or some hardware specific to this cartridge, right?
    - wizzwizz4 7y ago
      I think it just used an arbitrary code execution thing to change the system clock, so that the faulty code was still there, but never failed.

matheusmoreira 7y ago

These test ROMs are interesting. Since they are designed to run on real hardware that lacks debugging capabilities, the user must be able to operate the test suite and check the results manually. They aren't like modern software test suites. According to the nesdev wiki, automation consists of playing back controller input, waiting for the tests to run, taking a screen shot of the results screen, hashing it and comparing the result to the hash of the expected results screen. This method also allows testing real games to check for reproducible bugs. https://wiki.nesdev.com/w/index.php/Emulator_tests#Automated_testing https://wiki.nesdev.com/w/index.php/Emulator_tests#Automated... Pretty nice. I expected the test ROM to write the results to some special location in memory and have the emulator report those results.