6 ms·

The Edge of Emulation

245 points by NieDzejkob 7y ago

krallja 7y ago
I love that there are people whose hobby is preserving things. SNES was an important part of so many 80s and 90s kids’ childhoods! But how many of them have digital logic experience, especially at the reverse engineering level? And actually want to do it outside of compensated labor?
- near 7y ago
  > how many of them have digital logic experience, especially at the reverse engineering level? Like programming and reverse-engineering, I taught myself the basics of electrical engineering. The Neo Geo Pocket's SoC is basically nothing but raw logic circuits for explanations of how things work. Imagine 200 pages of diagrams like this: https://i.imgur.com/2LZ2UWY.png https://i.imgur.com/2LZ2UWY.png (oh and as a bonus, the diagrams often contain errors.) If you're only looking to create a basic emulator, you can mostly get away without knowing all of this stuff by reading tech docs and the source code of other emulators. But if you really want to get low-level and get things clock-cycle accurate, or want to work on a system that's not been well-emulated to date, digital logic pretty much a hard requirement. > And actually want to do it outside of compensated labor? From 2004 - 2018 for the SNES, it was mostly only me. As of today, three people. In general only one person is really needed. I couldn't say if someone else would have taken my place had I not been around. I suspect so, given how popular the SNES continues to be. Whether they would have done a better job of it than I have is a question that often keeps me up at night. In any case, the answer is certainly "not nearly enough."
nitrogen 7y ago
One of the later comments on the article offers custom PCB design to make a test harness for the PPUs. It seems like a perfect partner for a manually controllable clock for the whole SNES, e.g. as Ben Eater's 6502 demos use. If you can freeze the clock at will, you can more easily inspect the data and address lines. But this wouldn't work if the PPU latch timing depends on analog effects or the PPUs have an internal clock. Combine the test harness with a USB logic analyzer and signal generator, and borrow some concepts from fuzzers like AFL, and it should be possible to automatically identify critical phases of the PPU operations. Question: do the PPUs themselves generate analog video, or is there a separate DAC whose bus could be tapped to figure out what color the PPU is producing?
- near 7y ago
  The PPUs use static logic and an external 21MHz oscillator (which also powers the CPU), so they're perfect for single-stepping in isolation from the rest of the SNES. The PPUs themselves output only analog RGB, there's an analog pin for each color channel rather than a digital pin for each color bit. On some level, I don't doubt it's possible to build a test harness that can automate things based on the analog RGB values with some fuzzy matching, but if we are going that far, it just makes more sense to snoop the bus traffic directly. That would reveal a lot more information and in digital form.
  - londons_explore 7y ago
    > it just makes more sense to snoop the bus traffic directly It seems the bus traffic won't tell you everything. Just because the PPU reads a specific piece of a sprite from RAM, doesn't tell you which pixels of a scanline it'll get rendered on. If I were approaching this problem, I would observe that the 'analog' output of a SNES is probably a bunch of discrete values with discrete timing intervals, and therefore can be perfectly captured with no error, especially if you can slow the clock down to eliminate signal reflections etc. I would then make a test harness consisting of a PPU chip, the ability to read and write registers (either with the rest of the SNES, or some other microcontroller, whichever is easier) and the ability to capture complete frame outputs from the 'analog' outputs without error. I would then do the same with an emulator, and run code on both which pokes random registers at random (clock accurate) times, and whenever a difference in output appears, debug it.
    - near 7y ago
      > It seems the bus traffic won't tell you everything. Oh it absolutely won't, it's only the external state. It's just another important piece of the puzzle that I think would help us out a lot. Solving this would most likely require reverse-engineering the netlists from a decapped die scan. I've since (post article writing) learned that a German forum found a hidden test mode that outputs digital RGB values, but it has a lot of issues on various edge cases, so while it's not a perfect solution, it's definitely a huge help along the way if we can get a testing rig set up around an appropriately modded SNES.
wgjordan 7y ago
> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it might cause issues in another game. > We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles. Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. This sounds like standard software-engineering practice to me. I'm surprised that a test-harness like this doesn't already exist by now given the relatively mature state of SNES emulation. It seems like it would be an extremely helpful tool to benchmark accuracy of new or existing emulators, prevent regressions from advanced optimizations, etc. Automated testing could transform the 'blind guessing' approach on tweaking PPU timings from a game of 'whack a mole' to a structured, iterative linear algebra solver. Which could end up being quite a bit more feasible than acquiring 100x die scans of the chip and tracing the logic from the hardware directly.
- xenadu02 7y ago
  I don't know if SNES dev kits still exist but if they do they might prove useful. Such kits sometimes have additional debugging facilities because they're also used for validation (or they share some elements). Not sure if anyone was doing that sort of thing as early as SNES though.
- kevingadd 7y ago
  You can't just start it up and run it for 5 seconds, though. A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole package and then verify the whole stream against a known good. So you'd need to identify the key points in each game that are important, and replay those with save states and hope that the save states aren't unintentionally encoding corrupted data that came from an existing bug in the emulator you used to run them. Also, none of that automation you just did will work w/the actual hardware.
  - theresistor 7y ago
    The input stream for an SNES isn't very complicated - just the state of a few buttons at every cycle, and the transition edges are quite sparse relative to the clock frequency. You could easily record human play sessions and replay them.
jimmaswell 7y ago
Why are test roms too slow if there are at most 41 cases to try? Is getting a game on an snes cart arduous?
- chungy 7y ago
  It can take several minutes to write out an EEPROM. It's enough to be frustrating.
  - vardump 7y ago
    https://en.wikipedia.org/wiki/Flash_memory_emulator https://en.wikipedia.org/wiki/Flash_memory_emulator "A flash emulator or flash memory emulator is a tool that is used to temporarily replace flash memory or ROM chips in an embedded device for the purpose of debugging embedded software. Such tools contain Dual-ported RAM, one port of which is connected to a target system (i.e. system, that is being debugged), and second is connected to a host (i.e. PC, which runs debugger)."
- near 7y ago
  There are 52 registers comprising approximately 100 settings, there are probably close to twice that number if you include the internal register latches (some of which we know about, some of which we don't know exist yet.) The SNES PPUs are heavily based around combinatorial logic. Changing the timing of one variable can alter the timing needed by other variables. Think of each setting as potentially doubling the number of tests needed (at least when it comes to a blind, brute-force approach.) There are more combinations of settings and pixel generation patterns than atoms in the universe. The way we've gotten as far as we have is that not every combination needs to be tested. It is a ballpark estimate on my part, but I'd estimate us needing a few million tests to have a high degree of confidence our emulation is correct. If we had to make those tests into visual patterns on a screen that had to be checked by eye each time, it would be overwhelming even if we only needed a few thousand tests made. I can test code on a live SNES extremely easily with my 21fx board ( https://github.com/defparam/21FX https://github.com/defparam/21FX ), but it's still far too much to do this by hand.
  - 7y ago
gregoryl 7y ago
What's the barrier to getting the chip decapped and scanned at the needed resolution?
- near 7y ago
  Really, just someone with the skill and interest in doing it. Money's no issue unless they want something crazy like $10,000+.
  - zamadatix 7y ago
    John D McMaster seems to have previously done 100x scans for some chips if I'm reading the site right e.g. the latest has a link for 20x and 100x https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/ https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/. Is it as simple as seeing if he'll rescan the PPUs at 100x or are these not the quality that I was thinking?
    - near 7y ago
      He has a tremendous backlog of work to do at the moment. I would prefer to not bother him unless it was a last resort, but that is indeed an option if my breadboard idea does not work out and no others are up to the task of decapping the PPUs.
peter_d_sherman 7y ago
Excerpt: "If we could recruit a talented electrical engineer, I believe that a custom PPU breakout board could be designed that would aid us substantially in reverse engineering..." Perhaps Andrew "Bunnie" Huang -- would be someone to go to, for this?: https://en.wikipedia.org/wiki/Andrew_Huang_(hacker) https://en.wikipedia.org/wiki/Andrew_Huang_(hacker)

245 points by NieDzejkob 7y ago

I love that there are people whose hobby is preserving things. SNES was an important part of so many 80s and 90s kids’ childhoods! But how many of them have digital logic experience, especially at the reverse engineering level? And actually want to do it outside of compensated labor?

near 7y ago
> how many of them have digital logic experience, especially at the reverse engineering level? Like programming and reverse-engineering, I taught myself the basics of electrical engineering. The Neo Geo Pocket's SoC is basically nothing but raw logic circuits for explanations of how things work. Imagine 200 pages of diagrams like this: https://i.imgur.com/2LZ2UWY.png https://i.imgur.com/2LZ2UWY.png (oh and as a bonus, the diagrams often contain errors.) If you're only looking to create a basic emulator, you can mostly get away without knowing all of this stuff by reading tech docs and the source code of other emulators. But if you really want to get low-level and get things clock-cycle accurate, or want to work on a system that's not been well-emulated to date, digital logic pretty much a hard requirement. > And actually want to do it outside of compensated labor? From 2004 - 2018 for the SNES, it was mostly only me. As of today, three people. In general only one person is really needed. I couldn't say if someone else would have taken my place had I not been around. I suspect so, given how popular the SNES continues to be. Whether they would have done a better job of it than I have is a question that often keeps me up at night. In any case, the answer is certainly "not nearly enough."

nitrogen 7y ago

One of the later comments on the article offers custom PCB design to make a test harness for the PPUs. It seems like a perfect partner for a manually controllable clock for the whole SNES, e.g. as Ben Eater's 6502 demos use. If you can freeze the clock at will, you can more easily inspect the data and address lines. But this wouldn't work if the PPU latch timing depends on analog effects or the PPUs have an internal clock. Combine the test harness with a USB logic analyzer and signal generator, and borrow some concepts from fuzzers like AFL, and it should be possible to automatically identify critical phases of the PPU operations. Question: do the PPUs themselves generate analog video, or is there a separate DAC whose bus could be tapped to figure out what color the PPU is producing?

near 7y ago
The PPUs use static logic and an external 21MHz oscillator (which also powers the CPU), so they're perfect for single-stepping in isolation from the rest of the SNES. The PPUs themselves output only analog RGB, there's an analog pin for each color channel rather than a digital pin for each color bit. On some level, I don't doubt it's possible to build a test harness that can automate things based on the analog RGB values with some fuzzy matching, but if we are going that far, it just makes more sense to snoop the bus traffic directly. That would reveal a lot more information and in digital form.
- londons_explore 7y ago
  > it just makes more sense to snoop the bus traffic directly It seems the bus traffic won't tell you everything. Just because the PPU reads a specific piece of a sprite from RAM, doesn't tell you which pixels of a scanline it'll get rendered on. If I were approaching this problem, I would observe that the 'analog' output of a SNES is probably a bunch of discrete values with discrete timing intervals, and therefore can be perfectly captured with no error, especially if you can slow the clock down to eliminate signal reflections etc. I would then make a test harness consisting of a PPU chip, the ability to read and write registers (either with the rest of the SNES, or some other microcontroller, whichever is easier) and the ability to capture complete frame outputs from the 'analog' outputs without error. I would then do the same with an emulator, and run code on both which pokes random registers at random (clock accurate) times, and whenever a difference in output appears, debug it.
  - near 7y ago
    > It seems the bus traffic won't tell you everything. Oh it absolutely won't, it's only the external state. It's just another important piece of the puzzle that I think would help us out a lot. Solving this would most likely require reverse-engineering the netlists from a decapped die scan. I've since (post article writing) learned that a German forum found a hidden test mode that outputs digital RGB values, but it has a lot of issues on various edge cases, so while it's not a perfect solution, it's definitely a huge help along the way if we can get a testing rig set up around an appropriately modded SNES.

wgjordan 7y ago

> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it might cause issues in another game. > We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles. Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. This sounds like standard software-engineering practice to me. I'm surprised that a test-harness like this doesn't already exist by now given the relatively mature state of SNES emulation. It seems like it would be an extremely helpful tool to benchmark accuracy of new or existing emulators, prevent regressions from advanced optimizations, etc. Automated testing could transform the 'blind guessing' approach on tweaking PPU timings from a game of 'whack a mole' to a structured, iterative linear algebra solver. Which could end up being quite a bit more feasible than acquiring 100x die scans of the chip and tracing the logic from the hardware directly.

xenadu02 7y ago
I don't know if SNES dev kits still exist but if they do they might prove useful. Such kits sometimes have additional debugging facilities because they're also used for validation (or they share some elements). Not sure if anyone was doing that sort of thing as early as SNES though.
kevingadd 7y ago
You can't just start it up and run it for 5 seconds, though. A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole package and then verify the whole stream against a known good. So you'd need to identify the key points in each game that are important, and replay those with save states and hope that the save states aren't unintentionally encoding corrupted data that came from an existing bug in the emulator you used to run them. Also, none of that automation you just did will work w/the actual hardware.
- theresistor 7y ago
  The input stream for an SNES isn't very complicated - just the state of a few buttons at every cycle, and the transition edges are quite sparse relative to the clock frequency. You could easily record human play sessions and replay them.

jimmaswell 7y ago

Why are test roms too slow if there are at most 41 cases to try? Is getting a game on an snes cart arduous?

chungy 7y ago
It can take several minutes to write out an EEPROM. It's enough to be frustrating.
- vardump 7y ago
  https://en.wikipedia.org/wiki/Flash_memory_emulator https://en.wikipedia.org/wiki/Flash_memory_emulator "A flash emulator or flash memory emulator is a tool that is used to temporarily replace flash memory or ROM chips in an embedded device for the purpose of debugging embedded software. Such tools contain Dual-ported RAM, one port of which is connected to a target system (i.e. system, that is being debugged), and second is connected to a host (i.e. PC, which runs debugger)."
near 7y ago
There are 52 registers comprising approximately 100 settings, there are probably close to twice that number if you include the internal register latches (some of which we know about, some of which we don't know exist yet.) The SNES PPUs are heavily based around combinatorial logic. Changing the timing of one variable can alter the timing needed by other variables. Think of each setting as potentially doubling the number of tests needed (at least when it comes to a blind, brute-force approach.) There are more combinations of settings and pixel generation patterns than atoms in the universe. The way we've gotten as far as we have is that not every combination needs to be tested. It is a ballpark estimate on my part, but I'd estimate us needing a few million tests to have a high degree of confidence our emulation is correct. If we had to make those tests into visual patterns on a screen that had to be checked by eye each time, it would be overwhelming even if we only needed a few thousand tests made. I can test code on a live SNES extremely easily with my 21fx board ( https://github.com/defparam/21FX https://github.com/defparam/21FX ), but it's still far too much to do this by hand.
- 7y ago

gregoryl 7y ago

What's the barrier to getting the chip decapped and scanned at the needed resolution?

near 7y ago
Really, just someone with the skill and interest in doing it. Money's no issue unless they want something crazy like $10,000+.
- zamadatix 7y ago
  John D McMaster seems to have previously done 100x scans for some chips if I'm reading the site right e.g. the latest has a link for 20x and 100x https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/ https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/. Is it as simple as seeing if he'll rescan the PPUs at 100x or are these not the quality that I was thinking?
  - near 7y ago
    He has a tremendous backlog of work to do at the moment. I would prefer to not bother him unless it was a last resort, but that is indeed an option if my breadboard idea does not work out and no others are up to the task of decapping the PPUs.

peter_d_sherman 7y ago

Excerpt: "If we could recruit a talented electrical engineer, I believe that a custom PPU breakout board could be designed that would aid us substantially in reverse engineering..." Perhaps Andrew "Bunnie" Huang -- would be someone to go to, for this?: https://en.wikipedia.org/wiki/Andrew_Huang_(hacker) https://en.wikipedia.org/wiki/Andrew_Huang_(hacker)