4 ms·

> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it migh

> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it might cause issues in another game. > We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles. Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. This sounds like standard software-engineering practice to me. I'm surprised that a test-harness like this doesn't already exist by now given the relatively mature state of SNES emulation. It seems like it would be an extremely helpful tool to benchmark accuracy of new or existing emulators, prevent regressions from advanced optimizations, etc. Automated testing could transform the 'blind guessing' approach on tweaking PPU timings from a game of 'whack a mole' to a structured, iterative linear algebra solver. Which could end up being quite a bit more feasible than acquiring 100x die scans of the chip and tracing the logic from the hardware directly.

xenadu02 7y ago
I don't know if SNES dev kits still exist but if they do they might prove useful. Such kits sometimes have additional debugging facilities because they're also used for validation (or they share some elements). Not sure if anyone was doing that sort of thing as early as SNES though.
kevingadd 7y ago
You can't just start it up and run it for 5 seconds, though. A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole package and then verify the whole stream against a known good. So you'd need to identify the key points in each game that are important, and replay those with save states and hope that the save states aren't unintentionally encoding corrupted data that came from an existing bug in the emulator you used to run them. Also, none of that automation you just did will work w/the actual hardware.
- theresistor 7y ago
  The input stream for an SNES isn't very complicated - just the state of a few buttons at every cycle, and the transition edges are quite sparse relative to the clock frequency. You could easily record human play sessions and replay them.
  - _qwfv 7y ago
    There's already a community of people that does this called tool assisted speed runs (or TAS). These players already record their inputs for replay and are comfortable tuning them to consistently hit edge cases. I wonder how much of a lift it would be to take a bunch of TASes and turn them into regression tests...
  - near 7y ago
    The SNES output is non-deterministic across runs. Not only is there unitialized RAM and I/O registers, and some analog effects, the really big elephant in the room is that the system has two oscillators. A ~21MHz CPU/PPU crystal oscillator, and a ~24MHz SMP/DSP ceramic oscillator. Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.) Every SNES game has sound routines that synchronize the CPU to the SMP. It wouldn't be possible to make a literal 1:1 play log unless you a) ran a custom register and memory initialization at system startup, and b) replaced the two oscillators with a much faster single oscillator and then used a clock dividier to drive both the CPU and APU off of it. (You can TAS certain SNES games anyway, of course. It really depends on how the game is programmed to react when the exact CPU<>SMP communications change. If it seeds a random number generator based around the PPU H/V counters that are polled after a CPU<>SMP sync for example, forget about it.)
    - deleted 7y ago
      [deleted]
    - vardump 7y ago
      > Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.) Maybe it'd be possible to modify some boards to use a CPLD (or an FPGA) for synthesizing those clocks from a common source. This could eliminate most (all?) uncertainty.
nonbirithm 7y ago
> Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. Dolphin (the GameCube/Wii emulator) already does this, actually. They make dumps of various states of the graphics pipeline across multiple revisions of the source and generate image-based diffs of the results. https://fifoci.dolphin-emu.org/about/ https://fifoci.dolphin-emu.org/about/ Here's an example. https://fifoci.dolphin-emu.org/compare/5108854-5102201/ https://fifoci.dolphin-emu.org/compare/5108854-5102201/
near 7y ago
Screwtape built exactly such a regression-testing harness for bsnes already. We are still at a point where minor changes can result in dozens of "change detected" triggers, of which we are not sure how to determine whether they represent corrections or regressions (it's usually not at all obvious.) I've also been weary of building a large "accuracy testing suite" for the SNES, on the fear that if I come to the wrong conclusion on something, others will emulate that inaccuracy and won't question it because of deference to my code. Tests also have the unfortunate side-effect of being gamified (example: http://tasvideos.org/EmulatorResources/NESAccuracyTests.html http://tasvideos.org/EmulatorResources/NESAccuracyTests.html), which is a very bad thing. Failing a test because we don't know the answer is infinitely better than passing the test using am unverified guess. But when people reduce a comprehensive and complex set of tests into a percentage score, you get a very skewed picture of what actually matters. (think of how browsers jumped through hoops to pass the Acid CSS tests back in the day.) It is however something that's important, and hopefully I'll have the time one day to work on this more.
- sitkack 7y ago
  I don't know enough about SNES hardware or the level of in system debug, but could one generate a hardware test jig that was able to record player traces that could be re-run in the emulator and have the screen grabs compared? Given the system state, random number generator and the series of up/down/a/b one should be able to recreate the same game evolution and video output from the emulator? Then you can form a reference set of captures, real hardware and the emulator. Screen to screen emulation with diffs could then be an accurate tool to measure against actual hardware and the accuracy gap between the two. No? Controlling the random number generator sounds like it might be key. And some player traces might be really fragile in the input timing search space.
mlatu 7y ago
> more feasible than acquiring 100x die scans of the chip Last time I had to get SEM micrographs in resolutions between 2K and 10K, we paid 475€ for 2 hours of work. While I wouldn't want to pay this out of my pocket and gift it to the world, I don't see how it's less feasible to make better die scans than to build the testing equipment (PPU breakoutboard) needed for automated testing. Especially if the current owner of the already decapped dies were willing to lend them to someone else willing to do the footwork. I think it should be easy enough to find 19 people willing to donate 25€ for this kinda effort. On the other hand, the dies would have to be plasma coated with Gold/Platinum, which might be bad, I'm not an expert. So, perhaps you were refering to this when you said higher-res scans are less feasable than making automatic testing happen. ¯\_(ツ)_/¯

xenadu02 7y ago

I don't know if SNES dev kits still exist but if they do they might prove useful. Such kits sometimes have additional debugging facilities because they're also used for validation (or they share some elements). Not sure if anyone was doing that sort of thing as early as SNES though.

kevingadd 7y ago

You can't just start it up and run it for 5 seconds, though. A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole package and then verify the whole stream against a known good. So you'd need to identify the key points in each game that are important, and replay those with save states and hope that the save states aren't unintentionally encoding corrupted data that came from an existing bug in the emulator you used to run them. Also, none of that automation you just did will work w/the actual hardware.

theresistor 7y ago
The input stream for an SNES isn't very complicated - just the state of a few buttons at every cycle, and the transition edges are quite sparse relative to the clock frequency. You could easily record human play sessions and replay them.
- _qwfv 7y ago
  There's already a community of people that does this called tool assisted speed runs (or TAS). These players already record their inputs for replay and are comfortable tuning them to consistently hit edge cases. I wonder how much of a lift it would be to take a bunch of TASes and turn them into regression tests...
- near 7y ago
  The SNES output is non-deterministic across runs. Not only is there unitialized RAM and I/O registers, and some analog effects, the really big elephant in the room is that the system has two oscillators. A ~21MHz CPU/PPU crystal oscillator, and a ~24MHz SMP/DSP ceramic oscillator. Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.) Every SNES game has sound routines that synchronize the CPU to the SMP. It wouldn't be possible to make a literal 1:1 play log unless you a) ran a custom register and memory initialization at system startup, and b) replaced the two oscillators with a much faster single oscillator and then used a clock dividier to drive both the CPU and APU off of it. (You can TAS certain SNES games anyway, of course. It really depends on how the game is programmed to react when the exact CPU<>SMP communications change. If it seeds a random number generator based around the PPU H/V counters that are polled after a CPU<>SMP sync for example, forget about it.)
  - deleted 7y ago
    [deleted]
  - vardump 7y ago
    > Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.) Maybe it'd be possible to modify some boards to use a CPLD (or an FPGA) for synthesizing those clocks from a common source. This could eliminate most (all?) uncertainty.

nonbirithm 7y ago

> Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. Dolphin (the GameCube/Wii emulator) already does this, actually. They make dumps of various states of the graphics pipeline across multiple revisions of the source and generate image-based diffs of the results. https://fifoci.dolphin-emu.org/about/ https://fifoci.dolphin-emu.org/about/ Here's an example. https://fifoci.dolphin-emu.org/compare/5108854-5102201/ https://fifoci.dolphin-emu.org/compare/5108854-5102201/

near 7y ago

Screwtape built exactly such a regression-testing harness for bsnes already. We are still at a point where minor changes can result in dozens of "change detected" triggers, of which we are not sure how to determine whether they represent corrections or regressions (it's usually not at all obvious.) I've also been weary of building a large "accuracy testing suite" for the SNES, on the fear that if I come to the wrong conclusion on something, others will emulate that inaccuracy and won't question it because of deference to my code. Tests also have the unfortunate side-effect of being gamified (example: http://tasvideos.org/EmulatorResources/NESAccuracyTests.html http://tasvideos.org/EmulatorResources/NESAccuracyTests.html), which is a very bad thing. Failing a test because we don't know the answer is infinitely better than passing the test using am unverified guess. But when people reduce a comprehensive and complex set of tests into a percentage score, you get a very skewed picture of what actually matters. (think of how browsers jumped through hoops to pass the Acid CSS tests back in the day.) It is however something that's important, and hopefully I'll have the time one day to work on this more.

sitkack 7y ago
I don't know enough about SNES hardware or the level of in system debug, but could one generate a hardware test jig that was able to record player traces that could be re-run in the emulator and have the screen grabs compared? Given the system state, random number generator and the series of up/down/a/b one should be able to recreate the same game evolution and video output from the emulator? Then you can form a reference set of captures, real hardware and the emulator. Screen to screen emulation with diffs could then be an accurate tool to measure against actual hardware and the accuracy gap between the two. No? Controlling the random number generator sounds like it might be key. And some player traces might be really fragile in the input timing search space.

mlatu 7y ago

> more feasible than acquiring 100x die scans of the chip Last time I had to get SEM micrographs in resolutions between 2K and 10K, we paid 475€ for 2 hours of work. While I wouldn't want to pay this out of my pocket and gift it to the world, I don't see how it's less feasible to make better die scans than to build the testing equipment (PPU breakoutboard) needed for automated testing. Especially if the current owner of the already decapped dies were willing to lend them to someone else willing to do the footwork. I think it should be easy enough to find 19 people willing to donate 25€ for this kinda effort. On the other hand, the dies would have to be plasma coated with Gold/Platinum, which might be bad, I'm not an expert. So, perhaps you were refering to this when you said higher-res scans are less feasable than making automatic testing happen. ¯\_(ツ)_/¯