7 ms·

Bsnes has emulated every SNES DSP

212 points by pkmays 15y ago

shin_lao 15y ago
Congratulations on this very difficult project. I however think it is a prime example of great teamwork rather than open source. Open source is a mean, not an end. We tend to forget that.
- nitrogen 15y ago
  Open source is a mean, not an end. We tend to forget that. Open Source is both a means and an end. It can be a means to collaboration, but the ongoing sharing of knowledge is itself a virtue.
  - shin_lao 15y ago
    Open source does not imply proper documentation or easy knowledge extraction. I prefer to read a book about why and how a software is made than to read the source code.
    - byuu 15y ago
      Documentation is great in that it can elaborate in much more detail, especially with actual pictures and graphs. However, unlike source code, documentation cannot be machine tested for correctness. Honestly, they're both important, and for different reasons. In this instance, Cydrak and I were able to pass revisions of the ARM core back and forth to fully debug it.
    - nitrogen 15y ago
      No doubt books are very useful learning tools. However, depending on the code, source code can be equally if not more useful. For example, years ago I learned a lot about C by reading (and later writing) Linux kernel source code. It's also true that code being open source doesn't mean the code is useful, but a book being published doesn't mean the material is presented well, either. Maybe it's just my pro-open source bias, but it feels like you've got an axe to grind against open source. If so, would you mind sharing why (maybe you know something I don't)?
      - shin_lao 15y ago
        Absolutely no axe to grind I've contributed to open source projects and will certainly continue to. What I don't like is that people overstate the importance of open source and think it's the solution to everything. Open source is great for some projects, useless and even counterproductive for others.
        byuu 15y ago
        I can agree with that. I'm sure you weren't implying me in particular, but to clarify, I certainly don't think open source is the solution to everything. I just find it particularly important in the field of emulation. This is much like a science, with constant refinements to theories. Since we're running against the clock to preserve these systems before they die, I assign much more value to open source emulators than I do to other applications (for instance, I'm happy to use the closed source nVidia driver on Linux.)
        shin_lao 15y ago
        I think open source makes a lot of sense for research. I'm actually surprised researchers don't publish that much source code.
kevingadd 15y ago
It blows my mind that there's a SNES game out there that shipped with an ARMv3 core inside it. I wonder how much that ate into their margins?
- DiabloD3 15y ago
  Man, look at the games that shipped with an SA-1. Its an independent (not slaved to the system CPU) 10mhz C5C816 with its own scratch RAM and can DMA to main system RAM; the CPU of the SNES itself is a 3.58 mhz C5C816 (technically a 5A22, Richoh's clone of the 65C816), and the scratch RAM (something like 256kb of it) is faster than system RAM. Remember how Super Mario RPG was really expensive when it came out? Thats why. Quite a few other games used this chip, but it seems to be all Japan only except for Kirby Super Star and Kirby's Dream Land 3, neither of which were particularly exceptional games.
- jcromartie 15y ago
  These games were much more expensive than games today. Accounting for inflation, SNES games originally cost about $100 in today's dollars. I am sure that's enough to cover the cost of an ARM6.
  - tibbon 15y ago
    And yet for some reason gamers complain that games of 2012 are too expensive. $60 of 2012USD really isn't that much compared to $50 of 1992USD. Add in the fact that you can get some excellent an amazing games on mobile platforms now for a dollar, and gaming is cheaper than ever.
    - codezero 15y ago
      Difference was that back then a game from start to end lasted a whole lot more than 4-8 hours. Yes I know lots of games now do as well, but lots of the most popular ones don't.
      - bonch 15y ago
pkmays 15y ago
Can any MAME/MESS hackers tell us if we'll be seeing accuracy improvements now that this information is available. MESS is my console emulator of choice.
- kstenerud 15y ago
  I wrote the g65816 and spc700 cores used in MAME/MESS, but I haven't been involved in the emu scene for almost a decade now. I could take a look and see if there are any quick things I could graft in, but my time is so limited these days...
- simcop2387 15y ago
  I wouldn't be too surprised if the details from all of this get pulled into MAME/MESS. It will take some rewriting though as I believe BSNES is written using a number of C++ features (I seem to recall it even needing some C++11 stuff from when I last played with the source). The CPU cores themselves I'd imagine would be fairly easy to port over but I'm not sure how tied in everything is for timing and everything else in there.
  - byuu 15y ago
    The main magic I do for RISC CPUs is allowing a write callback binding to the registers (overloading operator=) This allows us to handle pipeline invalidation on writes to the PC register, without having to special case the hell out of every single instruction that can modify it.
    - simcop2387 15y ago
      Ah seems my memory was bad there. I must have had it confused with another emulator, however that sounds while it wouldn't be trivial to rewrite those sections it shouldn't be very difficult to do so either, just very very tedious. I think other than that there could only be a licensing issue, and I seem to recall that bsnes had a fairly liberal license that shouldn't run into any problems. Though obviously you'll know better than me byuu :) In that case it's all about time and who has it that'll determine when/if it'll happen
SoftwarePatent 15y ago
"Yet once again, we were stuck looking at a binary blob. And this time, we had no real HLE code to go off of. Luckily, Cydrak recognized the ISA just from looking at the binary: it was an ARMv3 CPU!" From the perspective of a software guy, this sounds very impressive. Can some hardware guy chime in and tell us just how impressive it is to recognize a CPU from looking at a binary dump?
- sern 15y ago
  Not particularly - ARM instruction encoding is rather distinct. Repeating "always" values of the ARM condition field really stand out.
- wladimir 15y ago
  The instruction size is the first thing that is apparent. By looking at the pattern you can distinguish between fixed-sized, or variable-sized instructions, telling a lot about the architecture. ARM either has 16 bit (thumb mode) or 32 bit (normal) fixed-size instructions. Also for every ISA there are some instructions that are very common, for example x86 has very-recognizable NOPs (0x90). Having said that, seeing it is ARMv3 code specifically in one glance is pretty impressive.
  - rcfox 15y ago
    > Having said that, seeing it is ARMv3 code specifically in one glance is pretty impressive. You seem to have added some hyperbole. There was no mention of "in one glance". Also, it's likely that he recognized that it was ARMvSomething (which is relatively easy) and then did some comparisons to narrow it down.
- ootachi 15y ago
  ARM is by far the easiest ISA to do this with: since the vast majority of instructions begins with an "always" condition code, and the condition code is a nibble (0xe), you can pick out ARM instructions easily.
funkah 15y ago
Great stuff! I found myself missing a bit of background, though, which you can find here: http://byuu.org/articles/emulation/decap http://byuu.org/articles/emulation/decap
mistercow 15y ago
This is fascinating, but I'm not really clear on what role the electron microscope played. Did it just allow them to see where they needed to modify the chips to allow them to dump their ROM?
- Leynos 15y ago
  That's correct. The data was also extracted using probes under a microsope in this case. From Dr Decapitator's page: "The data was extracted by laser-cutting the top two ROM bits from the actual CPU logic (so it would read all opcodes as OP(00) or LD(11) opcodes depending on whether lines were pulled high or low, and neither OP nor LD can branch address), then microprobing, 8 bits at a time, the ROM outputs from before the cut. The program counter was also monitored to ensure that the data read started at 0×000 and didn’t jump anywhere." (http://decap.mameworld.info/2010/12/16/snes-dsp1b/ http://decap.mameworld.info/2010/12/16/snes-dsp1b/) You can read more about chip decapping here: http://guru.mameworld.info/decap/index.html http://guru.mameworld.info/decap/index.html And here: http://www.defcon.org/images/defcon-12/dc-12-presentations/Grand/dc-12-grand.ppt http://www.defcon.org/images/defcon-12/dc-12-presentations/G...
  - byuu 15y ago
    Oh, good. I was intentionally vague there, as I wasn't sure how much was public knowledge. Yes, basically we bypassed the JP (jump) and RT (return) opcodes, so that the program counter always stepped by one. Externally clock the CPU to move to the next instruction, and keep logging the data lines prior to the OP/LD cuts.
feralchimp 15y ago
Add one to the list of "people who should never pay for another beer in their lifetime": In order to extract these programs from most of the DSPs, we had to decap the chips, scan them in with an electron microscope, and 'tweak' the processors to allow us to dump their protected program ROMs. As you can imagine, this kind of work costs a substantial amount of money. Professional firms that do this can charge upwards of $100,000 per processor for this kind of work. Thankfully, Lord Nightmare from the MAME/MESS project was in contact with a savant named Dr. Decapitator, with the necessary knowledge and hardware to make this possible. Even better, he was willing to do it just for the cost of the donor cartridges and supplies. This worked out to $250 per coprocessor.
- plaes 15y ago
  I hate to state the obvious, but the quote you gave actually talks about 3 people.
  - feralchimp 15y ago
    FWIW, I meant Dr. Decapitator. But really, take your pick.
bri3d 15y ago
I love how game consoles are always providing exciting hacks - from the variety of CPUs and DSPs used in SNES cartridges to the brute-forced SHA collision exploited to hack the PSP. For what it's worth, this is almost identical to the process initially used to hack satellite TV smartcards, except it seems that these devices had a lot less physical tamper protection (and can't be reprogrammed with a new ROM over-the-air). I think this kind of console hack is a dying art - as process shrinks and most consoles move closer and closer to commodity hardware and off-the-shelf software, LLE and chip decapping become less and less appealing and software exploits are the name of the game. Absolutely awesome, though - this is the kind of story I come to HN to read.
scw 15y ago
I found this Wikipedia article helpful in explaining the roles of the DSPs in the games that used them: http://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_chips http://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_c...

212 points by pkmays 15y ago

Congratulations on this very difficult project. I however think it is a prime example of great teamwork rather than open source. Open source is a mean, not an end. We tend to forget that.

nitrogen 15y ago
Open source is a mean, not an end. We tend to forget that. Open Source is both a means and an end. It can be a means to collaboration, but the ongoing sharing of knowledge is itself a virtue.
- shin_lao 15y ago
  Open source does not imply proper documentation or easy knowledge extraction. I prefer to read a book about why and how a software is made than to read the source code.
  - byuu 15y ago
    Documentation is great in that it can elaborate in much more detail, especially with actual pictures and graphs. However, unlike source code, documentation cannot be machine tested for correctness. Honestly, they're both important, and for different reasons. In this instance, Cydrak and I were able to pass revisions of the ARM core back and forth to fully debug it.
  - nitrogen 15y ago
    No doubt books are very useful learning tools. However, depending on the code, source code can be equally if not more useful. For example, years ago I learned a lot about C by reading (and later writing) Linux kernel source code. It's also true that code being open source doesn't mean the code is useful, but a book being published doesn't mean the material is presented well, either. Maybe it's just my pro-open source bias, but it feels like you've got an axe to grind against open source. If so, would you mind sharing why (maybe you know something I don't)?
    - shin_lao 15y ago
      Absolutely no axe to grind I've contributed to open source projects and will certainly continue to. What I don't like is that people overstate the importance of open source and think it's the solution to everything. Open source is great for some projects, useless and even counterproductive for others.
      - byuu 15y ago
        I can agree with that. I'm sure you weren't implying me in particular, but to clarify, I certainly don't think open source is the solution to everything. I just find it particularly important in the field of emulation. This is much like a science, with constant refinements to theories. Since we're running against the clock to preserve these systems before they die, I assign much more value to open source emulators than I do to other applications (for instance, I'm happy to use the closed source nVidia driver on Linux.)
        shin_lao 15y ago
        I think open source makes a lot of sense for research. I'm actually surprised researchers don't publish that much source code.

kevingadd 15y ago

It blows my mind that there's a SNES game out there that shipped with an ARMv3 core inside it. I wonder how much that ate into their margins?

DiabloD3 15y ago
Man, look at the games that shipped with an SA-1. Its an independent (not slaved to the system CPU) 10mhz C5C816 with its own scratch RAM and can DMA to main system RAM; the CPU of the SNES itself is a 3.58 mhz C5C816 (technically a 5A22, Richoh's clone of the 65C816), and the scratch RAM (something like 256kb of it) is faster than system RAM. Remember how Super Mario RPG was really expensive when it came out? Thats why. Quite a few other games used this chip, but it seems to be all Japan only except for Kirby Super Star and Kirby's Dream Land 3, neither of which were particularly exceptional games.
jcromartie 15y ago
These games were much more expensive than games today. Accounting for inflation, SNES games originally cost about $100 in today's dollars. I am sure that's enough to cover the cost of an ARM6.
- tibbon 15y ago
  And yet for some reason gamers complain that games of 2012 are too expensive. $60 of 2012USD really isn't that much compared to $50 of 1992USD. Add in the fact that you can get some excellent an amazing games on mobile platforms now for a dollar, and gaming is cheaper than ever.
  - codezero 15y ago
    Difference was that back then a game from start to end lasted a whole lot more than 4-8 hours. Yes I know lots of games now do as well, but lots of the most popular ones don't.
    - bonch 15y ago

pkmays 15y ago

Can any MAME/MESS hackers tell us if we'll be seeing accuracy improvements now that this information is available. MESS is my console emulator of choice.

kstenerud 15y ago
I wrote the g65816 and spc700 cores used in MAME/MESS, but I haven't been involved in the emu scene for almost a decade now. I could take a look and see if there are any quick things I could graft in, but my time is so limited these days...
simcop2387 15y ago
I wouldn't be too surprised if the details from all of this get pulled into MAME/MESS. It will take some rewriting though as I believe BSNES is written using a number of C++ features (I seem to recall it even needing some C++11 stuff from when I last played with the source). The CPU cores themselves I'd imagine would be fairly easy to port over but I'm not sure how tied in everything is for timing and everything else in there.
- byuu 15y ago
  The main magic I do for RISC CPUs is allowing a write callback binding to the registers (overloading operator=) This allows us to handle pipeline invalidation on writes to the PC register, without having to special case the hell out of every single instruction that can modify it.
  - simcop2387 15y ago
    Ah seems my memory was bad there. I must have had it confused with another emulator, however that sounds while it wouldn't be trivial to rewrite those sections it shouldn't be very difficult to do so either, just very very tedious. I think other than that there could only be a licensing issue, and I seem to recall that bsnes had a fairly liberal license that shouldn't run into any problems. Though obviously you'll know better than me byuu :) In that case it's all about time and who has it that'll determine when/if it'll happen

SoftwarePatent 15y ago

"Yet once again, we were stuck looking at a binary blob. And this time, we had no real HLE code to go off of. Luckily, Cydrak recognized the ISA just from looking at the binary: it was an ARMv3 CPU!" From the perspective of a software guy, this sounds very impressive. Can some hardware guy chime in and tell us just how impressive it is to recognize a CPU from looking at a binary dump?

sern 15y ago
Not particularly - ARM instruction encoding is rather distinct. Repeating "always" values of the ARM condition field really stand out.
wladimir 15y ago
The instruction size is the first thing that is apparent. By looking at the pattern you can distinguish between fixed-sized, or variable-sized instructions, telling a lot about the architecture. ARM either has 16 bit (thumb mode) or 32 bit (normal) fixed-size instructions. Also for every ISA there are some instructions that are very common, for example x86 has very-recognizable NOPs (0x90). Having said that, seeing it is ARMv3 code specifically in one glance is pretty impressive.
- rcfox 15y ago
  > Having said that, seeing it is ARMv3 code specifically in one glance is pretty impressive. You seem to have added some hyperbole. There was no mention of "in one glance". Also, it's likely that he recognized that it was ARMvSomething (which is relatively easy) and then did some comparisons to narrow it down.
ootachi 15y ago
ARM is by far the easiest ISA to do this with: since the vast majority of instructions begins with an "always" condition code, and the condition code is a nibble (0xe), you can pick out ARM instructions easily.

funkah 15y ago

Great stuff! I found myself missing a bit of background, though, which you can find here: http://byuu.org/articles/emulation/decap http://byuu.org/articles/emulation/decap

mistercow 15y ago

This is fascinating, but I'm not really clear on what role the electron microscope played. Did it just allow them to see where they needed to modify the chips to allow them to dump their ROM?

Leynos 15y ago
That's correct. The data was also extracted using probes under a microsope in this case. From Dr Decapitator's page: "The data was extracted by laser-cutting the top two ROM bits from the actual CPU logic (so it would read all opcodes as OP(00) or LD(11) opcodes depending on whether lines were pulled high or low, and neither OP nor LD can branch address), then microprobing, 8 bits at a time, the ROM outputs from before the cut. The program counter was also monitored to ensure that the data read started at 0×000 and didn’t jump anywhere." (http://decap.mameworld.info/2010/12/16/snes-dsp1b/ http://decap.mameworld.info/2010/12/16/snes-dsp1b/) You can read more about chip decapping here: http://guru.mameworld.info/decap/index.html http://guru.mameworld.info/decap/index.html And here: http://www.defcon.org/images/defcon-12/dc-12-presentations/Grand/dc-12-grand.ppt http://www.defcon.org/images/defcon-12/dc-12-presentations/G...
- byuu 15y ago
  Oh, good. I was intentionally vague there, as I wasn't sure how much was public knowledge. Yes, basically we bypassed the JP (jump) and RT (return) opcodes, so that the program counter always stepped by one. Externally clock the CPU to move to the next instruction, and keep logging the data lines prior to the OP/LD cuts.

feralchimp 15y ago

Add one to the list of "people who should never pay for another beer in their lifetime": In order to extract these programs from most of the DSPs, we had to decap the chips, scan them in with an electron microscope, and 'tweak' the processors to allow us to dump their protected program ROMs. As you can imagine, this kind of work costs a substantial amount of money. Professional firms that do this can charge upwards of $100,000 per processor for this kind of work. Thankfully, Lord Nightmare from the MAME/MESS project was in contact with a savant named Dr. Decapitator, with the necessary knowledge and hardware to make this possible. Even better, he was willing to do it just for the cost of the donor cartridges and supplies. This worked out to $250 per coprocessor.

plaes 15y ago
I hate to state the obvious, but the quote you gave actually talks about 3 people.
- feralchimp 15y ago
  FWIW, I meant Dr. Decapitator. But really, take your pick.

bri3d 15y ago

I love how game consoles are always providing exciting hacks - from the variety of CPUs and DSPs used in SNES cartridges to the brute-forced SHA collision exploited to hack the PSP. For what it's worth, this is almost identical to the process initially used to hack satellite TV smartcards, except it seems that these devices had a lot less physical tamper protection (and can't be reprogrammed with a new ROM over-the-air). I think this kind of console hack is a dying art - as process shrinks and most consoles move closer and closer to commodity hardware and off-the-shelf software, LLE and chip decapping become less and less appealing and software exploits are the name of the game. Absolutely awesome, though - this is the kind of story I come to HN to read.

scw 15y ago

I found this Wikipedia article helpful in explaining the roles of the DSPs in the games that used them: http://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_chips http://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_c...