7 ms·

Significant progress made on Xbox 360 recompilation

150 points by tetrisgm 7mo ago

andrewstuart 7mo ago
What is Xbox 360 recompilation?
- fwip 7mo ago
  You take a binary that's intended to run on the Xbox 360, and emit a new binary that runs on a modern x86 computer.
- MisterTea 7mo ago
  You take an Xbox game designed to run on an Xbox 360, a 64 bit PowerPC system and decompile its binaries back into source code. You now have the ability to modify the game as well as port it to other systems and architectures such as Windows on X86_64 or Linux on ARM64.
  - kaladin-jasnah 7mo ago
    Seeing this [1], I thought it was something related to taking assembly instructions in the original code, emitting C statements that match the instruction, and then compiling that C code. [1] https://github.com/N64Recomp/N64Recomp https://github.com/N64Recomp/N64Recomp
    - MisterTea 7mo ago
      Ah, you are right. I haven't looked at the details of the recomp efforts.
    - bri3d 7mo ago
      Your idea is much more accurate; see my sibling comment. It's basically using C or C++ as an intermediate representation for machine code, rather than trying to recreate the game's higher-order logic/structure or source code.
      - krackers 7mo ago
        This vaguely reminds me of futamura projections. Normally with futamura's first projection the input is source code, and you partial-evaluate that source against an interpreter for that source, resulting in a "compiled" binary that has the logic of the source inlined into the interpreter (and hopefully optimized). This is similar to what Truffle does I believe, where you have an interpreter (written in java) and then during runtime Truffle "JIT" optimizes the interpreted program's AST against the interpreter's logic. All of this can be considered a specialization of the JVM running an interpreter interpreting your program. In this case with "recompilation" you have a binary made to run on certain hardware. You then take an emulator of the hardware (registers, PC, etc.) and then "partial evaluate" the binary against the hardware emulator, producing a new program that contains a software emulation of just that specific binary. So while you're still conceptually emulating the underlying hardware, you both avoid the instruction dispatch overhead at runtime (it's statically compiled in) and also benefit from the optimization passes of modern compilers.
  - bri3d 7mo ago
    It's more nuanced than that; the approach you're describing is usually called "decompilation." The difference is how far one goes in hoisting the "source code;" in this "recompliation" approach the source code, while C++, is basically an IR (intermediate representation) between the original game's assembly and a host platform, and the hardware itself is emulated (for example, the original architecture's CPU registers are represented as variables in the host architecture's memory). The machine code is translated to C++ using a custom tool. In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms. The product is something that attempts to resemble the original game's source code. Of course, they lie on a continuum and both approaches can be mixed, but, while they both involve C++ in the middle, the process is starkly different. Recompilation is much more copyright-friendly, because in many implementations only the modifications are distributed and the original binary is translated by the end user (who owns the software/a license to it), whereas decompilation produces an artifact (source code) which is a derivative work encumbered by the original software's license and generally should not be distributed.
    - kombine 7mo ago
      > In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms There definitely is a lot of scope to apply LLMs here
      - iscisjvije 7mo ago
        no one absolutely no one not a single soul on this Earth LLM nut: OMG LLM!!!!!!! Can’t you just drop it, please?
giancarlostoro 7mo ago
The blog post reminds me of similar efforts with Shockwave. There's people building decompilers, and runtimes, and if you join enough Discords, you will notice the people in them are cross-contaminating between communities if you will, they share insights with one another, in their efforts towards specific goals. They're hyper focused on making one game or another come back. There's Habbo Hotel, and Coke Studios, as well as other games. The Coke Studios effort is interesting because there were no "private servers" developed at the time, unlike Habbo which had many, and there are Shockwave Xtras that no open runtime supports currently. There's several attempts at a full runtime as well, that run in-browser. Projector Rays (decompiler) really was the biggest release to date, and recently people have been really hacking at it, to some extent AI has helped to reverse engineer bytecode far as I can tell. For anyone curious, one of the runtimes is called DirPlayer: https://github.com/igorlira/dirplayer-rs https://github.com/igorlira/dirplayer-rs
- xnx 7mo ago
  > cross-contaminating between communities if you will The positive term for this is cross-polinating
- YesBox 7mo ago
  Where can I boot up Coke Studios? Enjoyed that game as a kid, including making music (which was so friggen cool! They had tons of samples you could just drag and drop into a timeline). Edit: fun memory. I figured out how scamming works at some point. IIRC (and I may not) I set up a minigame in my studio, and contestants had to give me their furniture to participate in a game. Since furniture could stack weirdly, people made all sorts of crazy mazes that were actually quite difficult to navigate due to the fixed isometric perspective. Each stage required handing over a more expensive item than the last. At some point someone handed over a super rare/expensive piece of furniture. Not gonna erase that. ACTUALLY I think I figured out how to clone items due to a bug. Or mine infinite DBs. Used that as cover to get someone to give me their item. Was a long time ago. Anyways, a user handed me their super rare item and I then gave them a stern warning on how this is a common scam and I just stole their item(s). Then I handed everything back over to them and they profusely thanked me. lol
- 7mo ago
triangleman 7mo ago
What's the latest on homebrewing on the actual Xbox 360 these days?
- opan 7mo ago
  You can play game backups, use unofficial HDDs at bigger sizes, send over new games with FTP, use third party and DIY controllers like GuliKit's offerings or a Flatbox leverless arcade controller. You can play a modded version of Rock Band 3 with all the songs from all the games, using a Wii Guitar Hero controller with a Pi Pico-based adapter in place of the Wiimote. Any game you installed from disc for faster load times becomes playable without the disc in automatically once you get Aurora running (this is pretty cool, on some other modded consoles like PS4 you'd have to reinstall the games another way). As far as homebrew in the sense of people making their own games and applications, I don't know that there's much going on. Maybe you can find a controller tester app. Sadly the 360 scene has a lot of proprietary stuff going on, and uses some official dev SDKs, IIRC. Where 3DS and Switch modders have everything on GitHub and freely licensed, 360 has a lot of binary-only releases on random forums and such. It would probably be a major effort to change this and involve rewriting stuff people already know and love. I did the the RGH 3 mod to half a dozen or so 360s a few years back for myself and friends. If you just wanna play games with modern comforts, things are in a pretty good state. We don't need to worry about the original disc drives, HDDs, or controllers wearing out and making the console useless, we can replace or avoid them all. The wireless card is also easily replaceable, a friend of mine had one that seemed non-functional after he spilled some Mtn Dew all over his 360 back in the day. Everything else seemed good after a clean but I just got him a new wireless card off eBay and swapped that out. If anyone wants to get into this and doesn't already have a 360, the S models are easiest to mod and most reliable (no RRoD). Some are Trinity and some are Corona motherboards inside, but both are hackable. Corona may need a post-fix adapter, but they're cheap and easy to install. Just avoid the E series. For modding tutorials, MrMario2011 on YT is great. There is also a software-only exploit that's a bit worse but more accessible. It came out after I already did all my modding, so I haven't got experience with it.
asdff 7mo ago
Crazy that it has taken so long. I understand these are small teams trudging off in the dark but you could have imagined there would have been more eyes on this and hands tinkering in the time when the Xbox 360 was being sold than today. Right at that same time in history the iPhone was getting cracked basically every iOS release, sometimes by teenagers. Seems like there were a ton of hackers around back then.
- wmf 7mo ago
  Most people just want emulation which was solved a while ago. Recompilation is mostly for modders which is a very small community.
  - tetrisgm 7mo ago
    The whole decomp and recomp scene has really started popping off this past year in the emulation community. The N64 recomp tool was the first one to really get interest going.
Venn1 7mo ago
These projects are really neat. Last week, I was able to build (and play) the Xbox 360 release of Sonic Unleashed on a couple of ARM SBCs using Sonic Unleashed Recomp.
wesfenlon 7mo ago
Hey, that's my newsletter -- thanks for sharing! :D Some exciting stuff has already happened in the week since I wrote the interview. Here's Viva Pinata: Trouble in Paradise: https://www.youtube.com/watch?v=Kih42KlocII https://www.youtube.com/watch?v=Kih42KlocII
- tetrisgm 7mo ago
  Was playing that last night. So many great recomps lately. Ty for the newsletter
- testing22321 7mo ago
  How does the static recompiler handle self modifying code?
- tetrisgm 7mo ago
  btw any way to reach out to you (twitter, discord?). big fan of your blog. I'd say on par with Time Extension!
rendaw 7mo ago
Years ago I was searching for why emulators don't recompile to native bytecode and it seemed like the consensus at the time was that JIT etc was faster. I can't find it now, but IIRC I remember replies from maintainers saying it was a bad idea. What changed since then, since this seems to be a trend? Is this more for more modern systems that have more static code?
- tetrisgm 7mo ago
  It's a combination of things. The floodgates were opened when the N64 recompiler popped up a couple (?) of years back. Really got interest going. I also think tooling got better, and with the advent of AI engineering it made all of this a lot more accessible.
- CJefferson 7mo ago
  Static code. Also, very fine details of the machine because less important. When emulating a NES (for example), you really have to emulate every register, how registers change, and also weird effects like instructions that take longer to read or write values, as games rely on that stuff. Once you have modern systems where much of the code was originally C, it becomes less important to ensure every register has exactly the right values when a subroutine finishes in most cases, you can rely that (most) of the code follows standard calling conventions.
  - charcircuit 7mo ago
    You really don't. Most of the time the code will be doing normal things. So you can have other code that also does the same thing. >and also weird effects like instructions that take longer to read or write values The percentage of a code in the ROM that depends on such things is a small percent.
    - CJefferson 7mo ago
      The problem with NES emulation (having written a fairly bad one) is it can be hard to predict when you need to be careful, particularly in the presence of self-modifying code and interrupts. If it was easy to figure out, emulators wouldn't have to keep such careful track of this stuff either.
cosmic_cheese 7mo ago
Very nice. It’d be interesting if recomps of some games that were originally for both 360 and PC become the superior experience… Fable 3 and the whole Games for Windows Live nonsense the PC version had come to mind.
zxspectrum1982 7mo ago
I wonder whether this "recompilation" technology could be used to run Windows software on Linux, as an alternative/complementary technique to Wine. Back in the day (year 2000, until 10-15 years ago) we had Project Odin to dynamically translate Windows software to run on OS/2: https://github.com/netlabsorg/odin32 https://github.com/netlabsorg/odin32
- pezezin 7mo ago
  Windows and Linux run on the same x86 CPU architecture, you don't need to recompile anything. What WINE does is to translate/replace system calls and libraries.
  - zxspectrum1982 6mo ago
    I know how Wine works, thank you. I've used winelib in the past to port Windows software (that I had the source code of) to Linux myself. My point is the way wine works today is: WinAPI --> winelib --> Linux --> x86 I. e. winelib is reimplementing WineAPI on top of Linux. What if we could just decompile those Windows functions and recompile them on Linux, or even x86, directly? The workaround all of these console recompilation projects use is you must have the original game in order to have the binaries (the ones you theoretically decompile and recompile, but actually you take pre-decompiled-recompiled ones by someone else) and assets (graphics, sounds, etc). For Windows applications on Linux, we could do the same: bring your own Windows, then we can decompile and recompile.

150 points by tetrisgm 7mo ago

andrewstuart 7mo ago

What is Xbox 360 recompilation?

fwip 7mo ago
You take a binary that's intended to run on the Xbox 360, and emit a new binary that runs on a modern x86 computer.
MisterTea 7mo ago
You take an Xbox game designed to run on an Xbox 360, a 64 bit PowerPC system and decompile its binaries back into source code. You now have the ability to modify the game as well as port it to other systems and architectures such as Windows on X86_64 or Linux on ARM64.
- kaladin-jasnah 7mo ago
  Seeing this [1], I thought it was something related to taking assembly instructions in the original code, emitting C statements that match the instruction, and then compiling that C code. [1] https://github.com/N64Recomp/N64Recomp https://github.com/N64Recomp/N64Recomp
  - MisterTea 7mo ago
    Ah, you are right. I haven't looked at the details of the recomp efforts.
  - bri3d 7mo ago
    Your idea is much more accurate; see my sibling comment. It's basically using C or C++ as an intermediate representation for machine code, rather than trying to recreate the game's higher-order logic/structure or source code.
    - krackers 7mo ago
      This vaguely reminds me of futamura projections. Normally with futamura's first projection the input is source code, and you partial-evaluate that source against an interpreter for that source, resulting in a "compiled" binary that has the logic of the source inlined into the interpreter (and hopefully optimized). This is similar to what Truffle does I believe, where you have an interpreter (written in java) and then during runtime Truffle "JIT" optimizes the interpreted program's AST against the interpreter's logic. All of this can be considered a specialization of the JVM running an interpreter interpreting your program. In this case with "recompilation" you have a binary made to run on certain hardware. You then take an emulator of the hardware (registers, PC, etc.) and then "partial evaluate" the binary against the hardware emulator, producing a new program that contains a software emulation of just that specific binary. So while you're still conceptually emulating the underlying hardware, you both avoid the instruction dispatch overhead at runtime (it's statically compiled in) and also benefit from the optimization passes of modern compilers.
- bri3d 7mo ago
  It's more nuanced than that; the approach you're describing is usually called "decompilation." The difference is how far one goes in hoisting the "source code;" in this "recompliation" approach the source code, while C++, is basically an IR (intermediate representation) between the original game's assembly and a host platform, and the hardware itself is emulated (for example, the original architecture's CPU registers are represented as variables in the host architecture's memory). The machine code is translated to C++ using a custom tool. In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms. The product is something that attempts to resemble the original game's source code. Of course, they lie on a continuum and both approaches can be mixed, but, while they both involve C++ in the middle, the process is starkly different. Recompilation is much more copyright-friendly, because in many implementations only the modifications are distributed and the original binary is translated by the end user (who owns the software/a license to it), whereas decompilation produces an artifact (source code) which is a derivative work encumbered by the original software's license and generally should not be distributed.
  - kombine 7mo ago
    > In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms There definitely is a lot of scope to apply LLMs here
    - iscisjvije 7mo ago
      no one absolutely no one not a single soul on this Earth LLM nut: OMG LLM!!!!!!! Can’t you just drop it, please?

giancarlostoro 7mo ago

The blog post reminds me of similar efforts with Shockwave. There's people building decompilers, and runtimes, and if you join enough Discords, you will notice the people in them are cross-contaminating between communities if you will, they share insights with one another, in their efforts towards specific goals. They're hyper focused on making one game or another come back. There's Habbo Hotel, and Coke Studios, as well as other games. The Coke Studios effort is interesting because there were no "private servers" developed at the time, unlike Habbo which had many, and there are Shockwave Xtras that no open runtime supports currently. There's several attempts at a full runtime as well, that run in-browser. Projector Rays (decompiler) really was the biggest release to date, and recently people have been really hacking at it, to some extent AI has helped to reverse engineer bytecode far as I can tell. For anyone curious, one of the runtimes is called DirPlayer: https://github.com/igorlira/dirplayer-rs https://github.com/igorlira/dirplayer-rs

xnx 7mo ago
> cross-contaminating between communities if you will The positive term for this is cross-polinating
YesBox 7mo ago
Where can I boot up Coke Studios? Enjoyed that game as a kid, including making music (which was so friggen cool! They had tons of samples you could just drag and drop into a timeline). Edit: fun memory. I figured out how scamming works at some point. IIRC (and I may not) I set up a minigame in my studio, and contestants had to give me their furniture to participate in a game. Since furniture could stack weirdly, people made all sorts of crazy mazes that were actually quite difficult to navigate due to the fixed isometric perspective. Each stage required handing over a more expensive item than the last. At some point someone handed over a super rare/expensive piece of furniture. Not gonna erase that. ACTUALLY I think I figured out how to clone items due to a bug. Or mine infinite DBs. Used that as cover to get someone to give me their item. Was a long time ago. Anyways, a user handed me their super rare item and I then gave them a stern warning on how this is a common scam and I just stole their item(s). Then I handed everything back over to them and they profusely thanked me. lol
7mo ago

triangleman 7mo ago

What's the latest on homebrewing on the actual Xbox 360 these days?

opan 7mo ago
You can play game backups, use unofficial HDDs at bigger sizes, send over new games with FTP, use third party and DIY controllers like GuliKit's offerings or a Flatbox leverless arcade controller. You can play a modded version of Rock Band 3 with all the songs from all the games, using a Wii Guitar Hero controller with a Pi Pico-based adapter in place of the Wiimote. Any game you installed from disc for faster load times becomes playable without the disc in automatically once you get Aurora running (this is pretty cool, on some other modded consoles like PS4 you'd have to reinstall the games another way). As far as homebrew in the sense of people making their own games and applications, I don't know that there's much going on. Maybe you can find a controller tester app. Sadly the 360 scene has a lot of proprietary stuff going on, and uses some official dev SDKs, IIRC. Where 3DS and Switch modders have everything on GitHub and freely licensed, 360 has a lot of binary-only releases on random forums and such. It would probably be a major effort to change this and involve rewriting stuff people already know and love. I did the the RGH 3 mod to half a dozen or so 360s a few years back for myself and friends. If you just wanna play games with modern comforts, things are in a pretty good state. We don't need to worry about the original disc drives, HDDs, or controllers wearing out and making the console useless, we can replace or avoid them all. The wireless card is also easily replaceable, a friend of mine had one that seemed non-functional after he spilled some Mtn Dew all over his 360 back in the day. Everything else seemed good after a clean but I just got him a new wireless card off eBay and swapped that out. If anyone wants to get into this and doesn't already have a 360, the S models are easiest to mod and most reliable (no RRoD). Some are Trinity and some are Corona motherboards inside, but both are hackable. Corona may need a post-fix adapter, but they're cheap and easy to install. Just avoid the E series. For modding tutorials, MrMario2011 on YT is great. There is also a software-only exploit that's a bit worse but more accessible. It came out after I already did all my modding, so I haven't got experience with it.

asdff 7mo ago

Crazy that it has taken so long. I understand these are small teams trudging off in the dark but you could have imagined there would have been more eyes on this and hands tinkering in the time when the Xbox 360 was being sold than today. Right at that same time in history the iPhone was getting cracked basically every iOS release, sometimes by teenagers. Seems like there were a ton of hackers around back then.

wmf 7mo ago
Most people just want emulation which was solved a while ago. Recompilation is mostly for modders which is a very small community.
- tetrisgm 7mo ago
  The whole decomp and recomp scene has really started popping off this past year in the emulation community. The N64 recomp tool was the first one to really get interest going.

Venn1 7mo ago

These projects are really neat. Last week, I was able to build (and play) the Xbox 360 release of Sonic Unleashed on a couple of ARM SBCs using Sonic Unleashed Recomp.

wesfenlon 7mo ago

Hey, that's my newsletter -- thanks for sharing! :D Some exciting stuff has already happened in the week since I wrote the interview. Here's Viva Pinata: Trouble in Paradise: https://www.youtube.com/watch?v=Kih42KlocII https://www.youtube.com/watch?v=Kih42KlocII

tetrisgm 7mo ago
Was playing that last night. So many great recomps lately. Ty for the newsletter
testing22321 7mo ago
How does the static recompiler handle self modifying code?
tetrisgm 7mo ago
btw any way to reach out to you (twitter, discord?). big fan of your blog. I'd say on par with Time Extension!

rendaw 7mo ago

Years ago I was searching for why emulators don't recompile to native bytecode and it seemed like the consensus at the time was that JIT etc was faster. I can't find it now, but IIRC I remember replies from maintainers saying it was a bad idea. What changed since then, since this seems to be a trend? Is this more for more modern systems that have more static code?

tetrisgm 7mo ago
It's a combination of things. The floodgates were opened when the N64 recompiler popped up a couple (?) of years back. Really got interest going. I also think tooling got better, and with the advent of AI engineering it made all of this a lot more accessible.
CJefferson 7mo ago
Static code. Also, very fine details of the machine because less important. When emulating a NES (for example), you really have to emulate every register, how registers change, and also weird effects like instructions that take longer to read or write values, as games rely on that stuff. Once you have modern systems where much of the code was originally C, it becomes less important to ensure every register has exactly the right values when a subroutine finishes in most cases, you can rely that (most) of the code follows standard calling conventions.
- charcircuit 7mo ago
  You really don't. Most of the time the code will be doing normal things. So you can have other code that also does the same thing. >and also weird effects like instructions that take longer to read or write values The percentage of a code in the ROM that depends on such things is a small percent.
  - CJefferson 7mo ago
    The problem with NES emulation (having written a fairly bad one) is it can be hard to predict when you need to be careful, particularly in the presence of self-modifying code and interrupts. If it was easy to figure out, emulators wouldn't have to keep such careful track of this stuff either.

cosmic_cheese 7mo ago

Very nice. It’d be interesting if recomps of some games that were originally for both 360 and PC become the superior experience… Fable 3 and the whole Games for Windows Live nonsense the PC version had come to mind.

zxspectrum1982 7mo ago

I wonder whether this "recompilation" technology could be used to run Windows software on Linux, as an alternative/complementary technique to Wine. Back in the day (year 2000, until 10-15 years ago) we had Project Odin to dynamically translate Windows software to run on OS/2: https://github.com/netlabsorg/odin32 https://github.com/netlabsorg/odin32

pezezin 7mo ago
Windows and Linux run on the same x86 CPU architecture, you don't need to recompile anything. What WINE does is to translate/replace system calls and libraries.
- zxspectrum1982 6mo ago
  I know how Wine works, thank you. I've used winelib in the past to port Windows software (that I had the source code of) to Linux myself. My point is the way wine works today is: WinAPI --> winelib --> Linux --> x86 I. e. winelib is reimplementing WineAPI on top of Linux. What if we could just decompile those Windows functions and recompile them on Linux, or even x86, directly? The workaround all of these console recompilation projects use is you must have the original game in order to have the binaries (the ones you theoretically decompile and recompile, but actually you take pre-decompiled-recompiled ones by someone else) and assets (graphics, sounds, etc). For Windows applications on Linux, we could do the same: bring your own Windows, then we can decompile and recompile.