5 ms·

Spider-Man (Neversoft) Decompilation Project Progress Checkpoint – May 2024

113 points by krystalgamer 2y ago

foxtacles 2y ago
I've been working on a similar project (LEGO Island decompilation). We've developed an extensive set of annotations and corresponding tools that facilitate matching the assembly/binary: https://github.com/isledecomp/isle/tree/master/tools https://github.com/isledecomp/isle/tree/master/tools We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well.
- bee_rider 2y ago
  I loved that game growing up. Now the brick-by-brick song is stuck in my head. It really captured the open ended nonsense feeling of sitting down and playing with legos.
- tnetenbaa 2y ago
  Is this Mr.MattKC himself on HN?!
  - rzzzt 2y ago
    The profile picture tells me it's a different person, unless he is also a master of disguise.
    - foxtacles 2y ago
      Yeah I'm not Matt :) No idea if he is on HN
- krystalgamer 2y ago
  > We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well. Happy to use anything that makes my life easier! I myself also started to prototype some tools - https://github.com/krystalgamer/spidey-decomp/tree/master/tools https://github.com/krystalgamer/spidey-decomp/tree/master/to... - but didn't get much far. I see there's quite a bit of overlap so I might integrate them on my workflow Curious to know about `patch_c2.py` it mentions a bugged warning. Is that patch just to remove the warning or something deeper?
  - foxtacles 2y ago
    It just removes the warning, which is impossible to get rid of otherwise. This bug has persisted all the way into MSVC 6.0: https://web.archive.org/web/20000519112833/https://support.microsoft.com/support/kb/articles/Q167/3/55.ASP https://web.archive.org/web/20000519112833/https://support.m... Since the warning is relatively useless we opted to fix C2.EXE so that it is never emitted. It generates a lot of noise during compilation otherwise (if you are using templates).
- jchw 2y ago
  As someone who is very interested in decompiling old Windows executables, this is very relevant to my interests. Thanks for the link!
semolino 2y ago
Cool to see krystalgamer here on HN. I found them a few years ago through the (then super tiny) SM2000 modding scene on youtube. It was my first 3D game, and I still have dreams about the graphics. Their work on the resource unpacker utility allowed me to make custom skins of my own characters for the game, which I screenshotted and used for a comic. It's weird: I often think and dream in the visual language of that game. Anywayyyyy, will be following this development more closely now. Would be really cool to eventually see some type of level editor for this game, or maybe a multiplayer server (akin to this Bomb Rush Cyberfunk mod: https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopCrew/ https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopC... )
- krystalgamer 2y ago
  Thanks so much for the kind words! Did you end up publishing your skins? > Would be really cool to eventually see some type of level editor for this game So there's already a tool by Zedek that can export the models to blender (not sure about re-importing). I heard that PSXPrev by rickomax can work with the game files but never tried myself. As for level editor, there was this guy that came into the community and made a proper level unpacker + packer, it was primitive but worked much better than anything that I've ever done. Sadly he disappeared but his tool still floats around from what I remember. > maybe a multiplayer server I get asked about this a lot :D Multiplayer will be a challenge in the way it handles events, there's a lot of hardcoded stuff. Not saying it's impossible, but it'd require a big re-write of the engine.
xyst 2y ago
I can’t imagine decompiling a project that was likely written by at least a team of a dozen developers over the course of 2-3 yrs to be completed by a single person. Looking at the commit history, I see only contributions by the author “krystalgamer”. Wild. Rare to find a person with such resolve and enjoyment. Wish this person good luck!
- krystalgamer 2y ago
  Appreciate your kind words!
AdmiralAsshat 2y ago
Nice project. Spider-Man (2000) was a great game for the time. It had a solid action/platforming engine, it had voice-acting with a number of actors pulled from some of the animated shows that were running at the time, and it came out before the 2002 Sam Riami film that forced most subsequent games to follow the movie timelines. I played the N64 version at the time, and found it fairly polished and much better than most of the Spider-Man games that came out before it (Arcade's Revenge, Maximum Carnage, Separation Anxiety, to name a few). I would argue it would hold the crown for best Spider-Man game for some time until the flawed masterpiece Spider-Man 2 (2004) gave us truly amazing web-slinging physics in a sandbox environment, or the less-free-but-tighter-overall Ultimate Spider-Man in 2005.
- causality0 2y ago
  If you haven't tried it, I highly recommend the Dreamcast version. It has the graphics and voice-acting of the PC version with the tight analog controls of the N64 version.
- tombert 2y ago
  I thought Ultimate Spider-Man was a severe downgrade over Spider-Man 2. The mechanics for swinging just felt a lot more automatic and were less satisfying overall. Spider-Man 2 had, in my opinion, nearly perfect swinging mechanics and did a great job feeling truly open. Its biggest problem was that the actual missions weren't terribly fun, particularly the boss fights were clunky and unfun. Still, it's fun enough to just swing around the city that I can just ignore the missions and just play around. Spider-Man 2000 was still fun. I haven't played the Dreamcast version, but I played the hell out of the PS1 version, unlocking nearly everything in the process.
  - AdmiralAsshat 2y ago
    That's the thing--Spider-Man 2 was rather flawed overall. The main story missions were meh, and the side missions got tedious after awhile (how many people in this damn city are hanging from a ledge at any given moment?). It was much like GTA3, where it was almost a chore to make myself finish the thing, and I'd normally just fire it up to swing around town aimlessly. I'd probably only rate it an 8/10 overall (at the time). It just had the advantage of absolutely nailing the feeling of web-slinging in a way that no game ever did or arguably did again until Spider-Man (2018). USM was definitely a downgrade on the freedom/webslinging front, but pretty much everything else was better, and it was a tighter experience. I guess it comes down to whether you're the type who prefers the game with the better overall package, or the game with one killer feature that it does really well and you can't find anywhere else.
boricj 2y ago
I've been working on my own reverse-engineering/decompilation project (Tenchu: Stealth Assassins) and I've created a Ghidra extension that can export a program selection as a working, relocatable object file [1]. I've had some really good results on x86 since writing an analyzer for an architecture where relocation spots target 4-byte immediate fields inside of instructions is fairly easy. Unfortunately, the PlayStation uses a MIPS processor and writing an analyzer for split HI16/LO16 relocations is proving to be a devilishly tricky problem. I got it to a point where it works well enough on MIPS most of the time, but there's always a new weird edge case hidden inside a function thousands of instructions long where it breaks down... [1] https://github.com/boricj/ghidra-delinker-extension https://github.com/boricj/ghidra-delinker-extension
- krystalgamer 2y ago
  Seems like a neater alternative than the linker scripts used by most projects. > Creating libraries, by extracting parts of a program and reusing them in another context; This seems like a big win when creating (un-)packers for custom formats. For this game after isolating the routine that decompresses the textures it was so annoying to not mess up the logic - https://github.com/krystalgamer/spidey-tools/blob/master/psx_extractor/psx.c https://github.com/krystalgamer/spidey-tools/blob/master/psx...
jchw 2y ago
I've been doing a bit of research on and off for the past few years on decompilation and it's definitely challenging to decide how close you want to go to matching. If you can get the exact compiler and exact compilation settings, it's totally feasible to do matching decompilation, and if you're able to make this somehow incremental such that you can incrementally work up to 100% matching over time, it seems like a really good approach, but it requires a lot of groundwork and understanding how the compiler and linker really work. In the process of matching compilation of functions on a binary I was analyzing that was compiled with Visual Studio 2003, I realized that very subtle differences can cause e.g. different register allocation, even in an old compiler with dramatically less sophisticated optimization passes. Anyway, I guess this tangent is really unrelated, but I think more people should be embarking on decompilation projects. It's very fun, and it's uniquely rewarding if you manage to get some non-trivial decompilation of code to work properly. I had one odd use case for decompiling that was actually, as far as I know, completely licit: WebView2Loader. Microsoft distributed the WebView2 SDK as 3-BSD so that you could integrate it into your applications without worrying about licensing, but the glue logic that actually interacts with the WebView2 installation and instantiates the COM objects is closed source. But... since it is closed-source 3-BSD, without a EULA... we can reverse engineer it. It being a relatively small shim, I did just that[1]. This was an easy exercise armed with an interactive disassembler, and since it was relatively simple and very small I didn't need to bother with matching anything: I just roughly replicated the behavior instead. The use case for this was allowing people to make WebView2 bindings that didn't have any external dependencies; the OpenWebView2Loader code was ported to Pascal and Go by others, making it possible to have pure bindings that don't require any C code or external DLLs and can directly talk to the WebView2 installation. There's now a static copy of the WebView2Loader with the SDK, which obviates some of the use of this, but this is still a nice approach for Go where you can entirely avoid CGo or messing with weird object format conversion. (It's way better than my original approach for WebView2 in Go, which is to emulate the Windows linker to link and execute an entirely in-memory copy of the WebView2Loader DLL using a lot of unsafe code. That also works, but it is much more bug prone and frankly horrifying.) [1]: https://github.com/jchv/OpenWebView2Loader https://github.com/jchv/OpenWebView2Loader
- boricj 2y ago
suroot 2y ago
Which decompiler?
wiseowise 2y ago
Man, makes me sad just how much time and effort is wasted on decompiling those old games. All of this effort could’ve been spent on implementing features and fixing bugs by super dedicated fans of the games. Instead, there’s a constant fight with code and bloodsucking lawyers of greedy companies who don’t give a single shit about those games. Most of those aren’t even sold anymore. Destined to rot, because of petty reasons.
- deleted 2y ago
  [deleted]
- xen2xen1 2y ago
  The journey is it's own reward, my son.
- quadral 2y ago
  I disagree. There will be times where a game source code has been lost in the void... and then you get these godsend people who are able to reverse the game and recreate it. We need these talented people.
  - wiseowise 2y ago
    I meant that companies should release source code for 20+ old games, especially if they’re not sold or actively developed anymore.
- krystalgamer 2y ago
  If you check my repository of mods for the game - https://github.com/krystalgamer/spidey-tools https://github.com/krystalgamer/spidey-tools You can see bug fixes and new features. There's just a moment where bigger features become unfeasible to be done this way. It's the same dedicated fans that implement new features and fix bugs that then move into decompilation projects.
  - wiseowise 2y ago
    I meant that all the time and energy you're using on decompilation could've been used more productively if original source code was made readily available by the companies that produced the games.
king_geedorah 2y ago
Awesome project. The PS1 version of this game is one of my favorites from growing up and also the reason my home server's hostname was "eelnats" for many years.
chris_wot 2y ago
“ As described in this post, decompilation project is tedious and laborious. Therefore it's easy to lose motivation and leave it in backburner for a long-time. The solution I have found is consistency, everyday try to work on it a bit as all effort will compound.” I’m current self studying calculus. This is exactly the situation I am falling into. I now spend an extra hour and a half or so at work in the lunch room (my workplace is 24x7) and go through the calculus book. I’ve been doing this for 6 weeks or so, and though my progress may seem slow to many (I just got through limits) I understand the material properly. (In terms of 6 weeks, I did the prerequisites chapters first which took some time, then I had a holiday, and I spent almost a week and a half on the Delta-Epsion definition of a limit - and I do all the problems in the book and there are a lot)

113 points by krystalgamer 2y ago

I've been working on a similar project (LEGO Island decompilation). We've developed an extensive set of annotations and corresponding tools that facilitate matching the assembly/binary: https://github.com/isledecomp/isle/tree/master/tools https://github.com/isledecomp/isle/tree/master/tools We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well.

bee_rider 2y ago
I loved that game growing up. Now the brick-by-brick song is stuck in my head. It really captured the open ended nonsense feeling of sitting down and playing with legos.
tnetenbaa 2y ago
Is this Mr.MattKC himself on HN?!
- rzzzt 2y ago
  The profile picture tells me it's a different person, unless he is also a master of disguise.
  - foxtacles 2y ago
    Yeah I'm not Matt :) No idea if he is on HN
krystalgamer 2y ago
> We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well. Happy to use anything that makes my life easier! I myself also started to prototype some tools - https://github.com/krystalgamer/spidey-decomp/tree/master/tools https://github.com/krystalgamer/spidey-decomp/tree/master/to... - but didn't get much far. I see there's quite a bit of overlap so I might integrate them on my workflow Curious to know about `patch_c2.py` it mentions a bugged warning. Is that patch just to remove the warning or something deeper?
- foxtacles 2y ago
  It just removes the warning, which is impossible to get rid of otherwise. This bug has persisted all the way into MSVC 6.0: https://web.archive.org/web/20000519112833/https://support.microsoft.com/support/kb/articles/Q167/3/55.ASP https://web.archive.org/web/20000519112833/https://support.m... Since the warning is relatively useless we opted to fix C2.EXE so that it is never emitted. It generates a lot of noise during compilation otherwise (if you are using templates).
jchw 2y ago
As someone who is very interested in decompiling old Windows executables, this is very relevant to my interests. Thanks for the link!

semolino 2y ago

Cool to see krystalgamer here on HN. I found them a few years ago through the (then super tiny) SM2000 modding scene on youtube. It was my first 3D game, and I still have dreams about the graphics. Their work on the resource unpacker utility allowed me to make custom skins of my own characters for the game, which I screenshotted and used for a comic. It's weird: I often think and dream in the visual language of that game. Anywayyyyy, will be following this development more closely now. Would be really cool to eventually see some type of level editor for this game, or maybe a multiplayer server (akin to this Bomb Rush Cyberfunk mod: https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopCrew/ https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopC... )

krystalgamer 2y ago
Thanks so much for the kind words! Did you end up publishing your skins? > Would be really cool to eventually see some type of level editor for this game So there's already a tool by Zedek that can export the models to blender (not sure about re-importing). I heard that PSXPrev by rickomax can work with the game files but never tried myself. As for level editor, there was this guy that came into the community and made a proper level unpacker + packer, it was primitive but worked much better than anything that I've ever done. Sadly he disappeared but his tool still floats around from what I remember. > maybe a multiplayer server I get asked about this a lot :D Multiplayer will be a challenge in the way it handles events, there's a lot of hardcoded stuff. Not saying it's impossible, but it'd require a big re-write of the engine.

xyst 2y ago

I can’t imagine decompiling a project that was likely written by at least a team of a dozen developers over the course of 2-3 yrs to be completed by a single person. Looking at the commit history, I see only contributions by the author “krystalgamer”. Wild. Rare to find a person with such resolve and enjoyment. Wish this person good luck!

krystalgamer 2y ago
Appreciate your kind words!

AdmiralAsshat 2y ago

Nice project. Spider-Man (2000) was a great game for the time. It had a solid action/platforming engine, it had voice-acting with a number of actors pulled from some of the animated shows that were running at the time, and it came out before the 2002 Sam Riami film that forced most subsequent games to follow the movie timelines. I played the N64 version at the time, and found it fairly polished and much better than most of the Spider-Man games that came out before it (Arcade's Revenge, Maximum Carnage, Separation Anxiety, to name a few). I would argue it would hold the crown for best Spider-Man game for some time until the flawed masterpiece Spider-Man 2 (2004) gave us truly amazing web-slinging physics in a sandbox environment, or the less-free-but-tighter-overall Ultimate Spider-Man in 2005.

causality0 2y ago
If you haven't tried it, I highly recommend the Dreamcast version. It has the graphics and voice-acting of the PC version with the tight analog controls of the N64 version.
tombert 2y ago
I thought Ultimate Spider-Man was a severe downgrade over Spider-Man 2. The mechanics for swinging just felt a lot more automatic and were less satisfying overall. Spider-Man 2 had, in my opinion, nearly perfect swinging mechanics and did a great job feeling truly open. Its biggest problem was that the actual missions weren't terribly fun, particularly the boss fights were clunky and unfun. Still, it's fun enough to just swing around the city that I can just ignore the missions and just play around. Spider-Man 2000 was still fun. I haven't played the Dreamcast version, but I played the hell out of the PS1 version, unlocking nearly everything in the process.
- AdmiralAsshat 2y ago
  That's the thing--Spider-Man 2 was rather flawed overall. The main story missions were meh, and the side missions got tedious after awhile (how many people in this damn city are hanging from a ledge at any given moment?). It was much like GTA3, where it was almost a chore to make myself finish the thing, and I'd normally just fire it up to swing around town aimlessly. I'd probably only rate it an 8/10 overall (at the time). It just had the advantage of absolutely nailing the feeling of web-slinging in a way that no game ever did or arguably did again until Spider-Man (2018). USM was definitely a downgrade on the freedom/webslinging front, but pretty much everything else was better, and it was a tighter experience. I guess it comes down to whether you're the type who prefers the game with the better overall package, or the game with one killer feature that it does really well and you can't find anywhere else.

boricj 2y ago

I've been working on my own reverse-engineering/decompilation project (Tenchu: Stealth Assassins) and I've created a Ghidra extension that can export a program selection as a working, relocatable object file [1]. I've had some really good results on x86 since writing an analyzer for an architecture where relocation spots target 4-byte immediate fields inside of instructions is fairly easy. Unfortunately, the PlayStation uses a MIPS processor and writing an analyzer for split HI16/LO16 relocations is proving to be a devilishly tricky problem. I got it to a point where it works well enough on MIPS most of the time, but there's always a new weird edge case hidden inside a function thousands of instructions long where it breaks down... [1] https://github.com/boricj/ghidra-delinker-extension https://github.com/boricj/ghidra-delinker-extension

krystalgamer 2y ago
Seems like a neater alternative than the linker scripts used by most projects. > Creating libraries, by extracting parts of a program and reusing them in another context; This seems like a big win when creating (un-)packers for custom formats. For this game after isolating the routine that decompresses the textures it was so annoying to not mess up the logic - https://github.com/krystalgamer/spidey-tools/blob/master/psx_extractor/psx.c https://github.com/krystalgamer/spidey-tools/blob/master/psx...

jchw 2y ago

I've been doing a bit of research on and off for the past few years on decompilation and it's definitely challenging to decide how close you want to go to matching. If you can get the exact compiler and exact compilation settings, it's totally feasible to do matching decompilation, and if you're able to make this somehow incremental such that you can incrementally work up to 100% matching over time, it seems like a really good approach, but it requires a lot of groundwork and understanding how the compiler and linker really work. In the process of matching compilation of functions on a binary I was analyzing that was compiled with Visual Studio 2003, I realized that very subtle differences can cause e.g. different register allocation, even in an old compiler with dramatically less sophisticated optimization passes. Anyway, I guess this tangent is really unrelated, but I think more people should be embarking on decompilation projects. It's very fun, and it's uniquely rewarding if you manage to get some non-trivial decompilation of code to work properly. I had one odd use case for decompiling that was actually, as far as I know, completely licit: WebView2Loader. Microsoft distributed the WebView2 SDK as 3-BSD so that you could integrate it into your applications without worrying about licensing, but the glue logic that actually interacts with the WebView2 installation and instantiates the COM objects is closed source. But... since it is closed-source 3-BSD, without a EULA... we can reverse engineer it. It being a relatively small shim, I did just that[1]. This was an easy exercise armed with an interactive disassembler, and since it was relatively simple and very small I didn't need to bother with matching anything: I just roughly replicated the behavior instead. The use case for this was allowing people to make WebView2 bindings that didn't have any external dependencies; the OpenWebView2Loader code was ported to Pascal and Go by others, making it possible to have pure bindings that don't require any C code or external DLLs and can directly talk to the WebView2 installation. There's now a static copy of the WebView2Loader with the SDK, which obviates some of the use of this, but this is still a nice approach for Go where you can entirely avoid CGo or messing with weird object format conversion. (It's way better than my original approach for WebView2 in Go, which is to emulate the Windows linker to link and execute an entirely in-memory copy of the WebView2Loader DLL using a lot of unsafe code. That also works, but it is much more bug prone and frankly horrifying.) [1]: https://github.com/jchv/OpenWebView2Loader https://github.com/jchv/OpenWebView2Loader

boricj 2y ago

suroot 2y ago

Which decompiler?

wiseowise 2y ago

Man, makes me sad just how much time and effort is wasted on decompiling those old games. All of this effort could’ve been spent on implementing features and fixing bugs by super dedicated fans of the games. Instead, there’s a constant fight with code and bloodsucking lawyers of greedy companies who don’t give a single shit about those games. Most of those aren’t even sold anymore. Destined to rot, because of petty reasons.

deleted 2y ago
[deleted]
xen2xen1 2y ago
The journey is it's own reward, my son.
quadral 2y ago
I disagree. There will be times where a game source code has been lost in the void... and then you get these godsend people who are able to reverse the game and recreate it. We need these talented people.
- wiseowise 2y ago
  I meant that companies should release source code for 20+ old games, especially if they’re not sold or actively developed anymore.
krystalgamer 2y ago
If you check my repository of mods for the game - https://github.com/krystalgamer/spidey-tools https://github.com/krystalgamer/spidey-tools You can see bug fixes and new features. There's just a moment where bigger features become unfeasible to be done this way. It's the same dedicated fans that implement new features and fix bugs that then move into decompilation projects.
- wiseowise 2y ago
  I meant that all the time and energy you're using on decompilation could've been used more productively if original source code was made readily available by the companies that produced the games.

king_geedorah 2y ago

Awesome project. The PS1 version of this game is one of my favorites from growing up and also the reason my home server's hostname was "eelnats" for many years.

chris_wot 2y ago

“ As described in this post, decompilation project is tedious and laborious. Therefore it's easy to lose motivation and leave it in backburner for a long-time. The solution I have found is consistency, everyday try to work on it a bit as all effort will compound.” I’m current self studying calculus. This is exactly the situation I am falling into. I now spend an extra hour and a half or so at work in the lunch room (my workplace is 24x7) and go through the calculus book. I’ve been doing this for 6 weeks or so, and though my progress may seem slow to many (I just got through limits) I understand the material properly. (In terms of 6 weeks, I did the prerequisites chapters first which took some time, then I had a holiday, and I spent almost a week and a half on the Delta-Epsion definition of a limit - and I do all the problems in the book and there are a lot)