5 ms·

Spice86 – A PC emulator for real mode reverse engineering

175 points by alberto-m 2y ago

kevin286 2y ago
Hello! I'm the creator of the Java version of Spice86, and I just saw this thread. Let me address a few topics: Why Java / C#? I initially chose Java because it's a decently fast language I am comfortable working with, and it has a lot of tooling available to investigate performance issues, debug easily, and provide basic, easy access to multiplatform sound and graphics. However, we eventually migrated to C# for several reasons: - Control Structures: Re-implementing Cryo Dune DOS assembly code in Java proved challenging, especially when mapping jumps to high-level control structures. Just adding a goto and taking care of it later was easier. - Unsigned Integers: Java's lack of support for unsigned integers was a source of bugs. - Similarities: C# and Java have similar syntaxes so migration was not too crazy, toolings are equally good, and performances are comparable. Why Not Just Use Ghidra / IDA? I wish we could, but there are many reasons why this isn't straightforward: - Ghidra Support for 16-bit x86 real mode isn't great, with some bugs requiring significant investment to fix. For example, this issue: https://github.com/NationalSecurityAgency/ghidra/issues/981 https://github.com/NationalSecurityAgency/ghidra/issues/981. I guess no one is willing to invest in that because there is no market. - IDA is not free and hex-rays doesn't support decompiling DOS code. Additionally, code from that era often involved hand-crafted assembly mixed with C, compiled by forgotten tools. As a result, tools like Ghidra and IDA struggle with static analysis due to practices like self-modifying code, jumping in the middle of instructions, and editing the function call stack to redirect return statements. One simple yet concrete example is the switch statement. When a switch statement is written, one implementation method a compiler might use is creating a table in memory with the addresses of all the case statements. The assembly code will then compute the address in that table based on the condition and jump to the appropriate case. When decompiling such code, if the compiler version is known and supported, you can infer the location of the jump table and reconstruct the addresses of all the code in the switch statement. However, if the code is handwritten assembly, you would need to debug it at execution time to find where the rest of the code is, as it is not statically reachable. Very concretely, if you see something like JMP AX, you need to debug to see where is the rest of the code. How Do We Get Work Done despite all that? Spice86 provides two main strategies: - Override assembly with high-level code, that way you can rewrite your game bit by bit, testing as you go. - Execution flow recording and code generation. It records execution flow and tells you what was executed. We have a Ghidra plugin to import that data and generate code from it. For example, see https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_OriginalAsm.cs https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_Or.... Interestingly, since it's now C#, you can decompile it again to extract control structures, as shown here https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_DecompiledAsm.cs https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_De.... Issue with code generation is that ghidra does not like real mode code so the plugin is full of workarounds (and broken at the moment), and there are some fundamental things that can't be done. For instance, self modifying code support is murky with the ghidra plugin. Future Improvements We're working on eliminating Ghidra by generating code directly from Spice86 from the recorded execution trace. Our goal is to fully support self-modifying code and generate functional code from an execution trace with minimal human intervention. I thought I would make a quick reply, but it turns out there's a lot to say on the topic :)
- LowLevelMahn 2y ago
  so Spice86 helps in translating realmode 16bit DOS code into similar assembler looking like C# code - which got the exact same semantic and behavior but its pure C# - you can then use the normal Visual Studio debugger/IDE to port or extend the code - that is a huge benefit over beeing trapped using >30 years old DOS debuggers, compilers and IDEs for this already very hard work its to ease the process of rewriting assembler code into high level code by having a highlevel-assembler-code in between Spice86 is not intended for Binary exact reversing, like with https://www.scottsmitelli.com/projects/cosmore/ https://www.scottsmitelli.com/projects/cosmore/ or Duke Nukem, ZZT or F15 Eeagle reversing project - these are using the same compiler and try to replicate every byte of the original exe - to prevent any reversing bugs at first - but this is the hardest form of reversing and takes easily years of hard work
- alberto-m 2y ago
  Thanks for the great answer and for building this tool! Wish I had known about it sooner, I'll surely adopt it in my future retrohacking projects!
johnklos 2y ago
Forty years ago I had a Sinclair QL with an 8086 emulator. Because the Sinclair QL had preemptive multitasking, I could easily search memory for patterns, monitor locations, stop and start the emulation, or change memory programmatically and easily from the QDOS side. It was worlds easier than using a debugger, particularly since I didn't own an 8086 system. I always thought it was a clever way to get insights in to software while it was running that wasn't available to people with 8086 systems, and it's interesting to see this idea so many years later.
- mananaysiempre 2y ago
  Bochs and MAME both have superb and widely-used debuggers, while Qemu is more limited but still has some debugging capabilities in its monitor, as well as a gdb integration. (Can’t say anything about PCem/86Box.) It seems that developers of emulators targeting good coverage of old stuff simply can’t not build a debugger, because it’s an integral part of their task to figure out what the hell the devs of the latest failing thing did to make it fail. Bochs is (was?) also quite popular in the OSDev scene as a debugging tool.
  - rzzzt 2y ago
    DOSBox can be configured to include a debugger. The feature is not enabled in the official binary but the enhanced derivative projects probably have it (DOSBox-X definitely does): - https://www.vogons.org/viewtopic.php?t=3944 https://www.vogons.org/viewtopic.php?t=3944 - https://github.com/joncampbell123/dosbox-x/wiki/DOSBox%E2%80%90X%E2%80%99s-Feature-Highlights https://github.com/joncampbell123/dosbox-x/wiki/DOSBox%E2%80...
    - 2mlWQbCK 2y ago
      DOSBox-x on FreeBSD at least has is disabled by default, but it can be enabled when building from ports (make config). https://github.com/joncampbell123/dosbox-x/blob/master/README.debugger https://github.com/joncampbell123/dosbox-x/blob/master/READM...
DrNosferatu 2y ago
A tutorial on how to reverse engineer a simple DOS game would be absolutely awesome!
- pjturpeau 2y ago
  https://cosmodoc.org/ https://cosmodoc.org/
- alberto-m 2y ago
  From my brief experience, it seems that reversing old games is one of those disciplines where there is no good step-by-step course. One can start learning some theory (I did so by reading “The Art of Assembly Language Programming” – if I had more time, I'd try “Reverse Engineering for Beginners”) but then one has to get his hands dirty. Real-life games are typically not simple, but one usually just needs to reverse some small parts to produce new interesting modifications. But surely reading tutorials is useful to learn techniques and tricks. I recommend this article to start: https://www.lodsb.com/reversing-lz91-from-commander-keen https://www.lodsb.com/reversing-lz91-from-commander-keen (not totally for beginners, but very friendly, and it can help to get used with the jargon). I am also publishing war stories on this topic on my blog (marnetto.net).
- deleted 2y ago
  [deleted]
- stevekemp 2y ago
  Tutorials are hard, but there are some great writeups like this which discuss some of the specific problems and trial/error involved https://neuviemeporte.github.io/category/f15-se2 https://neuviemeporte.github.io/category/f15-se2 In this case one hard part was trying to get code to compile to identical byte for byte output. Which meant working out which compiler options were used, and which specific compiler too. That gives you a hint of what kinda things are involved.
  - genewitch 2y ago
    Have you mentioned this before? I saw a similar context comment maybe 3 days ago on here, down to "byte for byte output [...W]hich compiler options were used"
bernadus_edwin 2y ago
Why are so many emulators written in C#?
- sixothree 2y ago
  My guess is portability, then obviously performance. edit: actually there is a specific answer for this particular project - "We had to rewrite the project in C# to add automated code generation (java doesn't have the goto keyword, making automated ASM translation challenging)". There you are.
  - anyfoo 2y ago
    I mean, that's more or less the reason why it isn't Java, not why it's ultimately C#. My guess is that Java is just what they're most comfortable with, with C# being similar enough but avoiding specific limitations in that case.
    - genewitch 2y ago
      Wasn't C# essentially microsoft throwing their hat in the ring against Oracle and to show off how cool this .net stuff is? I dabbled in both at around the same time a long time ago for console apps and visual studio's autocomplete / assist / library fetch etc made it easier than Java to get working in but... Its been so long I forget the origin stories sometimes.
      - maximilienNoal 2y ago
        .NET began its life as Microsoft's Java (the platform) along with C# being a mix between Java, C++, and Pascal. Considering the timing (early 2000s), one can't help but think that it was in response to the legal actions from Sun Microsystems over their usage of the JVM in Windows. "OK, I'll make my own Java language, with properties and an integrated DSL for queries!"
    - sixothree 2y ago
      I think my question is why not choose c# for this? What’s the apprehension here if any?
gexos 2y ago
[flagged]
eminence32 2y ago
Question from a reverse-engineering noob: Why can't ghidra (or any other reverse engineering tool) be used directly on the .exe? Why do you have to go through this emulator? Is it because the thing you want to debug only runs in x86 realmode?
- rzzzt 2y ago
  Obfuscation and compression are two potential extra hoops to jump through. It's easier to let the executable run for a bit and start from there.
- dmitrygr 2y ago
  x86 segmentation makes it very hard to statically analyze anything. In real mode, any byte can be referenced in 4096 different ways. It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless. So, without runtime analysis, there is no way to tell if 04:1234 is or is not the same byte as fa:1204
  - jcranmer 2y ago
    > It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless. Actually, my experience is that things are much easier in protected mode. Since selector values are chosen by the OS, that means you rely a lot more on internal relocations. And the use of segment selectors is a strong indicator that you have a pointer in the first place. Unfortunately, ghidra itself struggles to apply these techniques, especially in the decompiler, which seems completely unable to cope with the concept of far pointers.
    - dmitrygr 2y ago
      In DOS, plenty of applications/games load selectors and do nasty things with them so indeed you'd know it is a far pointer, but may not know what to :D
      - 2y ago
ggambetta 2y ago
Oooh, I LOVE this! Especially the ability to "Overriding emulated code with C# code" I had a similar idea years ago (https://gabrielgambetta.com/remakes.html https://gabrielgambetta.com/remakes.html), not in the context of a debugger or reverse engineering per se, but in the context of remakes and "special edition" games. Not entirely surprised that this is a byproduct of OpenRakis. Amazing work!
- vunderba 2y ago
  I tried doing something like this about 15 years ago but specifically for audio by routing NES NSF rom audio data (square, triangle, PWM, etc) to virtual midi cables attached to VSTs so you could play any old school Nintendo game with modern instrumentation. Was a pretty fun project. The closest thing I can think of for graphical rehauls is probably shader pack type stuff - Minecraft is a great example of this. https://www.sonicether.com/seus https://www.sonicether.com/seus

175 points by alberto-m 2y ago

Hello! I'm the creator of the Java version of Spice86, and I just saw this thread. Let me address a few topics: Why Java / C#? I initially chose Java because it's a decently fast language I am comfortable working with, and it has a lot of tooling available to investigate performance issues, debug easily, and provide basic, easy access to multiplatform sound and graphics. However, we eventually migrated to C# for several reasons: - Control Structures: Re-implementing Cryo Dune DOS assembly code in Java proved challenging, especially when mapping jumps to high-level control structures. Just adding a goto and taking care of it later was easier. - Unsigned Integers: Java's lack of support for unsigned integers was a source of bugs. - Similarities: C# and Java have similar syntaxes so migration was not too crazy, toolings are equally good, and performances are comparable. Why Not Just Use Ghidra / IDA? I wish we could, but there are many reasons why this isn't straightforward: - Ghidra Support for 16-bit x86 real mode isn't great, with some bugs requiring significant investment to fix. For example, this issue: https://github.com/NationalSecurityAgency/ghidra/issues/981 https://github.com/NationalSecurityAgency/ghidra/issues/981. I guess no one is willing to invest in that because there is no market. - IDA is not free and hex-rays doesn't support decompiling DOS code. Additionally, code from that era often involved hand-crafted assembly mixed with C, compiled by forgotten tools. As a result, tools like Ghidra and IDA struggle with static analysis due to practices like self-modifying code, jumping in the middle of instructions, and editing the function call stack to redirect return statements. One simple yet concrete example is the switch statement. When a switch statement is written, one implementation method a compiler might use is creating a table in memory with the addresses of all the case statements. The assembly code will then compute the address in that table based on the condition and jump to the appropriate case. When decompiling such code, if the compiler version is known and supported, you can infer the location of the jump table and reconstruct the addresses of all the code in the switch statement. However, if the code is handwritten assembly, you would need to debug it at execution time to find where the rest of the code is, as it is not statically reachable. Very concretely, if you see something like JMP AX, you need to debug to see where is the rest of the code. How Do We Get Work Done despite all that? Spice86 provides two main strategies: - Override assembly with high-level code, that way you can rewrite your game bit by bit, testing as you go. - Execution flow recording and code generation. It records execution flow and tells you what was executed. We have a Ghidra plugin to import that data and generate code from it. For example, see https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_OriginalAsm.cs https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_Or.... Interestingly, since it's now C#, you can decompile it again to extract control structures, as shown here https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_DecompiledAsm.cs https://github.com/OpenRakis/LOGO/blob/main/GeneratedCode_De.... Issue with code generation is that ghidra does not like real mode code so the plugin is full of workarounds (and broken at the moment), and there are some fundamental things that can't be done. For instance, self modifying code support is murky with the ghidra plugin. Future Improvements We're working on eliminating Ghidra by generating code directly from Spice86 from the recorded execution trace. Our goal is to fully support self-modifying code and generate functional code from an execution trace with minimal human intervention. I thought I would make a quick reply, but it turns out there's a lot to say on the topic :)

LowLevelMahn 2y ago
so Spice86 helps in translating realmode 16bit DOS code into similar assembler looking like C# code - which got the exact same semantic and behavior but its pure C# - you can then use the normal Visual Studio debugger/IDE to port or extend the code - that is a huge benefit over beeing trapped using >30 years old DOS debuggers, compilers and IDEs for this already very hard work its to ease the process of rewriting assembler code into high level code by having a highlevel-assembler-code in between Spice86 is not intended for Binary exact reversing, like with https://www.scottsmitelli.com/projects/cosmore/ https://www.scottsmitelli.com/projects/cosmore/ or Duke Nukem, ZZT or F15 Eeagle reversing project - these are using the same compiler and try to replicate every byte of the original exe - to prevent any reversing bugs at first - but this is the hardest form of reversing and takes easily years of hard work
alberto-m 2y ago
Thanks for the great answer and for building this tool! Wish I had known about it sooner, I'll surely adopt it in my future retrohacking projects!

johnklos 2y ago

Forty years ago I had a Sinclair QL with an 8086 emulator. Because the Sinclair QL had preemptive multitasking, I could easily search memory for patterns, monitor locations, stop and start the emulation, or change memory programmatically and easily from the QDOS side. It was worlds easier than using a debugger, particularly since I didn't own an 8086 system. I always thought it was a clever way to get insights in to software while it was running that wasn't available to people with 8086 systems, and it's interesting to see this idea so many years later.

mananaysiempre 2y ago
Bochs and MAME both have superb and widely-used debuggers, while Qemu is more limited but still has some debugging capabilities in its monitor, as well as a gdb integration. (Can’t say anything about PCem/86Box.) It seems that developers of emulators targeting good coverage of old stuff simply can’t not build a debugger, because it’s an integral part of their task to figure out what the hell the devs of the latest failing thing did to make it fail. Bochs is (was?) also quite popular in the OSDev scene as a debugging tool.
- rzzzt 2y ago
  DOSBox can be configured to include a debugger. The feature is not enabled in the official binary but the enhanced derivative projects probably have it (DOSBox-X definitely does): - https://www.vogons.org/viewtopic.php?t=3944 https://www.vogons.org/viewtopic.php?t=3944 - https://github.com/joncampbell123/dosbox-x/wiki/DOSBox%E2%80%90X%E2%80%99s-Feature-Highlights https://github.com/joncampbell123/dosbox-x/wiki/DOSBox%E2%80...
  - 2mlWQbCK 2y ago
    DOSBox-x on FreeBSD at least has is disabled by default, but it can be enabled when building from ports (make config). https://github.com/joncampbell123/dosbox-x/blob/master/README.debugger https://github.com/joncampbell123/dosbox-x/blob/master/READM...

DrNosferatu 2y ago

A tutorial on how to reverse engineer a simple DOS game would be absolutely awesome!

pjturpeau 2y ago
https://cosmodoc.org/ https://cosmodoc.org/
alberto-m 2y ago
From my brief experience, it seems that reversing old games is one of those disciplines where there is no good step-by-step course. One can start learning some theory (I did so by reading “The Art of Assembly Language Programming” – if I had more time, I'd try “Reverse Engineering for Beginners”) but then one has to get his hands dirty. Real-life games are typically not simple, but one usually just needs to reverse some small parts to produce new interesting modifications. But surely reading tutorials is useful to learn techniques and tricks. I recommend this article to start: https://www.lodsb.com/reversing-lz91-from-commander-keen https://www.lodsb.com/reversing-lz91-from-commander-keen (not totally for beginners, but very friendly, and it can help to get used with the jargon). I am also publishing war stories on this topic on my blog (marnetto.net).
deleted 2y ago
[deleted]
stevekemp 2y ago
Tutorials are hard, but there are some great writeups like this which discuss some of the specific problems and trial/error involved https://neuviemeporte.github.io/category/f15-se2 https://neuviemeporte.github.io/category/f15-se2 In this case one hard part was trying to get code to compile to identical byte for byte output. Which meant working out which compiler options were used, and which specific compiler too. That gives you a hint of what kinda things are involved.
- genewitch 2y ago
  Have you mentioned this before? I saw a similar context comment maybe 3 days ago on here, down to "byte for byte output [...W]hich compiler options were used"

bernadus_edwin 2y ago

Why are so many emulators written in C#?

sixothree 2y ago
My guess is portability, then obviously performance. edit: actually there is a specific answer for this particular project - "We had to rewrite the project in C# to add automated code generation (java doesn't have the goto keyword, making automated ASM translation challenging)". There you are.
- anyfoo 2y ago
  I mean, that's more or less the reason why it isn't Java, not why it's ultimately C#. My guess is that Java is just what they're most comfortable with, with C# being similar enough but avoiding specific limitations in that case.
  - genewitch 2y ago
    Wasn't C# essentially microsoft throwing their hat in the ring against Oracle and to show off how cool this .net stuff is? I dabbled in both at around the same time a long time ago for console apps and visual studio's autocomplete / assist / library fetch etc made it easier than Java to get working in but... Its been so long I forget the origin stories sometimes.
    - maximilienNoal 2y ago
      .NET began its life as Microsoft's Java (the platform) along with C# being a mix between Java, C++, and Pascal. Considering the timing (early 2000s), one can't help but think that it was in response to the legal actions from Sun Microsystems over their usage of the JVM in Windows. "OK, I'll make my own Java language, with properties and an integrated DSL for queries!"
  - sixothree 2y ago
    I think my question is why not choose c# for this? What’s the apprehension here if any?

gexos 2y ago

[flagged]

eminence32 2y ago

Question from a reverse-engineering noob: Why can't ghidra (or any other reverse engineering tool) be used directly on the .exe? Why do you have to go through this emulator? Is it because the thing you want to debug only runs in x86 realmode?

rzzzt 2y ago
Obfuscation and compression are two potential extra hoops to jump through. It's easier to let the executable run for a bit and start from there.
dmitrygr 2y ago
x86 segmentation makes it very hard to statically analyze anything. In real mode, any byte can be referenced in 4096 different ways. It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless. So, without runtime analysis, there is no way to tell if 04:1234 is or is not the same byte as fa:1204
- jcranmer 2y ago
  > It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless. Actually, my experience is that things are much easier in protected mode. Since selector values are chosen by the OS, that means you rely a lot more on internal relocations. And the use of segment selectors is a strong indicator that you have a pointer in the first place. Unfortunately, ghidra itself struggles to apply these techniques, especially in the decompiler, which seems completely unable to cope with the concept of far pointers.
  - dmitrygr 2y ago
    In DOS, plenty of applications/games load selectors and do nasty things with them so indeed you'd know it is a far pointer, but may not know what to :D
    - 2y ago

ggambetta 2y ago

Oooh, I LOVE this! Especially the ability to "Overriding emulated code with C# code" I had a similar idea years ago (https://gabrielgambetta.com/remakes.html https://gabrielgambetta.com/remakes.html), not in the context of a debugger or reverse engineering per se, but in the context of remakes and "special edition" games. Not entirely surprised that this is a byproduct of OpenRakis. Amazing work!

vunderba 2y ago
I tried doing something like this about 15 years ago but specifically for audio by routing NES NSF rom audio data (square, triangle, PWM, etc) to virtual midi cables attached to VSTs so you could play any old school Nintendo game with modern instrumentation. Was a pretty fun project. The closest thing I can think of for graphical rehauls is probably shader pack type stuff - Minecraft is a great example of this. https://www.sonicether.com/seus https://www.sonicether.com/seus