6 ms·

Show HN: A Ghidra extension for exporting parts of a program as object files

304 points by boricj 2y ago

This Ghidra extension unrelocates machine code through analysis and then synthesizes a working object file from a listing selection. It effectively turns computer programs into Lego bricks, to be torn down into pieces and reused into something new. It supports the COFF and ELF object file formats, for the x86 and MIPS architectures. It has been successfully used on Linux, Windows and PlayStation executables. One user report is on a commercial video game from 2009 with a ~7 MiB Windows executable written in C++: it was delinked without its C runtime library and then relinked into a new executable at a different base address, with no visible change in functionality, as a prelude to a decompilation project. Use-cases I've demonstrated on my blog include modding, making software ports, converting executable file formats, creating libraries... I've originally built this as part of a video game decompilation project ; I've been working on this over the past 2.5 years and recently it has started gaining some users besides me.

bigdict 2y ago
See also: objcopy. https://sourceware.org/binutils/docs/binutils/objcopy.html https://sourceware.org/binutils/docs/binutils/objcopy.html
- boricj 2y ago
  While objcopy can do many things, it can't undo the work of the linker. If relocations aren't unapplied and a new relocation table generated, these spots inside the new object file will reference the original program's address space, leading to some exotic undefined behavior. Delinking is a subject with very few resources online, but there are a couple of other tools for it out there: - https://github.com/endrazine/wcc - https://github.com/jonwil/unlinkerida - https://github.com/jnider/delinker
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]
jchw 2y ago
Oh, great to see this here. I think this is an extremely cool project, and I helped to add MS COFF support. (P.S.: I will note that my initial PR was notably worse than the ELF support that was already present, so if you run into problems with it... probably my fault :P I can see it is being improved, though.) That said, I haven't done anything big with it yet. The most fun I had was delinking a Hello World executable compiled with Visual Studio 2003, relinking it to Linux x86 with GCC+glibc, and then relinking that to MinGW+msvcrt again. Doing anything larger than hello world is a bit beyond me yet, though, in part because I'm actually a pretty big n00b when it comes to Ghidra and haven't even really figured out a good way to select the ranges for delinking from a large binary. I should've probably asked someone by now, but oh well. :) Coincidentally, a derivation for this just got merged into Nixpkgs earlier today, so if you're using NixOS unstable it's possible to install it using ghidra.withExtensions; it is under ghidra-extensions.ghidra-delinker-extension. Only one problem: There was a new version released a few days ago and I didn't rebase my PR, so it is out of date. I will try to push an update soon.
- boricj 2y ago
  > I'm actually a pretty big n00b when it comes to Ghidra and haven't even really figured out a good way to select the ranges for delinking from a large binary. One way to keep track of things to delink is to use folders and fragments inside a program tree. For example, I have a Ghidra program where I've figured out the name and ranges of the various object files that originally made up the executable. These folders or fragments can then be selected as a whole with right-click > Select Addresses. The relocation synthesizer analyzer and the exporter can also be scripted, either independently or using the program's tree manager. This removes the need to select by hand the ranges you want as well as invoking manually the analyzer and the exporter.
toomuchtodo 2y ago
Previous: Show HN: A Ghidra extension that turns programs back into object files - https://news.ycombinator.com/item?id=38852362 https://news.ycombinator.com/item?id=38852362 - Jan 2024 (4 comments)
- deleted 2y ago
  [deleted]
mhh__ 2y ago
It might be interesting to tie this into something I had a daydream about once and then never bothered to actually do: generate header files from debug info (and then possibly have some LLM tidy it up)
- jchw 2y ago
  Actually there are a few attempts at this! Here's one for Microsoft Program Database: https://github.com/wbenny/pdbex https://github.com/wbenny/pdbex As for using an LLM to tidy it up... It doesn't seem like there has been a ton of success applying LLM models to reverse engineering yet... A part of me is wondering if this will wind up being a place where the LLM architecture proves insufficient. I'm not an expert but if I had to place a bet I'd bet on diffusion models being more interesting for a lot of reverse engineering use cases. That said, it's not really the same thing, but with Binary Ninja they have a feature called Sidekick that uses an LLM to try to clean up the disassembly; I'm kind of unimpressed but maybe it is useful to somebody.
  - dvdkon 2y ago
    I'll add my attempt here: https://gitlab.com/dvdkon/pdb2hpp https://gitlab.com/dvdkon/pdb2hpp Its output is kind of ugly, limited by limitations of either the PDB format or Microsoft's terrible parser library, but I've successfully used it for calling functions from a proprietary DLL.
  - mhh__ 2y ago
    The idea is more that you use the LLM for quick guesses about behaviour/names and so on rather than actually relying on it for reverse engineering as per se.
- boricj 2y ago
  Tangentially, I've considered generating debugging symbols for the exported object files, based on the contents of the Ghidra database, in order to improve the debugging experience when using them. I haven't implemented that feature yet because so far I've managed to get by without it. Also, it sounds like a rather deep rabbit hole to fall into and the one I'm currently inside of is big enough as it is.
almostgotcaught 2y ago
So is this a completely fool-proof process? Ie i'm asking if it's guaranteed to succeed or if the analysis is conservative. Ie if some piece/datum/feature is missing in the ELF then the delinking will fail?
- boricj 2y ago
  > So is this a completely fool-proof process? That's... complicated to answer. My analyzers rely on an accurate Ghidra database, at least for the parts you want to export. While I've put a fair amount of effort into logging the various issues than can crop up which require fixing, they can't see what isn't there. In particular, missing references and truncation of variables won't be detected and will result in exotic undefined behavior. There are ways to track down some of these issues. The best I've found so far is to relink the executable at a different base address and making sure that the original program's address ranges are unmapped ; that should lead to segmentation faults when absolute relocation spots are missed that can be debugged (but that only works if your target has a MMU). Truncated variables are very tricky to troubleshoot (especially if you don't suspect it) since it's the memory following the truncated variable that gets corrupted. An integer that is mistaken for a pointer can also be very tricky to track down, as the integer's value will vary depending on the address the target symbol gets, leading to erratic program behavior (that's especially an issue for a program loaded very low in the address space). That being said, if the Ghidra database is accurate enough and you export back to the same object file format used originally and you subsequently use it onto the same platform with the same toolchain, you can delink megabytes of program code and data successfully. I consider that if the linker did it, then it should be possible to undo it. Now, if you start cross-delinking to something that doesn't match the original program's platform and toolchain (like delinking from a Linux i386 ELF executable into a COFF object file and using it with a i386 Windows toolchain) then it's another story. If the exporter can express the relocations then you might end up with a working relocatable object file, but you'll still have potentially mismatched ABIs to contend with. It can be done, but that's not something I would recommend as a first project. TL;DR Depending on what you do and the accuracy of the Ghidra database, it can range from "it just works" all the way to praying to Cthulhu for mercy.
- archgoon 2y ago
jxjx 2y ago
This sounds very interesting. And is tempting me to delve back into a game reverse engineering project I abandoned a few years back. Do you have a fully worked example of how to use this and then how to make use of its output? Would love to see an end-to-end walkthrough.
- boricj 2y ago
  There are links to various case studies on my blog inside the README of the repository. https://github.com/boricj/ghidra-delinker-extension/blob/master/README.md https://github.com/boricj/ghidra-delinker-extension/blob/mas...
sweeter 2y ago
That sounds like magic, I'm not going to lie. I have to understand how this is possible.
- jchw 2y ago
  It is definitely not going to be easy to do for every CPU architecture, but it's not as ridiculous as it seems. Basically, the difference between an "object file" and an executable file or shared library is not very large. Often times, except for Microsoft platforms, they are in the same actual file format, e.g. ELF. The big thing is relocations. Object files have granular relocations that executable files don't; at least on Windows, executable images just have minimal relocations that point to addresses of code and data that will need to be fixed up if the executable is relocated, but the executable image itself is only able to be relocated with regards to its image base address as a whole unit. In contrast, object files contain symbol-level relocations. To be able to accurately reconstruct this information, you need to annotate the disassembly with somewhat accurate information about symbols. The other big difference with an object file is well, it is not linked. None of the symbols are "resolved". This is particularly easy to fix actually: during delinking if the symbol is outside of the current scope then it just needs to be replaced with an unresolved symbol, pretty much. Then when relinking, another object file or library needs to provide the symbol so that it can be linked back up. (There are a few smaller differences too, like the lack of an entrypoint, but it really isn't a whole lot of significance.) This means that the boundaries for which you carve object files out of an executable image or shared object is actually completely arbitrary. It obviously was segmented into translation units when it was originally compiled, but nothing really cares about those boundaries at the linking stage. (Of course, you probably want to try to figure it out if possible, since it will probably be very hard to do a matching decompilation if your object boundaries are incorrect.) Take this with a grain of salt as despite having literally worked on this problem I feel like I might be messing up some of the details a bit. I've been meaning to write a blog post about object files, though there are actually a couple of good ones already floating around.
  - fngjdflmdflg 2y ago
    >It obviously was segmented into translation units when it was originally compiled, but nothing really cares about those boundaries at the linking stage. (Of course, you probably want to try to figure it out if possible, since it will probably be very hard to do a matching decompilation if your object boundaries are incorrect.) Can you expound on this? Why are the boundaries not important at the linking stage - aren't you linking the wrong code then? Or did I not understand the point here?
hmfrh 2y ago
How much work is it to figure out which sections of the executable to export? Would it be realistic to be able to export a modern-ish (2008-2015) Win32 game into objects and then compile/link it into a full executable again with less than a few hours work?
- boricj 2y ago
  > How much work is it to figure out which sections of the executable to export? As long as you do not cut across a variable or a function, you can export pretty much however you want, you don't have to follow the original object file boundaries. What to export is a separate matter and requires some knowledge about the program. Having debugging symbols makes this much easier, otherwise by the time you've made the Ghidra database accurate enough for exportation you'll usually have an idea of where's what. > Would it be realistic to be able to export a modern-ish (2008-2015) Win32 game into objects and then compile/link it into a full executable again with less than a few hours work? About the user report in my submission, they first raised an issue in early July and by mid-August they got a fully working, functionally identical relinked executable. To be fair, the COFF exporter had a lot of bugs that needed to be fixed and the i386 analyzer needed some touch-ups, things that somebody else should hopefully won't stumble over now. I don't know how long it would take, but unless you have debugging symbols and are really lucky it will take more than a few hours of work. A skilled reverse-engineer can probably manage to get something executing in that timeframe (even if it crashes halfway during the first loading screen), but it's one of these tasks that you won't know when it will be done until it is done.
  - hmfrh 2y ago
    > As long as you do not cut across a variable or a function, you can export pretty much however you want, you don't have to follow the original object file boundaries. Would it be possible to export basically the entire program at once and then slice off individual functions one by one? Do you have any guides/examples of the > Decompilation projects, by splitting a program into multiple object files and reimplementing these Ship of Theseus-style style project?
pinum 2y ago
This looks fantastic and is relevant to some game modding ideas I've had. I love your blog series about decompiling Tenchu too. Thank you for releasing this stuff!
- boricj 2y ago
  Thanks! I should get back to this project one of these days. I did one version tracking session too many in a row and had to take a break, plus that delinking side-quest keeps snowballing out of control.
jcul 2y ago
This looks really cool. I don't have a real use for it in my current work but in a past life it would have been so useful. Hopefully I'll have some time / opportunity to try it out soon.
- egberts1 2y ago
  "... past life ..."? Are you some kind of a ressurectionist?
123456789 2y ago
[dead]
luke-stanley 2y ago
This seems cool. Maybe one day it could help people take apart existing programs and use them how they want in a more accessible way. There is research like Meta's LLM Compiler and decompiling with LLM's, breaking apart programs and replacing parts of them would be an interesting, data rich space for LLM's to explore and maybe self-improve at! For people training, there are lots of interesting tokens lurking there, and interesting things to do with them!

304 points by boricj 2y ago

bigdict 2y ago

See also: objcopy. https://sourceware.org/binutils/docs/binutils/objcopy.html https://sourceware.org/binutils/docs/binutils/objcopy.html

boricj 2y ago
While objcopy can do many things, it can't undo the work of the linker. If relocations aren't unapplied and a new relocation table generated, these spots inside the new object file will reference the original program's address space, leading to some exotic undefined behavior. Delinking is a subject with very few resources online, but there are a couple of other tools for it out there: - https://github.com/endrazine/wcc - https://github.com/jonwil/unlinkerida - https://github.com/jnider/delinker
deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

jchw 2y ago

Oh, great to see this here. I think this is an extremely cool project, and I helped to add MS COFF support. (P.S.: I will note that my initial PR was notably worse than the ELF support that was already present, so if you run into problems with it... probably my fault :P I can see it is being improved, though.) That said, I haven't done anything big with it yet. The most fun I had was delinking a Hello World executable compiled with Visual Studio 2003, relinking it to Linux x86 with GCC+glibc, and then relinking that to MinGW+msvcrt again. Doing anything larger than hello world is a bit beyond me yet, though, in part because I'm actually a pretty big n00b when it comes to Ghidra and haven't even really figured out a good way to select the ranges for delinking from a large binary. I should've probably asked someone by now, but oh well. :) Coincidentally, a derivation for this just got merged into Nixpkgs earlier today, so if you're using NixOS unstable it's possible to install it using ghidra.withExtensions; it is under ghidra-extensions.ghidra-delinker-extension. Only one problem: There was a new version released a few days ago and I didn't rebase my PR, so it is out of date. I will try to push an update soon.

boricj 2y ago
> I'm actually a pretty big n00b when it comes to Ghidra and haven't even really figured out a good way to select the ranges for delinking from a large binary. One way to keep track of things to delink is to use folders and fragments inside a program tree. For example, I have a Ghidra program where I've figured out the name and ranges of the various object files that originally made up the executable. These folders or fragments can then be selected as a whole with right-click > Select Addresses. The relocation synthesizer analyzer and the exporter can also be scripted, either independently or using the program's tree manager. This removes the need to select by hand the ranges you want as well as invoking manually the analyzer and the exporter.

toomuchtodo 2y ago

Previous: Show HN: A Ghidra extension that turns programs back into object files - https://news.ycombinator.com/item?id=38852362 https://news.ycombinator.com/item?id=38852362 - Jan 2024 (4 comments)

deleted 2y ago
[deleted]

mhh__ 2y ago

It might be interesting to tie this into something I had a daydream about once and then never bothered to actually do: generate header files from debug info (and then possibly have some LLM tidy it up)

jchw 2y ago
Actually there are a few attempts at this! Here's one for Microsoft Program Database: https://github.com/wbenny/pdbex https://github.com/wbenny/pdbex As for using an LLM to tidy it up... It doesn't seem like there has been a ton of success applying LLM models to reverse engineering yet... A part of me is wondering if this will wind up being a place where the LLM architecture proves insufficient. I'm not an expert but if I had to place a bet I'd bet on diffusion models being more interesting for a lot of reverse engineering use cases. That said, it's not really the same thing, but with Binary Ninja they have a feature called Sidekick that uses an LLM to try to clean up the disassembly; I'm kind of unimpressed but maybe it is useful to somebody.
- dvdkon 2y ago
  I'll add my attempt here: https://gitlab.com/dvdkon/pdb2hpp https://gitlab.com/dvdkon/pdb2hpp Its output is kind of ugly, limited by limitations of either the PDB format or Microsoft's terrible parser library, but I've successfully used it for calling functions from a proprietary DLL.
- mhh__ 2y ago
  The idea is more that you use the LLM for quick guesses about behaviour/names and so on rather than actually relying on it for reverse engineering as per se.
boricj 2y ago
Tangentially, I've considered generating debugging symbols for the exported object files, based on the contents of the Ghidra database, in order to improve the debugging experience when using them. I haven't implemented that feature yet because so far I've managed to get by without it. Also, it sounds like a rather deep rabbit hole to fall into and the one I'm currently inside of is big enough as it is.

almostgotcaught 2y ago

So is this a completely fool-proof process? Ie i'm asking if it's guaranteed to succeed or if the analysis is conservative. Ie if some piece/datum/feature is missing in the ELF then the delinking will fail?

boricj 2y ago
> So is this a completely fool-proof process? That's... complicated to answer. My analyzers rely on an accurate Ghidra database, at least for the parts you want to export. While I've put a fair amount of effort into logging the various issues than can crop up which require fixing, they can't see what isn't there. In particular, missing references and truncation of variables won't be detected and will result in exotic undefined behavior. There are ways to track down some of these issues. The best I've found so far is to relink the executable at a different base address and making sure that the original program's address ranges are unmapped ; that should lead to segmentation faults when absolute relocation spots are missed that can be debugged (but that only works if your target has a MMU). Truncated variables are very tricky to troubleshoot (especially if you don't suspect it) since it's the memory following the truncated variable that gets corrupted. An integer that is mistaken for a pointer can also be very tricky to track down, as the integer's value will vary depending on the address the target symbol gets, leading to erratic program behavior (that's especially an issue for a program loaded very low in the address space). That being said, if the Ghidra database is accurate enough and you export back to the same object file format used originally and you subsequently use it onto the same platform with the same toolchain, you can delink megabytes of program code and data successfully. I consider that if the linker did it, then it should be possible to undo it. Now, if you start cross-delinking to something that doesn't match the original program's platform and toolchain (like delinking from a Linux i386 ELF executable into a COFF object file and using it with a i386 Windows toolchain) then it's another story. If the exporter can express the relocations then you might end up with a working relocatable object file, but you'll still have potentially mismatched ABIs to contend with. It can be done, but that's not something I would recommend as a first project. TL;DR Depending on what you do and the accuracy of the Ghidra database, it can range from "it just works" all the way to praying to Cthulhu for mercy.
archgoon 2y ago

jxjx 2y ago

This sounds very interesting. And is tempting me to delve back into a game reverse engineering project I abandoned a few years back. Do you have a fully worked example of how to use this and then how to make use of its output? Would love to see an end-to-end walkthrough.

boricj 2y ago
There are links to various case studies on my blog inside the README of the repository. https://github.com/boricj/ghidra-delinker-extension/blob/master/README.md https://github.com/boricj/ghidra-delinker-extension/blob/mas...

sweeter 2y ago

That sounds like magic, I'm not going to lie. I have to understand how this is possible.

jchw 2y ago
It is definitely not going to be easy to do for every CPU architecture, but it's not as ridiculous as it seems. Basically, the difference between an "object file" and an executable file or shared library is not very large. Often times, except for Microsoft platforms, they are in the same actual file format, e.g. ELF. The big thing is relocations. Object files have granular relocations that executable files don't; at least on Windows, executable images just have minimal relocations that point to addresses of code and data that will need to be fixed up if the executable is relocated, but the executable image itself is only able to be relocated with regards to its image base address as a whole unit. In contrast, object files contain symbol-level relocations. To be able to accurately reconstruct this information, you need to annotate the disassembly with somewhat accurate information about symbols. The other big difference with an object file is well, it is not linked. None of the symbols are "resolved". This is particularly easy to fix actually: during delinking if the symbol is outside of the current scope then it just needs to be replaced with an unresolved symbol, pretty much. Then when relinking, another object file or library needs to provide the symbol so that it can be linked back up. (There are a few smaller differences too, like the lack of an entrypoint, but it really isn't a whole lot of significance.) This means that the boundaries for which you carve object files out of an executable image or shared object is actually completely arbitrary. It obviously was segmented into translation units when it was originally compiled, but nothing really cares about those boundaries at the linking stage. (Of course, you probably want to try to figure it out if possible, since it will probably be very hard to do a matching decompilation if your object boundaries are incorrect.) Take this with a grain of salt as despite having literally worked on this problem I feel like I might be messing up some of the details a bit. I've been meaning to write a blog post about object files, though there are actually a couple of good ones already floating around.
- fngjdflmdflg 2y ago
  >It obviously was segmented into translation units when it was originally compiled, but nothing really cares about those boundaries at the linking stage. (Of course, you probably want to try to figure it out if possible, since it will probably be very hard to do a matching decompilation if your object boundaries are incorrect.) Can you expound on this? Why are the boundaries not important at the linking stage - aren't you linking the wrong code then? Or did I not understand the point here?

hmfrh 2y ago

How much work is it to figure out which sections of the executable to export? Would it be realistic to be able to export a modern-ish (2008-2015) Win32 game into objects and then compile/link it into a full executable again with less than a few hours work?

boricj 2y ago
> How much work is it to figure out which sections of the executable to export? As long as you do not cut across a variable or a function, you can export pretty much however you want, you don't have to follow the original object file boundaries. What to export is a separate matter and requires some knowledge about the program. Having debugging symbols makes this much easier, otherwise by the time you've made the Ghidra database accurate enough for exportation you'll usually have an idea of where's what. > Would it be realistic to be able to export a modern-ish (2008-2015) Win32 game into objects and then compile/link it into a full executable again with less than a few hours work? About the user report in my submission, they first raised an issue in early July and by mid-August they got a fully working, functionally identical relinked executable. To be fair, the COFF exporter had a lot of bugs that needed to be fixed and the i386 analyzer needed some touch-ups, things that somebody else should hopefully won't stumble over now. I don't know how long it would take, but unless you have debugging symbols and are really lucky it will take more than a few hours of work. A skilled reverse-engineer can probably manage to get something executing in that timeframe (even if it crashes halfway during the first loading screen), but it's one of these tasks that you won't know when it will be done until it is done.
- hmfrh 2y ago
  > As long as you do not cut across a variable or a function, you can export pretty much however you want, you don't have to follow the original object file boundaries. Would it be possible to export basically the entire program at once and then slice off individual functions one by one? Do you have any guides/examples of the > Decompilation projects, by splitting a program into multiple object files and reimplementing these Ship of Theseus-style style project?

pinum 2y ago

This looks fantastic and is relevant to some game modding ideas I've had. I love your blog series about decompiling Tenchu too. Thank you for releasing this stuff!

boricj 2y ago
Thanks! I should get back to this project one of these days. I did one version tracking session too many in a row and had to take a break, plus that delinking side-quest keeps snowballing out of control.

jcul 2y ago

This looks really cool. I don't have a real use for it in my current work but in a past life it would have been so useful. Hopefully I'll have some time / opportunity to try it out soon.

egberts1 2y ago
"... past life ..."? Are you some kind of a ressurectionist?

123456789 2y ago

[dead]

luke-stanley 2y ago

This seems cool. Maybe one day it could help people take apart existing programs and use them how they want in a more accessible way. There is research like Meta's LLM Compiler and decompiling with LLM's, breaking apart programs and replacing parts of them would be an interesting, data rich space for LLM's to explore and maybe self-improve at! For people training, there are lots of interesting tokens lurking there, and interesting things to do with them!