3 ms·

Decomp me: Collaboratively decompile code in the browser

60 points by anotherhue 3y ago

thelastparadise 3y ago
Neat. Makes me wonder if maybe this is a great domain for generating synthetic data to train models to at least partially automate this process.
- anotherhue 3y ago
  I wondered the same, lang-to-lang should be in an LLM's wheelhouse, and the built in correctness checking should prevent hallucination.
- ethteck 3y ago
  It's something a few of us have looked into, but it's a very complex problem with a lot of sub-problems to solve. Hopefully over the next few years, some new literature will emerge that might help.
Acrobatic_Road 3y ago
How can I get into decompilation? What skills do I need?
- Jtsummers 3y ago
  Learn to use the tooling (Ghidra is nice and also free), learn to read assembly and how to look up assembly instructions and understand the documentation on them (there are a lot of assembly instructions for each platform, you will likely not memorize them all nor memorize every ISA). It can also help to know how to write some assembly. With your favorite compiler, use whatever flag it offers to spit out assembly listings from its compilation process. On clang and gcc this is `-S`. Typically used like `gcc foo.c -S` and it'll produce a file `foo.S` which contains the assembly listings. Or use Compiler Explorer (aka "godbolt") https://godbolt.org/ https://godbolt.org/. Great resource, supports many languages and targets, has good highlighting of what code in the origin language became what assembly. That last bit will help you gain an intuition for what code looks like when it gets converted into assembly language. This is helpful for going in the other direction, it lets you at least guess at something. There are probably some books or other resources people can recommend, I just picked it up as I went but I'm also just a "dabbler" in the field. It's not my day job and I have no intentions of making it my day job so I've learned enough to accomplish my goals or satisfy my curiosity.
  - notbeuller 3y ago
    (An easier way I've found is to compile an entire project with --save-temps, for c, objc, or c++. Makes it easy to instrument a lot of files at once.)
  - ethteck 3y ago
    It's unfortunately a day job for none of us. I think we'd all be very elated if there was some way to turn this into a career, but matching decompilation is not really something that companies are asking for. It probably doesn't help that it's such a relatively new method of decompilation.
    - Jtsummers 3y ago
      It’s a day job (or was, lost touch) for a couple people I have known. But it is a small niche.
      - bkallus 3y ago
        Definitely still a day job for some people. These DARPA and IARPA programs all have a decompilation component: 1. https://www.darpa.mil/program/assured-micropatching https://www.darpa.mil/program/assured-micropatching 2. https://www.iarpa.gov/newsroom/article/annotated-malicious-binaries-and-surrogate-source-code-for-authorship-exploration-rfi https://www.iarpa.gov/newsroom/article/annotated-malicious-b... 3. https://www.darpa.mil/program/recovery-of-symbolic-mathematics-from-code https://www.darpa.mil/program/recovery-of-symbolic-mathemati...
    - upon_drumhead 3y ago
      Having the tools handy means sometimes you'll still get the opportunity to use them. I was able to fix a 3rd party binary that was broken on our platform via dis-assembly and binary patching. Having the tools in the toolbox to do so allowed us to solve it rapidly when normally it'd take months to work with the vendor.
    - foldor 3y ago
      There is at least one of us with it as a day job at Nightdive!
nneonneo 3y ago
There was a competition held a while back called Decompetition (https://decompetition.io/ https://decompetition.io/) in the same spirit: given a bunch of compiled programs, write the source code that compiles to the same (or very close) binary. Most languages/compilers were quite reasonable to work with. Swift had some issues where the line number of the source statement mattered (it would be integrated into inline error reporting routines), and compiled Nim code was just really hard to read. The brutal one was Golang: the register allocator seems to depend on the ordering of the local symbol table, meaning that changing the name of a variable could end up changing which registers got used for completely unrelated variables. In any case, it was an interesting exercise which demonstrated that full, 1:1 decompilation is really possible. I'm super happy to see a proper use-case for this approach implemented here!
- circuit10 3y ago
  Matching decompilation is not only possible but the whole of Super Mario 64 has been decompiled with it, to the point that it can be ported natively to other platforms: https://github.com/n64decomp/sm64 https://github.com/n64decomp/sm64

60 points by anotherhue 3y ago

Neat. Makes me wonder if maybe this is a great domain for generating synthetic data to train models to at least partially automate this process.

anotherhue 3y ago
I wondered the same, lang-to-lang should be in an LLM's wheelhouse, and the built in correctness checking should prevent hallucination.
ethteck 3y ago
It's something a few of us have looked into, but it's a very complex problem with a lot of sub-problems to solve. Hopefully over the next few years, some new literature will emerge that might help.

Acrobatic_Road 3y ago

How can I get into decompilation? What skills do I need?

Jtsummers 3y ago
Learn to use the tooling (Ghidra is nice and also free), learn to read assembly and how to look up assembly instructions and understand the documentation on them (there are a lot of assembly instructions for each platform, you will likely not memorize them all nor memorize every ISA). It can also help to know how to write some assembly. With your favorite compiler, use whatever flag it offers to spit out assembly listings from its compilation process. On clang and gcc this is `-S`. Typically used like `gcc foo.c -S` and it'll produce a file `foo.S` which contains the assembly listings. Or use Compiler Explorer (aka "godbolt") https://godbolt.org/ https://godbolt.org/. Great resource, supports many languages and targets, has good highlighting of what code in the origin language became what assembly. That last bit will help you gain an intuition for what code looks like when it gets converted into assembly language. This is helpful for going in the other direction, it lets you at least guess at something. There are probably some books or other resources people can recommend, I just picked it up as I went but I'm also just a "dabbler" in the field. It's not my day job and I have no intentions of making it my day job so I've learned enough to accomplish my goals or satisfy my curiosity.
- notbeuller 3y ago
  (An easier way I've found is to compile an entire project with --save-temps, for c, objc, or c++. Makes it easy to instrument a lot of files at once.)
- ethteck 3y ago
  It's unfortunately a day job for none of us. I think we'd all be very elated if there was some way to turn this into a career, but matching decompilation is not really something that companies are asking for. It probably doesn't help that it's such a relatively new method of decompilation.
  - Jtsummers 3y ago
    It’s a day job (or was, lost touch) for a couple people I have known. But it is a small niche.
    - bkallus 3y ago
      Definitely still a day job for some people. These DARPA and IARPA programs all have a decompilation component: 1. https://www.darpa.mil/program/assured-micropatching https://www.darpa.mil/program/assured-micropatching 2. https://www.iarpa.gov/newsroom/article/annotated-malicious-binaries-and-surrogate-source-code-for-authorship-exploration-rfi https://www.iarpa.gov/newsroom/article/annotated-malicious-b... 3. https://www.darpa.mil/program/recovery-of-symbolic-mathematics-from-code https://www.darpa.mil/program/recovery-of-symbolic-mathemati...
  - upon_drumhead 3y ago
    Having the tools handy means sometimes you'll still get the opportunity to use them. I was able to fix a 3rd party binary that was broken on our platform via dis-assembly and binary patching. Having the tools in the toolbox to do so allowed us to solve it rapidly when normally it'd take months to work with the vendor.
  - foldor 3y ago
    There is at least one of us with it as a day job at Nightdive!

nneonneo 3y ago

There was a competition held a while back called Decompetition (https://decompetition.io/ https://decompetition.io/) in the same spirit: given a bunch of compiled programs, write the source code that compiles to the same (or very close) binary. Most languages/compilers were quite reasonable to work with. Swift had some issues where the line number of the source statement mattered (it would be integrated into inline error reporting routines), and compiled Nim code was just really hard to read. The brutal one was Golang: the register allocator seems to depend on the ordering of the local symbol table, meaning that changing the name of a variable could end up changing which registers got used for completely unrelated variables. In any case, it was an interesting exercise which demonstrated that full, 1:1 decompilation is really possible. I'm super happy to see a proper use-case for this approach implemented here!

circuit10 3y ago
Matching decompilation is not only possible but the whole of Super Mario 64 has been decompiled with it, to the point that it can be ported natively to other platforms: https://github.com/n64decomp/sm64 https://github.com/n64decomp/sm64