4 ms·

Are LLMs well suited to this kind of reverse engineering?

viraptor 1y ago
You can definitely do a lot of relabeling that way. It may be also worth trying a loop of "fix until it matches binary" for separate files... But I haven't seen anyone actually write it up. There are attempts like this https://github.com/louisgthier/decompai https://github.com/louisgthier/decompai that are related, but not quite the same as this project. Edit: just gave it a go, and guessing reasonable variable names works extremely well when there's partial information already available. Especially nice for things like naming all the counters and temporaries when you know what you're iterating over (so it's just boring manual and trivial work), but can also figure out the meaning of larger patterns for function names.
- joshschreuder 1y ago
  > It may be also worth trying a loop of "fix until it matches binary" for separate files... But I haven't seen anyone actually write it up. I might be misunderstanding what you mean, but there is decomp-permuter which I think does just this. Rewrites the code in various heuristics and subtle ways to attempt to find a match at function level. https://github.com/simonlindholm/decomp-permuter https://github.com/simonlindholm/decomp-permuter
  - viraptor 1y ago
    Ah yeah, I forgot about this one! Something like that. LLM will be less restricted in choices, but permuter will get you compatible/valid code.
cactusplant7374 1y ago
I'm using an agent to port a game. I have the source. It's not going well. Lots of rabbit holes that are self-inflicted because the LLM doesn't want to port a lot of libraries because it's too much work for one round. It does a lot of stubbing and makes assumptions and that breaks the whole thing.
- viraptor 1y ago
  How did you approach it? Some specific harness? Planning?
hsbauauvhabzb 1y ago
I’ve not experimented but I thought they might be valuable for isolated variable / function renaming
ACCount37 1y ago
Fairly well. They aren't perfect, but they save a lot of time. They are also downright superhuman at recognizing common library functions or spotting well known algorithms, even if badly mangled by compilation and decompilation.
- a96 1y ago
  Really? Are there any articles about that. That sounds really interesting.
  - ACCount37 1y ago
    That's mostly my own experience. The results of reverse engineering can be sexy, but the work itself sure isn't, so we aren't exactly swimming in high quality writeups. But given how much of RE is just going into unfamiliar places and learning unfamiliar things? And how much of it is repetitive but-not-easy-to-automate tasks? I fully expect LLMs to proliferate.
jonhohle 1y ago
I’m not sure if EFF has made any statements about it, but I would be concerned with the copyright aspects. Decompilation works (legally) because a new creative work is being produced. It would be easy to argue LLMs are producing derivative, non-transformative works. AI companies have not made that any easier by paying publishers huge royalties to license training data. I personally would stay away from it to avoid the risk, but I’d imagine if it became “easy” enough to produce a matching decomp with an LLM, we would get a more specific legal precedent pretty quickly.

You can definitely do a lot of relabeling that way. It may be also worth trying a loop of "fix until it matches binary" for separate files... But I haven't seen anyone actually write it up. There are attempts like this https://github.com/louisgthier/decompai https://github.com/louisgthier/decompai that are related, but not quite the same as this project. Edit: just gave it a go, and guessing reasonable variable names works extremely well when there's partial information already available. Especially nice for things like naming all the counters and temporaries when you know what you're iterating over (so it's just boring manual and trivial work), but can also figure out the meaning of larger patterns for function names.

joshschreuder 1y ago
> It may be also worth trying a loop of "fix until it matches binary" for separate files... But I haven't seen anyone actually write it up. I might be misunderstanding what you mean, but there is decomp-permuter which I think does just this. Rewrites the code in various heuristics and subtle ways to attempt to find a match at function level. https://github.com/simonlindholm/decomp-permuter https://github.com/simonlindholm/decomp-permuter
- viraptor 1y ago
  Ah yeah, I forgot about this one! Something like that. LLM will be less restricted in choices, but permuter will get you compatible/valid code.

cactusplant7374 1y ago

I'm using an agent to port a game. I have the source. It's not going well. Lots of rabbit holes that are self-inflicted because the LLM doesn't want to port a lot of libraries because it's too much work for one round. It does a lot of stubbing and makes assumptions and that breaks the whole thing.

viraptor 1y ago
How did you approach it? Some specific harness? Planning?

hsbauauvhabzb 1y ago

I’ve not experimented but I thought they might be valuable for isolated variable / function renaming

ACCount37 1y ago

Fairly well. They aren't perfect, but they save a lot of time. They are also downright superhuman at recognizing common library functions or spotting well known algorithms, even if badly mangled by compilation and decompilation.

a96 1y ago
Really? Are there any articles about that. That sounds really interesting.
- ACCount37 1y ago
  That's mostly my own experience. The results of reverse engineering can be sexy, but the work itself sure isn't, so we aren't exactly swimming in high quality writeups. But given how much of RE is just going into unfamiliar places and learning unfamiliar things? And how much of it is repetitive but-not-easy-to-automate tasks? I fully expect LLMs to proliferate.

jonhohle 1y ago

I’m not sure if EFF has made any statements about it, but I would be concerned with the copyright aspects. Decompilation works (legally) because a new creative work is being produced. It would be easy to argue LLMs are producing derivative, non-transformative works. AI companies have not made that any easier by paying publishers huge royalties to license training data. I personally would stay away from it to avoid the risk, but I’d imagine if it became “easy” enough to produce a matching decomp with an LLM, we would get a more specific legal precedent pretty quickly.