5 ms·

The article contains a reference to a much more impressive task where a user automatically decompiled a binary exe game into Python. But I read their original p

The article contains a reference to a much more impressive task where a user automatically decompiled a binary exe game into Python. But I read their original post and here is what that user said. > Several critics seemed to assume I claimed Claude had "decompiled" the executable in the traditional sense. In reality, as I described in our conversation, it analyzed visible strings and inferred functionality - which is still impressive but different from true decompilation. So I’m not sure that the implications are as big as the article author is claiming. It seems Claude is good at de-minifying JavaScript but that is a long way away from decompiling highly optimized binary code.

haolez 2y ago
But it should be easy to generate such data to train an AI to do that, if someone wants, no?
- kelsey98765431 2y ago
  Replace the word easy there with the phrase "technically may be possible with great effort and expense, probably"
- kees99 2y ago
  Decent optimizing compiler would necessarily loose a fair bit of source information - loop unrolling, function inlining, tail-call optimizations, etc... There is no good way to reconstruct that, AI/ML or not.
  - brookst 2y ago
    For loop unrolling at least, wouldn’t most human reverse engineers see the original loop? It’s far more likely that the source had a loop than that some high level language programmer did the unrolling in source.
- soulofmischief 2y ago
  Some transformations irrecoverably lose information. A recontextualization engine such as an LLM might be able to "recover" some information by comparing it to other code in its training set, but it's still a guess and not all code will have representation in the training set.
  - Kye 2y ago
    I wonder if this would be an ideal use case for diffusion-based LLMs slowly piecing it together with each pass from what it could determine from the last.
    - soulofmischief 2y ago
      I've experimented with building diffusion-based LLMs but didn't scale them up enough to draw any worthwhile conclusions about their strength vs transformer-based LLMs. In this case however, theoretical ability to unminify a piece of code mainly comes down to the training set and amount of relevant context baked into the system, not the model architecture itself.
  - Retr0id 2y ago
    This is exactly what skilled human reverse engineers do. Almost no code is truly novel, at best it's a novel recombination of existing ideas. So yes, some information is lost, but all information relevant to the execution of the program can certainly be recovered.
    - soulofmischief 2y ago
      That's simply untrue, since you don't know if I named a minified variable "x" as "foo", "bar", "mungus" or "chungus". You can guess that it might be mungus based on context, but it could very well be chungus. Recovering lost information is an example of extrapolation, and it is inherently probabilistic. > all information relevant to the execution of the program can certainly be recovered. This is moving goalposts as it has little to do with my original premise.
ninetyninenine 2y ago
[flagged]
- markisus 2y ago
  It's a little hard to understand your criticism. From just reading the original article, I had assumed that Claude could decompile binaries. The author said > Understand dear reader that this technique can be done on any programming language and even from pre-existing binaries themselves. Following that sentence, the author included a twitter embed pointing to a reddit thread about decompiling a binary. Only after I went to the reddit thread, I found that there was no decompilation involved.
Snuggly73 2y ago
Indeed, if you read the attached Claude transcript to the original post, you will see that it finds some strings and infers the functionality (i.e. it has .wav files, thus it probably plays them). https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16 https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16
casey2 2y ago
I don't see why these claims are even being made. It's well known that a transformer could in theory (in practice implementation details prevent this) translate given all the data in, say, a nes cartridge translate it into a git repo on a linux machine that compiles for x86, or even just an elf x86-64 binary directly. Is the claim a general system exists? I'm extremely doubtful of that claim, but one that could do every published nes game to some current linux enviornment? Would definitely be easier than making something like current Claude.
ec109685 2y ago
The original post about Claude decompiling the game lied too. He had Claude gin up a post that would go viral and it added this hallucination: Perfectly replicated the functionality [of the original game] https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16 https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16

haolez 2y ago

But it should be easy to generate such data to train an AI to do that, if someone wants, no?

kelsey98765431 2y ago
Replace the word easy there with the phrase "technically may be possible with great effort and expense, probably"
kees99 2y ago
Decent optimizing compiler would necessarily loose a fair bit of source information - loop unrolling, function inlining, tail-call optimizations, etc... There is no good way to reconstruct that, AI/ML or not.
- brookst 2y ago
  For loop unrolling at least, wouldn’t most human reverse engineers see the original loop? It’s far more likely that the source had a loop than that some high level language programmer did the unrolling in source.
soulofmischief 2y ago
Some transformations irrecoverably lose information. A recontextualization engine such as an LLM might be able to "recover" some information by comparing it to other code in its training set, but it's still a guess and not all code will have representation in the training set.
- Kye 2y ago
  I wonder if this would be an ideal use case for diffusion-based LLMs slowly piecing it together with each pass from what it could determine from the last.
  - soulofmischief 2y ago
    I've experimented with building diffusion-based LLMs but didn't scale them up enough to draw any worthwhile conclusions about their strength vs transformer-based LLMs. In this case however, theoretical ability to unminify a piece of code mainly comes down to the training set and amount of relevant context baked into the system, not the model architecture itself.
- Retr0id 2y ago
  This is exactly what skilled human reverse engineers do. Almost no code is truly novel, at best it's a novel recombination of existing ideas. So yes, some information is lost, but all information relevant to the execution of the program can certainly be recovered.
  - soulofmischief 2y ago
    That's simply untrue, since you don't know if I named a minified variable "x" as "foo", "bar", "mungus" or "chungus". You can guess that it might be mungus based on context, but it could very well be chungus. Recovering lost information is an example of extrapolation, and it is inherently probabilistic. > all information relevant to the execution of the program can certainly be recovered. This is moving goalposts as it has little to do with my original premise.

ninetyninenine 2y ago

[flagged]

markisus 2y ago
It's a little hard to understand your criticism. From just reading the original article, I had assumed that Claude could decompile binaries. The author said > Understand dear reader that this technique can be done on any programming language and even from pre-existing binaries themselves. Following that sentence, the author included a twitter embed pointing to a reddit thread about decompiling a binary. Only after I went to the reddit thread, I found that there was no decompilation involved.

Snuggly73 2y ago

Indeed, if you read the attached Claude transcript to the original post, you will see that it finds some strings and infers the functionality (i.e. it has .wav files, thus it probably plays them). https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16 https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16

casey2 2y ago

I don't see why these claims are even being made. It's well known that a transformer could in theory (in practice implementation details prevent this) translate given all the data in, say, a nes cartridge translate it into a git repo on a linux machine that compiles for x86, or even just an elf x86-64 binary directly. Is the claim a general system exists? I'm extremely doubtful of that claim, but one that could do every published nes game to some current linux enviornment? Would definitely be easier than making something like current Claude.

ec109685 2y ago

The original post about Claude decompiling the game lied too. He had Claude gin up a post that would go viral and it added this hallucination: Perfectly replicated the functionality [of the original game] https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16 https://claude.ai/share/3eecebc5-ff9a-4363-a1e6-e5c245b81a16