3 ms·

I agree, I am aware that LLMs can synthesize data, I am a huge advocate of them for this purpose and use them daily for such. This is where I derive the authori

I agree, I am aware that LLMs can synthesize data, I am a huge advocate of them for this purpose and use them daily for such. This is where I derive the authority in making such a statement about them. I never made any statement relating the performance of LLMs to humans. > Stuff like variable names won’t be exact Yes, this is the point of my comment around information loss. Lossy transformations inherently cannot be recovered with absolute certainty, and the degree of certainty depends on available context. This is true for absolutely any system which produces lossy transformations, not just in the context of code and LLMs, but encoders, etc.

smallmancontrov 2y ago
Sure, but the ability to synthesize data challenges the implicit assumption that all reconstruction error is bad. Who is to say the LLM won't come up with better variable names than the original source? It probably won't, but I could see it often hitting roughly equal quality. The reconstruction error metric would complain but a human would not, so the problem is now with reconstruction error as a metric.
- soulofmischief 2y ago
  > Who is to say the LLM won't come up with better variable names than the original source? This is still a loss of information. You're completely misunderstanding the point: Minification is an irreversible process unless you happen to have exactly the context you need, and can verify the provenance of the source in question. Coming up with "better" variable names is not recovering information, it's extrapolating information about the old state to the point of error.
brookst 2y ago
Sure, but for the purposes of decompilation, does it matter if a variable name is exact versus merely being a good description of its function? The issue isn’t an idealized “let’s get the source including meaningless whitespace”, it’s “let’s construct source that can be used interchangeably with the original”. For that purpose there is no reason LLMs can’t be as good as the best humans.
- soulofmischief 2y ago
  No, it doesn't matter, and as I explained above, I never mentioned humans once in my comment, nor did I make any comparisons. I simply described a mathematical fact that a lot of people are having trouble understanding and digesting. I do appreciate your comment as it's not defensive, it's made in earnest, so please don't take my above statement to be a reflection of you in particular.
- nyrikki 2y ago
  > “let’s construct source that can be used interchangeably with the original”. You lose a lot more than whitespace, you lose semantics, metadata, and lots of other information that makes it so Obviously byte coded languages are a bit easier, and you may get lucky and experts help a lot, and perhaps LLMS can help a little. Both Rice's theorem and the system identification problem from the cybernetics days relate to why it is so hard. > Given a system in the form of a black box (BB) allowing finite input-output interactions, deduce a complete specification of the system’s machine table (i.e., algorithm or internal dynamics). AND > Given a complete specification of a machine table (i.e., algorithm or internal dynamics), recognize any BB having that description. Decompilation doesn't give you the equivalent of the original source code, it gives you new source code that appears to be functionally similar to it, and many people who has been forced to use decompilation to recover from lost source code or consultant time-bombs have run into the problems that admittently seem pretty counter intuitive. Remember that Rice-Shapiro and Kreisel-Lacombe-Shoenfield-Tseitin extend Rice to partial and total functions in finite time. Telling if a program is equivalent to a fixed other program, even for total functions is still undecidable as they are 'non-trivial' properties. Most of the time you can make it work, but it isn't a case of: > “let’s construct source that can be used interchangeably with the original”

smallmancontrov 2y ago

Sure, but the ability to synthesize data challenges the implicit assumption that all reconstruction error is bad. Who is to say the LLM won't come up with better variable names than the original source? It probably won't, but I could see it often hitting roughly equal quality. The reconstruction error metric would complain but a human would not, so the problem is now with reconstruction error as a metric.

soulofmischief 2y ago
> Who is to say the LLM won't come up with better variable names than the original source? This is still a loss of information. You're completely misunderstanding the point: Minification is an irreversible process unless you happen to have exactly the context you need, and can verify the provenance of the source in question. Coming up with "better" variable names is not recovering information, it's extrapolating information about the old state to the point of error.

brookst 2y ago

Sure, but for the purposes of decompilation, does it matter if a variable name is exact versus merely being a good description of its function? The issue isn’t an idealized “let’s get the source including meaningless whitespace”, it’s “let’s construct source that can be used interchangeably with the original”. For that purpose there is no reason LLMs can’t be as good as the best humans.

soulofmischief 2y ago
No, it doesn't matter, and as I explained above, I never mentioned humans once in my comment, nor did I make any comparisons. I simply described a mathematical fact that a lot of people are having trouble understanding and digesting. I do appreciate your comment as it's not defensive, it's made in earnest, so please don't take my above statement to be a reflection of you in particular.
nyrikki 2y ago
> “let’s construct source that can be used interchangeably with the original”. You lose a lot more than whitespace, you lose semantics, metadata, and lots of other information that makes it so Obviously byte coded languages are a bit easier, and you may get lucky and experts help a lot, and perhaps LLMS can help a little. Both Rice's theorem and the system identification problem from the cybernetics days relate to why it is so hard. > Given a system in the form of a black box (BB) allowing finite input-output interactions, deduce a complete specification of the system’s machine table (i.e., algorithm or internal dynamics). AND > Given a complete specification of a machine table (i.e., algorithm or internal dynamics), recognize any BB having that description. Decompilation doesn't give you the equivalent of the original source code, it gives you new source code that appears to be functionally similar to it, and many people who has been forced to use decompilation to recover from lost source code or consultant time-bombs have run into the problems that admittently seem pretty counter intuitive. Remember that Rice-Shapiro and Kreisel-Lacombe-Shoenfield-Tseitin extend Rice to partial and total functions in finite time. Telling if a program is equivalent to a fixed other program, even for total functions is still undecidable as they are 'non-trivial' properties. Most of the time you can make it work, but it isn't a case of: > “let’s construct source that can be used interchangeably with the original”