4 ms·
The real question is how a human should merge these results with their own reversing, honestly. I can't really trust GPT-3 to be accurate like I would actually
by saagarjha 4y ago
The real question is how a human should merge these results with their own reversing, honestly. I can't really trust GPT-3 to be accurate like I would actually trust the decompiler (and, as any reverser knows, you don't trust the decompiler). I think I would treat the output of this as I might a suggestion from a friend who I let glance over the code: "hmm, that might be a SHA-1?" and then I go confirm the results for myself.
- trenchgun 4y agoExactly. GPT-3 shines where we can make it solve hard problems into a format where they are easy to verify
- jcul 4y agoI think this plugin is not to decompile, but rather to explain decompiled code. They mention that the AI's decompilation is about as good as Ghidra's (though of course less trustworthy). The benefit is in explaining the decompiled code, they give an example where the prompt to the AI is something like "here is some code decompiled with ghidra, explain it in detail". From the article: "the paraphrase of disassembled or decompiled code into high-level commentary, can be assisted by automated tooling as well. And this is just what the G-3PO Ghidra script does. The output of such a tool, of course, would have to be carefully checked. Taking its soundness for granted would be a mistake, just as it would be a mistake to put too much faith in the decompiler. We should trust such a tool, backed as it is by an opaque LLM, far less than we trust decompilers, in fact. Fortunately reverse engineering is the sort of domain where we don’t need to trust much at all. It’s an essentially skeptical craft."
- xvilka 4y agoAt least most of the decompiler logic comes from formal methods, thus reducing possible edge cases compared to statistics. There's a room for AI in reversing, but it should be a specifically trained model with a carefully extracted features from binary, not only disassembly output: graphs, debug and demangling info, types, IL analysis results, etc. All these GPT-based plugins are just toys. There's more serious research like this[1][2][3] [1] https://keenlab.tencent.com/zh/2019/12/10/Tencent-Keen-Security-Lab-Order-Matters/#more https://keenlab.tencent.com/zh/2019/12/10/Tencent-Keen-Secur... [2] https://keenlab.tencent.com/zh/2020/11/03/neurips-2020-cameraready/#more https://keenlab.tencent.com/zh/2020/11/03/neurips-2020-camer... [3] https://keenlab.tencent.com/zh/2021/08/11/2021-binaryai-public-release/ https://keenlab.tencent.com/zh/2021/08/11/2021-binaryai-publ...