4 ms·

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra This is something all people using decompilers say and sort of shows how l

by aleclm 3y ago

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low. I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised. Apart from accuracy, which is difficult but engineering work, why don't decompilers emit syntactically valid C? Have you ever tried to re-compile code from any decompiler? It's a terrible experience. rev.ng only emits valid C code, and we test it with a bunch of -Wall -Wextra: https://github.com/revng/revng-c/blob/develop/share/revng-c/compile-flags.cfg#L14 https://github.com/revng/revng-c/blob/develop/share/revng-c/... Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none. rev.ng, by default, detects data structures on the whole binary, interprocedurally, including arrays. See the linked list example in the blog post. We also have plans to detect enums and other stuff. Clearly we're not there yet, we still need to work on robustness, but our goal is to increase the confidence in decompilers and actually offer features that save time. Certain tools have made progress in improving the UI and the scripting experience, but there's other things to do beyond that. I see this a bit like the transition from the phase in which C developers where using macros to ensure things were being inlined/unrolled to the phase where they stopped doing that because compilers got smart enough to the right thing and to do it much more effectively.

saagarjha 3y ago
Curious what you do when you encounter an instruction you don't model
- aleclm 3y ago
  That's unlikely, since we use QEMU as a lifter, which sometimes supports new instructions before they hit silicon. However, I think we'll emit a call to some `noreturn` function. Basically we emit a call to `abort`.
  - saagarjha 3y ago
    Right but you do see how this means that you need to lift code that has semantics that cannot be modeled in C?
    - aleclm 3y ago
      Sure, in those cases we emit calls to C functions. The only thing we need to know is what registers are taken as input, what registers are output and what registers are preserved. In QEMU parlance, these are helper functions, and they have actual implementations. But for decompilation purposes, you don't need to implement them. You just need to know how they interact with the registers.
selfmodruntime 3y ago
What happens if you put in a binary which outputs C-like machine code, like Rust (llvm) or zig?
- aleclm 3y ago
  Languages with a rich standard library and generating a lot of code for you usually need some love to get rid/represent idiomatically common patterns and to detect common data structures. We haven't looked into it yet, but the automatic data structure recognition might help. Frankly, Rust looks particularly scary: https://media.ccc.de/v/37c3-11684-rust_binary_analysis_feature_by_feature https://media.ccc.de/v/37c3-11684-rust_binary_analysis_featu...
  - tux3 3y ago
    Oh, very nice! I've dealt with forsaken deeply abstract vtable mazes of hell, but the idea of using a ton of sum types, dynamic dispatch, async everywhere, and long iterator chains would make for some deliciously unreadable binaries!
Sesse__ 3y ago
> Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none. That's funny, because I've used both Hex-Rays and Ghidra, and gotten lots of help with data structures. The interactivity really helps a bunch with filling in the blanks.
- aleclm 3y ago
  In IDA you basically have only detection of stack frame layout (in a quite confusing fashion) and "create struct out of this pointer", which is something you have to do manually and its intraprocedural. Imagine this being done automatically, across all of the binary. If you pass a pointer to another function the type is correct and you build the type from all the functions using it. Then obviously the user needs to fix things, but boostrapping can definitely be hugely improved.
  - Sesse__ 2y ago
    I'm sure user-defined structs can benefit from combining information from multiple functions, but saying that what you get today is “basically none” is a bit of an overstatement. Also, the special (and important!) case of operating system ABI structs is great, and that information propagates throughout function calls.
jcranmer 3y ago
Here's my issue with decompilers: I don't want to look at assembly code. I'd rather see expression trees, expressed in C-like syntax, than trying to piece together variables from two-address or three-address instructions. Looking at assembly tends to lead to brain farts like "wait, was the first or second operand the output operand?" (really, fuck AT&T syntax) or "wait, does ja implement ugt or sgt?" So that means I want to look at something vaguely C-like. But the problem is that the C type system is too powerful for decompilers to robustly lift to, and the resulting code is generally at best filled with distractions of wait-I-can-fix-this excessive casting and at worst just wrong. And when it's wrong, I have to resort to staring at the assembly, which (for Ghidra at least) means throwing away a lot of the notes I've accumulated because they don't correlate back to underlying assembly. So what I really want isn't something that can emit recompilable C code, that's optimizing for something that doesn't help me in the end. What I want is robust decompilation to something that lets me ignore the assembly entirely. I'm a compiler writer, I can handle a language where integers aren't signed but the operands are.
- aleclm 3y ago
  I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms. Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon). But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emitting valid C? For instance, thanks to the fact that we emit valid C, we could test if the assembly emitted by a compiler is the same before and after removing redundant casts. My point is that emitting valid C is kind of a prerequisite for what you're asking, a rather low bar to pass, but that, in practice, no mainstream decompiler passes. It's pretty obvious the decompiled code will often be redundant and outright wrong if you don't even guarantee it's syntactically valid. Then clearly it's not a panacea, but it's an important design criterion and shows the direction we want to go. As for comments: we still haven't implemented inline comments, but they will be attached to program addresses, so they will be available both in disassembly and decompiled C. It's not very hard to do, but that needs some love.
  - 3y ago

by aleclm 3y ago

saagarjha 3y ago

Curious what you do when you encounter an instruction you don't model

aleclm 3y ago
That's unlikely, since we use QEMU as a lifter, which sometimes supports new instructions before they hit silicon. However, I think we'll emit a call to some `noreturn` function. Basically we emit a call to `abort`.
- saagarjha 3y ago
  Right but you do see how this means that you need to lift code that has semantics that cannot be modeled in C?
  - aleclm 3y ago
    Sure, in those cases we emit calls to C functions. The only thing we need to know is what registers are taken as input, what registers are output and what registers are preserved. In QEMU parlance, these are helper functions, and they have actual implementations. But for decompilation purposes, you don't need to implement them. You just need to know how they interact with the registers.

selfmodruntime 3y ago

What happens if you put in a binary which outputs C-like machine code, like Rust (llvm) or zig?

aleclm 3y ago
Languages with a rich standard library and generating a lot of code for you usually need some love to get rid/represent idiomatically common patterns and to detect common data structures. We haven't looked into it yet, but the automatic data structure recognition might help. Frankly, Rust looks particularly scary: https://media.ccc.de/v/37c3-11684-rust_binary_analysis_feature_by_feature https://media.ccc.de/v/37c3-11684-rust_binary_analysis_featu...
- tux3 3y ago
  Oh, very nice! I've dealt with forsaken deeply abstract vtable mazes of hell, but the idea of using a ton of sum types, dynamic dispatch, async everywhere, and long iterator chains would make for some deliciously unreadable binaries!

Sesse__ 3y ago

> Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none. That's funny, because I've used both Hex-Rays and Ghidra, and gotten lots of help with data structures. The interactivity really helps a bunch with filling in the blanks.

aleclm 3y ago
In IDA you basically have only detection of stack frame layout (in a quite confusing fashion) and "create struct out of this pointer", which is something you have to do manually and its intraprocedural. Imagine this being done automatically, across all of the binary. If you pass a pointer to another function the type is correct and you build the type from all the functions using it. Then obviously the user needs to fix things, but boostrapping can definitely be hugely improved.
- Sesse__ 2y ago
  I'm sure user-defined structs can benefit from combining information from multiple functions, but saying that what you get today is “basically none” is a bit of an overstatement. Also, the special (and important!) case of operating system ABI structs is great, and that information propagates throughout function calls.

jcranmer 3y ago

Here's my issue with decompilers: I don't want to look at assembly code. I'd rather see expression trees, expressed in C-like syntax, than trying to piece together variables from two-address or three-address instructions. Looking at assembly tends to lead to brain farts like "wait, was the first or second operand the output operand?" (really, fuck AT&T syntax) or "wait, does ja implement ugt or sgt?" So that means I want to look at something vaguely C-like. But the problem is that the C type system is too powerful for decompilers to robustly lift to, and the resulting code is generally at best filled with distractions of wait-I-can-fix-this excessive casting and at worst just wrong. And when it's wrong, I have to resort to staring at the assembly, which (for Ghidra at least) means throwing away a lot of the notes I've accumulated because they don't correlate back to underlying assembly. So what I really want isn't something that can emit recompilable C code, that's optimizing for something that doesn't help me in the end. What I want is robust decompilation to something that lets me ignore the assembly entirely. I'm a compiler writer, I can handle a language where integers aren't signed but the operands are.

aleclm 3y ago
I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms. Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon). But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emitting valid C? For instance, thanks to the fact that we emit valid C, we could test if the assembly emitted by a compiler is the same before and after removing redundant casts. My point is that emitting valid C is kind of a prerequisite for what you're asking, a rather low bar to pass, but that, in practice, no mainstream decompiler passes. It's pretty obvious the decompiled code will often be redundant and outright wrong if you don't even guarantee it's syntactically valid. Then clearly it's not a panacea, but it's an important design criterion and shows the direction we want to go. As for comments: we still haven't implemented inline comments, but they will be attached to program addresses, so they will be available both in disassembly and decompiled C. It's not very hard to do, but that needs some love.
- 3y ago