4 ms·

Rizin[1] is also able to uplift native code to the new RzIL, which is based on the BAP Core Theory[2] and is essentially an extension of SMT theories of bitvect

by xvilka 3y ago

Rizin[1] is also able to uplift native code to the new RzIL, which is based on the BAP Core Theory[2] and is essentially an extension of SMT theories of bitvectors, bitvector-indexed arrays of bitvectors and effects[3]. [1] https://rizin.re/ https://rizin.re/ [2] https://binaryanalysisplatform.github.io/bap/api/master/bap-core-theory/Bap_core_theory/ https://binaryanalysisplatform.github.io/bap/api/master/bap-... [3] https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md

aleclm 3y ago
Cool projects, but let me make two observations: 1. Writing a binary analysis tool is an enormous task. Before getting to the useful and new part, you need a lot of infrastructure (e.g., a scalable intermediate representation). Rolling your own infrastructure is not a great idea IMO, there are great compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. We can literally run the LLVM -O2 optimization pipeline on our IR. 2. In binary analysis there's a trend to use SMT solvers. We absolutely do not, by design. They simply do not scale. Binary analysis theory is really just compiler theory. Have you ever seen a mainstream compiler use SMT solvers for the standard compilation pipeline? No, it does not scale. Depending on what you need to do, you go the old fashioned way and write a data-flow analysis that does what you need, instead of using an SMT solver that can solve much more complex problems but does not scale. At rev.ng, we're in love with Monotone Frameworks. If your analysis fits in the framework, it will run in linear time. It's more difficult than going "Z3 solve it for me", but it scales great. Suggested reading: https://link.springer.com/book/10.1007/978-3-662-03811-6 https://link.springer.com/book/10.1007/978-3-662-03811-6 Don't get me wrong, there are use cases where using an SMT solver makes sense, for instance looking for bugs, but SMT solvers should be a last resource measure for tasks where you're OK with saying "OK, I didn't find anything in 10 minutes, let's bail out". But that's not the case for simply lifting and even decompiling a binary. This said, I haven't been following closely the developments of rizin, but my comments are more on the general trend in binary analysis that I see.
- xvilka 3y ago
  > compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. While I agree with this, the binary code to LLVM IR uplifting loses a lot of context and semantics information because LLVM IR was designed to do precisely the opposite. Moreover, the increasing popularity of using "middle level" IRs in LLVM-based compilers, like MIR and Swift IR, makes the "distance" between code and the IR representation even bigger. This is why IRs specifically designed to tackle RE tasks would perform better (RzIL and any relatively modern intermediate representation). > They simply do not scale. I agree with this notion, but you don't need to "solve" a whole program; it's impossible for any real-world software; you can selectively pick places to do so. Currently, existing binary analysis software already does that, but usually in a more "manual" way, e.g., by emulating pieces of the code sparsely to figure out indirect jumps and so on.
  - aleclm 3y ago
    > the binary code to LLVM IR uplifting loses a lot of context Losing context is good in order to ensure you properly decoupled the frontend from the rest of the pipeline. We don't even keep track of what a "call" instruction is, we re-detect it on the LLVM IR. One reason you may want to preserve context is to let the user know where a specific piece of lifted code originated from. In order to preserve this information, we exploit LLVM's debugging metadata and it works pretty well. There's some loss there, but LLVM transformations strive to preserve it. After all, imagine you have `add rax, 4; add rax, 4`, you'll want to optimize it to a +8 and you'll either have to decide if you want to associate your +8 operation with the first or the second instruction. > the binary code to LLVM IR uplifting loses a lot of [...] semantics information Not sure what you mean here, we use QEMU as a lifter and that's very accurate in terms of semantics. I'm not sure what MIR and Swift IR have to do with the discussion, those are higher level IRs for specific languages. LLVM is rather low level and it's language agnostic. However, for going beyond lifting, i.e., decompilation, it's true that LLVM shows some significant limitations. That's why we're rolling our own MLIR dialect, but we can still benefit of all the MLIR/LLVM infrastructure, optimizations and analyses. We're not starting from scratch. > emulating pieces of the code sparsely to figure out indirect jumps and so on It's hard to emulate without starting from the beginning. Maybe you're thinking about symbolic execution? In any case, rev.ng does not emulate and does not do any symbolic execution: we have a data-flow analysis that detects destinations of indirect jumps and it's pretty scalable and effective. Example of things we handle: https://github.com/revng/revng-qa/blob/master/share/revng/test/tests/analysis/CollectCFG/arm/switch-disjoint-ranges.S https://github.com/revng/revng-qa/blob/master/share/revng/te...
no_time 3y ago
How is the schism from radare2 going? Is Rizin a superset of r2 features or are there things that r2 has gained since the fork and thus missing?
- xvilka 3y ago
  We don't follow radare2 development closely, to be honest. Instead, we prefer to focus on our own roadmap, which roughly is migrating to a new RzIL instead of ESIL in the context of the discussion, improving UX in both Rizin CLI and Cutter GUI, also providing consistent library API for every Rizin feature (before the code was riddled with command calls inside, for example). And it's not a superset of r2 features since some were actually removed because they are out of the scope as we feel it - for example, "mount" commands to mount the filesystems. We don't have the time and resources to maintain numerous filesystems, and this feature can be better implemented in other tools. We focus on the program analysis and raw data reverse engineering instead. As well as improving support for already existing but undervalued before architectures that are common in specific niches, like Hexagon (QDSP) and Tricore - many months of coding and manual testing were put to make Rizin/Cutter usable for reversing code of these architectures, and more will come.
  - no_time 3y ago
    Thanks for the answer. I wish you the best of luck developing Rizin.

by xvilka 3y ago

aleclm 3y ago

Cool projects, but let me make two observations: 1. Writing a binary analysis tool is an enormous task. Before getting to the useful and new part, you need a lot of infrastructure (e.g., a scalable intermediate representation). Rolling your own infrastructure is not a great idea IMO, there are great compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. We can literally run the LLVM -O2 optimization pipeline on our IR. 2. In binary analysis there's a trend to use SMT solvers. We absolutely do not, by design. They simply do not scale. Binary analysis theory is really just compiler theory. Have you ever seen a mainstream compiler use SMT solvers for the standard compilation pipeline? No, it does not scale. Depending on what you need to do, you go the old fashioned way and write a data-flow analysis that does what you need, instead of using an SMT solver that can solve much more complex problems but does not scale. At rev.ng, we're in love with Monotone Frameworks. If your analysis fits in the framework, it will run in linear time. It's more difficult than going "Z3 solve it for me", but it scales great. Suggested reading: https://link.springer.com/book/10.1007/978-3-662-03811-6 https://link.springer.com/book/10.1007/978-3-662-03811-6 Don't get me wrong, there are use cases where using an SMT solver makes sense, for instance looking for bugs, but SMT solvers should be a last resource measure for tasks where you're OK with saying "OK, I didn't find anything in 10 minutes, let's bail out". But that's not the case for simply lifting and even decompiling a binary. This said, I haven't been following closely the developments of rizin, but my comments are more on the general trend in binary analysis that I see.

xvilka 3y ago
> compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. While I agree with this, the binary code to LLVM IR uplifting loses a lot of context and semantics information because LLVM IR was designed to do precisely the opposite. Moreover, the increasing popularity of using "middle level" IRs in LLVM-based compilers, like MIR and Swift IR, makes the "distance" between code and the IR representation even bigger. This is why IRs specifically designed to tackle RE tasks would perform better (RzIL and any relatively modern intermediate representation). > They simply do not scale. I agree with this notion, but you don't need to "solve" a whole program; it's impossible for any real-world software; you can selectively pick places to do so. Currently, existing binary analysis software already does that, but usually in a more "manual" way, e.g., by emulating pieces of the code sparsely to figure out indirect jumps and so on.
- aleclm 3y ago
  > the binary code to LLVM IR uplifting loses a lot of context Losing context is good in order to ensure you properly decoupled the frontend from the rest of the pipeline. We don't even keep track of what a "call" instruction is, we re-detect it on the LLVM IR. One reason you may want to preserve context is to let the user know where a specific piece of lifted code originated from. In order to preserve this information, we exploit LLVM's debugging metadata and it works pretty well. There's some loss there, but LLVM transformations strive to preserve it. After all, imagine you have `add rax, 4; add rax, 4`, you'll want to optimize it to a +8 and you'll either have to decide if you want to associate your +8 operation with the first or the second instruction. > the binary code to LLVM IR uplifting loses a lot of [...] semantics information Not sure what you mean here, we use QEMU as a lifter and that's very accurate in terms of semantics. I'm not sure what MIR and Swift IR have to do with the discussion, those are higher level IRs for specific languages. LLVM is rather low level and it's language agnostic. However, for going beyond lifting, i.e., decompilation, it's true that LLVM shows some significant limitations. That's why we're rolling our own MLIR dialect, but we can still benefit of all the MLIR/LLVM infrastructure, optimizations and analyses. We're not starting from scratch. > emulating pieces of the code sparsely to figure out indirect jumps and so on It's hard to emulate without starting from the beginning. Maybe you're thinking about symbolic execution? In any case, rev.ng does not emulate and does not do any symbolic execution: we have a data-flow analysis that detects destinations of indirect jumps and it's pretty scalable and effective. Example of things we handle: https://github.com/revng/revng-qa/blob/master/share/revng/test/tests/analysis/CollectCFG/arm/switch-disjoint-ranges.S https://github.com/revng/revng-qa/blob/master/share/revng/te...

no_time 3y ago

How is the schism from radare2 going? Is Rizin a superset of r2 features or are there things that r2 has gained since the fork and thus missing?

xvilka 3y ago
We don't follow radare2 development closely, to be honest. Instead, we prefer to focus on our own roadmap, which roughly is migrating to a new RzIL instead of ESIL in the context of the discussion, improving UX in both Rizin CLI and Cutter GUI, also providing consistent library API for every Rizin feature (before the code was riddled with command calls inside, for example). And it's not a superset of r2 features since some were actually removed because they are out of the scope as we feel it - for example, "mount" commands to mount the filesystems. We don't have the time and resources to maintain numerous filesystems, and this feature can be better implemented in other tools. We focus on the program analysis and raw data reverse engineering instead. As well as improving support for already existing but undervalued before architectures that are common in specific niches, like Hexagon (QDSP) and Tricore - many months of coding and manual testing were put to make Rizin/Cutter usable for reversing code of these architectures, and more will come.
- no_time 3y ago
  Thanks for the answer. I wish you the best of luck developing Rizin.