6 ms·

revng translates (i386, x86-64, MIPS, ARM, AArch64, s390x) binaries to LLVM IR

124 points by davikr 3y ago

Someone 3y ago
License is weird and unclear to me. https://github.com/revng/revng/blob/develop/LICENSE.md https://github.com/revng/revng/blob/develop/LICENSE.md: “Licensing information Copyright (c) 2015-2017 Alessandro Di Federico <ale@clearmind.me> Copyright (c) 2017-2020 rev.ng Labs Srl <info@rev.ng> The project as a whole is released under the terms of the GNU General Public License, version 2. A copy of this license can be found in LICENSE.gpl. Each individual file, unless otherwise noted, is released under the terms of the MIT License. A copy of this license can be found in LICENSE.mit.” Does that paragraph about GPLv2 constitute a version of “unless otherwise noted”? If so, why mention the MIT license at all? If not, doesn’t that mean it effectively is dual licensed? If everything is GPLv2, and I take a single file, that’s still GPLv2, and if if have set of MIT licensed files and group them, the result still is MIT licensed. (Browsing the code, all source files seem to have a MIT header, so, presumably, they mean this to beMIT licensed) Also, it’s not “GPLv2 or later”, but I think you can take the MIT licensed files and rerelease them as GPLv3, if you’d want to (or is the attribution requirement a showstopper there?)
- mgaunard 3y ago
  It is dual licensed. Only the copyright holder can change the license.
- themerone 3y ago
  IANAL, but if you fork a project and change the licence, the new license only applies to your modifications.
- Brian_K_White 3y ago
  If you have a set of MIT files, and group them, the group is whatever you want it to be. It just means someone else can take all the files and make something else with them, but they just can't redistribute the new project under the name revng, unless it conforms to gpl2.
- agf 3y ago
  > Each individual file, unless otherwise noted, is released under the terms of the MIT License. A copy of this license can be found in LICENSE.mit. Worth . . . noting . . . that every file with license info says MIT; none say anything else. So the "unless otherwise noted" is purely future-proofing.
- edef 3y ago
  The qemu TCG parts that it presumably links to are GPLv2, so the combined project is GPLv2 by necessity[1]. I think that's all they're trying to get across. [1]: apart from viral licenses being pretty much meaningless under EU copyright law
  - aleclm 3y ago
    Exactly. We wanted MIT, but we were infected by the GPLv2 due to the QEMU dependency. So we licensed each file under MIT, but GPLv2 as a whole. QEMU does something similar: https://github.com/qemu/qemu/blob/master/LICENSE
    - Someone 3y ago
      Aha! I think you could phrase that better. Maybe something like “This projects builds on GPLv2-licensed code, so the project is GPLv2-licensed. If you want to use our additions in isolation, you can choose to use them as either GPLv2 or MIT licensed”?
CharlesW 3y ago
Could you, in turn, convert this to Wasm? (I'm wondering if it's possible to convert a macOS command-line tool to run on x86-64.)
- themerone 3y ago
  Unlikely, you would have to convert every linked library, and every system call would have to be supported by the wasm runtime.
  - flamedoge 3y ago
    but it is possible
  - aleclm 3y ago
    We were aiming at a static binary, but dynamic would be feasible too, with some love. For the wasm runtime, we planned to use this: https://browsix.org/ https://browsix.org/
- bloopernova 3y ago
  Now I'm curious what the macOS CLI binary is!
  - saagarjha 3y ago
    Placing my bets on codesign
- aleclm 3y ago
  We wanted to have a demo on our website where you could upload your binary and run it in the browser recompiling to WASM. Feasible, but not a negligible amount of work. About macOS: we can lift any binary for any platform, but we can recompile to an executable only programs for Linux. The reason for this is that the QEMU user mode only supports running Linux on Linux. It would be, in principle, possible to add support for macOS user mode (there's a half dead FreeBSD user mode), but it's quite some work. Windows would be almost impossible: in Windows, the compatibility layer for user applications (i.e., the boundary where Microsoft tells you that things will stay the same in the future) is not the boundary between the user space and the kernel (as it is in Linux) but the system DLLs. So QEMU should become KERNEL32.dll-aware, which would be weird and very different from the current design of QEMU user mode. This said, if you have a dynamic binary that does not perform any syscall (if not through other dynamic libraries), something can be worked out.
api 3y ago
Wonder if you could use this with binfmt and a cache to create a kind of super-Rosetta for Linux. Any Linux binary not for the host architecture will be translated, and translated binaries are cached somewhere by a hash of the original binary so the translation only has to happen once. All transparent to the user.
- dmitrygr 3y ago
  Just like Rosetta, that will be an incomplete solution, without also including a JIT. Nothing stops a binary from generating native code at runtime, loading it from a random file, and many such things. A static recompiler is not possible to write such that it covers ALL the possibilities. You need a runtime component
  - Brian_K_White 3y ago
    True but does not seem to make it not worth doing. Incomplete? OK but how incomplete really? I have never written anything myself that relied on generating and then executing machine code at run time. Sure it exists, but so what if one app out of everything I ever use turns out to do it? Or even say ten? Complete but only by dint of including a jit? OK but does that imply there might as well just be the jit alone? I don't think so. The general concept of caching has been well proven to be overwhelmingly worth it in pretty much any context it's possible.
    - api 3y ago
      If you ran the transpiled binaries with a security flag prohibiting execution of generated code, that would cause anything with a jit to crash neatly and possibly with an informative error message. Docs for this system would have to be clear that jits will not work and must be native. I think Linux has some kind of no-exec flag like that. I know macOS does.
      - dmitrygr 3y ago
        that would not save you from someone doing things like: ((void (*)(void))(((uintptr_t)&myfunc) + 1))(); and there you go - on x86 this could plausibly be a way to obfuscate code since it has variable-length instrs. And your static analysis may not catch this. This does not require writeable executable pages or JIT. Hell, i can receive a uint32 from the network, cast to a func pointer, and jump to it. And for all you know, my protocol relies on a lack of ASLR and sends me a valid entry point addr in my executable, which will have changed in the translation you do. (Fun story: long ago, AIM's anti-protocol-cloning code did that - server would send a VA and ask for a few bytes at that VA. Unless you had a current copy of the aim executable in memory running you had no way to correctly reply) The long tail of valid but freaky shit is ....very long, and disallowing W&X pages will not protect you from it all
vient 3y ago
Usually such things are called lifters. Wonder how this tool compares to other existing LLVM IR lifters, such as remill[0] and rellume[1]. 0: https://github.com/lifting-bits/remill https://github.com/lifting-bits/remill 1: https://github.com/aengelke/rellume https://github.com/aengelke/rellume
- aleclm 3y ago
  Hey, rev.ng core dev here. I guess the core difference is that we do not implement the instructions semantics ourselves, but we reuse QEMU. QEMU, when in emulation mode (i.e., not KVM), goes from executable code to an intermediate representation (tiny code instructions) and then compile it back to the host architecture. We use the first part of QEMU but then translate tiny code instructions into LLVM IR. This means that: 1. We don't have to implement instruction semantics on our own; 2. We can easily support the architectures supported by QEMU. Currently we support x86-64, i386, ARM, AArch64, MIPS and S390. https://wiki.qemu.org/Documentation/Platforms https://wiki.qemu.org/Documentation/Platforms 3. We get out of the box the accuracy in instructions semantics of a mature emulator such as QEMU. We used to be able to lift `gcc` to LLVM IR, recompile it and have it working. However, the real difference is that, while it still works, we're now no longer focused on binary-to-binary translation, but we're now 100% focused on writing a full blown decompiler, i.e., something that goes from binary code to (valid) C and has a nice interactive UI. As you can imagine, the lifter is really just the first half of a decompiler. For instance, one core feature of rev.ng is automatic detection of `struct`s, just by looking at how they're used: https://twitter.com/_revng/status/1575553313827069952 https://twitter.com/_revng/status/1575553313827069952 If you have been doing some reverse engineering, I'm sure you'd appreciate this feature. Bonus, take a look at our UI: https://twitter.com/_revng/status/1720440515265474631 https://twitter.com/_revng/status/1720440515265474631 As a side note, towards the end of the month we're releasing the new website and starting to invite people to the closed beta. If you wanna join: https://rev.ng/register-for-nightly.html https://rev.ng/register-for-nightly.html
  - saagarjha 3y ago
    Do you have examples of your decompilation, and how it compares to state of the art that uses other approaches? Do you run optimizations on your generated LLVM IR to clean it up and canonicalize it?
quic_bcain 3y ago
Alessandro presented revng at the 2016 llvm Bay area dev conf. video https://youtu.be/5CbuU4KwBCE?si=eUN_UGr7cjat3P8U https://youtu.be/5CbuU4KwBCE?si=eUN_UGr7cjat3P8U slides https://www.llvm.org/devmtg/2016-11/Slides/DiFederico-rev.ng.pdf https://www.llvm.org/devmtg/2016-11/Slides/DiFederico-rev.ng...
xvilka 3y ago
Rizin[1] is also able to uplift native code to the new RzIL, which is based on the BAP Core Theory[2] and is essentially an extension of SMT theories of bitvectors, bitvector-indexed arrays of bitvectors and effects[3]. [1] https://rizin.re/ https://rizin.re/ [2] https://binaryanalysisplatform.github.io/bap/api/master/bap-core-theory/Bap_core_theory/ https://binaryanalysisplatform.github.io/bap/api/master/bap-... [3] https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md
- aleclm 3y ago
  Cool projects, but let me make two observations: 1. Writing a binary analysis tool is an enormous task. Before getting to the useful and new part, you need a lot of infrastructure (e.g., a scalable intermediate representation). Rolling your own infrastructure is not a great idea IMO, there are great compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. We can literally run the LLVM -O2 optimization pipeline on our IR. 2. In binary analysis there's a trend to use SMT solvers. We absolutely do not, by design. They simply do not scale. Binary analysis theory is really just compiler theory. Have you ever seen a mainstream compiler use SMT solvers for the standard compilation pipeline? No, it does not scale. Depending on what you need to do, you go the old fashioned way and write a data-flow analysis that does what you need, instead of using an SMT solver that can solve much more complex problems but does not scale. At rev.ng, we're in love with Monotone Frameworks. If your analysis fits in the framework, it will run in linear time. It's more difficult than going "Z3 solve it for me", but it scales great. Suggested reading: https://link.springer.com/book/10.1007/978-3-662-03811-6 https://link.springer.com/book/10.1007/978-3-662-03811-6 Don't get me wrong, there are use cases where using an SMT solver makes sense, for instance looking for bugs, but SMT solvers should be a last resource measure for tasks where you're OK with saying "OK, I didn't find anything in 10 minutes, let's bail out". But that's not the case for simply lifting and even decompiling a binary. This said, I haven't been following closely the developments of rizin, but my comments are more on the general trend in binary analysis that I see.
  - xvilka 3y ago
    > compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. While I agree with this, the binary code to LLVM IR uplifting loses a lot of context and semantics information because LLVM IR was designed to do precisely the opposite. Moreover, the increasing popularity of using "middle level" IRs in LLVM-based compilers, like MIR and Swift IR, makes the "distance" between code and the IR representation even bigger. This is why IRs specifically designed to tackle RE tasks would perform better (RzIL and any relatively modern intermediate representation). > They simply do not scale. I agree with this notion, but you don't need to "solve" a whole program; it's impossible for any real-world software; you can selectively pick places to do so. Currently, existing binary analysis software already does that, but usually in a more "manual" way, e.g., by emulating pieces of the code sparsely to figure out indirect jumps and so on.
saagarjha 3y ago
How’s performance of the translated version?
- aleclm 3y ago
  Here you can find some data: https://rev.ng/downloads/bar-2019-slides.pdf https://rev.ng/downloads/bar-2019-slides.pdf

124 points by davikr 3y ago

License is weird and unclear to me. https://github.com/revng/revng/blob/develop/LICENSE.md https://github.com/revng/revng/blob/develop/LICENSE.md: “Licensing information Copyright (c) 2015-2017 Alessandro Di Federico <ale@clearmind.me> Copyright (c) 2017-2020 rev.ng Labs Srl <info@rev.ng> The project as a whole is released under the terms of the GNU General Public License, version 2. A copy of this license can be found in LICENSE.gpl. Each individual file, unless otherwise noted, is released under the terms of the MIT License. A copy of this license can be found in LICENSE.mit.” Does that paragraph about GPLv2 constitute a version of “unless otherwise noted”? If so, why mention the MIT license at all? If not, doesn’t that mean it effectively is dual licensed? If everything is GPLv2, and I take a single file, that’s still GPLv2, and if if have set of MIT licensed files and group them, the result still is MIT licensed. (Browsing the code, all source files seem to have a MIT header, so, presumably, they mean this to beMIT licensed) Also, it’s not “GPLv2 or later”, but I think you can take the MIT licensed files and rerelease them as GPLv3, if you’d want to (or is the attribution requirement a showstopper there?)

mgaunard 3y ago
It is dual licensed. Only the copyright holder can change the license.
themerone 3y ago
IANAL, but if you fork a project and change the licence, the new license only applies to your modifications.
Brian_K_White 3y ago
If you have a set of MIT files, and group them, the group is whatever you want it to be. It just means someone else can take all the files and make something else with them, but they just can't redistribute the new project under the name revng, unless it conforms to gpl2.
agf 3y ago
> Each individual file, unless otherwise noted, is released under the terms of the MIT License. A copy of this license can be found in LICENSE.mit. Worth . . . noting . . . that every file with license info says MIT; none say anything else. So the "unless otherwise noted" is purely future-proofing.
edef 3y ago
The qemu TCG parts that it presumably links to are GPLv2, so the combined project is GPLv2 by necessity[1]. I think that's all they're trying to get across. [1]: apart from viral licenses being pretty much meaningless under EU copyright law
- aleclm 3y ago
  Exactly. We wanted MIT, but we were infected by the GPLv2 due to the QEMU dependency. So we licensed each file under MIT, but GPLv2 as a whole. QEMU does something similar: https://github.com/qemu/qemu/blob/master/LICENSE
  - Someone 3y ago
    Aha! I think you could phrase that better. Maybe something like “This projects builds on GPLv2-licensed code, so the project is GPLv2-licensed. If you want to use our additions in isolation, you can choose to use them as either GPLv2 or MIT licensed”?

CharlesW 3y ago

Could you, in turn, convert this to Wasm? (I'm wondering if it's possible to convert a macOS command-line tool to run on x86-64.)

themerone 3y ago
Unlikely, you would have to convert every linked library, and every system call would have to be supported by the wasm runtime.
- flamedoge 3y ago
  but it is possible
- aleclm 3y ago
  We were aiming at a static binary, but dynamic would be feasible too, with some love. For the wasm runtime, we planned to use this: https://browsix.org/ https://browsix.org/
bloopernova 3y ago
Now I'm curious what the macOS CLI binary is!
- saagarjha 3y ago
  Placing my bets on codesign
aleclm 3y ago
We wanted to have a demo on our website where you could upload your binary and run it in the browser recompiling to WASM. Feasible, but not a negligible amount of work. About macOS: we can lift any binary for any platform, but we can recompile to an executable only programs for Linux. The reason for this is that the QEMU user mode only supports running Linux on Linux. It would be, in principle, possible to add support for macOS user mode (there's a half dead FreeBSD user mode), but it's quite some work. Windows would be almost impossible: in Windows, the compatibility layer for user applications (i.e., the boundary where Microsoft tells you that things will stay the same in the future) is not the boundary between the user space and the kernel (as it is in Linux) but the system DLLs. So QEMU should become KERNEL32.dll-aware, which would be weird and very different from the current design of QEMU user mode. This said, if you have a dynamic binary that does not perform any syscall (if not through other dynamic libraries), something can be worked out.

api 3y ago

Wonder if you could use this with binfmt and a cache to create a kind of super-Rosetta for Linux. Any Linux binary not for the host architecture will be translated, and translated binaries are cached somewhere by a hash of the original binary so the translation only has to happen once. All transparent to the user.

dmitrygr 3y ago
Just like Rosetta, that will be an incomplete solution, without also including a JIT. Nothing stops a binary from generating native code at runtime, loading it from a random file, and many such things. A static recompiler is not possible to write such that it covers ALL the possibilities. You need a runtime component
- Brian_K_White 3y ago
  True but does not seem to make it not worth doing. Incomplete? OK but how incomplete really? I have never written anything myself that relied on generating and then executing machine code at run time. Sure it exists, but so what if one app out of everything I ever use turns out to do it? Or even say ten? Complete but only by dint of including a jit? OK but does that imply there might as well just be the jit alone? I don't think so. The general concept of caching has been well proven to be overwhelmingly worth it in pretty much any context it's possible.
  - api 3y ago
    If you ran the transpiled binaries with a security flag prohibiting execution of generated code, that would cause anything with a jit to crash neatly and possibly with an informative error message. Docs for this system would have to be clear that jits will not work and must be native. I think Linux has some kind of no-exec flag like that. I know macOS does.
    - dmitrygr 3y ago
      that would not save you from someone doing things like: ((void (*)(void))(((uintptr_t)&myfunc) + 1))(); and there you go - on x86 this could plausibly be a way to obfuscate code since it has variable-length instrs. And your static analysis may not catch this. This does not require writeable executable pages or JIT. Hell, i can receive a uint32 from the network, cast to a func pointer, and jump to it. And for all you know, my protocol relies on a lack of ASLR and sends me a valid entry point addr in my executable, which will have changed in the translation you do. (Fun story: long ago, AIM's anti-protocol-cloning code did that - server would send a VA and ask for a few bytes at that VA. Unless you had a current copy of the aim executable in memory running you had no way to correctly reply) The long tail of valid but freaky shit is ....very long, and disallowing W&X pages will not protect you from it all

vient 3y ago

Usually such things are called lifters. Wonder how this tool compares to other existing LLVM IR lifters, such as remill[0] and rellume[1]. 0: https://github.com/lifting-bits/remill https://github.com/lifting-bits/remill 1: https://github.com/aengelke/rellume https://github.com/aengelke/rellume

aleclm 3y ago
Hey, rev.ng core dev here. I guess the core difference is that we do not implement the instructions semantics ourselves, but we reuse QEMU. QEMU, when in emulation mode (i.e., not KVM), goes from executable code to an intermediate representation (tiny code instructions) and then compile it back to the host architecture. We use the first part of QEMU but then translate tiny code instructions into LLVM IR. This means that: 1. We don't have to implement instruction semantics on our own; 2. We can easily support the architectures supported by QEMU. Currently we support x86-64, i386, ARM, AArch64, MIPS and S390. https://wiki.qemu.org/Documentation/Platforms https://wiki.qemu.org/Documentation/Platforms 3. We get out of the box the accuracy in instructions semantics of a mature emulator such as QEMU. We used to be able to lift `gcc` to LLVM IR, recompile it and have it working. However, the real difference is that, while it still works, we're now no longer focused on binary-to-binary translation, but we're now 100% focused on writing a full blown decompiler, i.e., something that goes from binary code to (valid) C and has a nice interactive UI. As you can imagine, the lifter is really just the first half of a decompiler. For instance, one core feature of rev.ng is automatic detection of `struct`s, just by looking at how they're used: https://twitter.com/_revng/status/1575553313827069952 https://twitter.com/_revng/status/1575553313827069952 If you have been doing some reverse engineering, I'm sure you'd appreciate this feature. Bonus, take a look at our UI: https://twitter.com/_revng/status/1720440515265474631 https://twitter.com/_revng/status/1720440515265474631 As a side note, towards the end of the month we're releasing the new website and starting to invite people to the closed beta. If you wanna join: https://rev.ng/register-for-nightly.html https://rev.ng/register-for-nightly.html
- saagarjha 3y ago
  Do you have examples of your decompilation, and how it compares to state of the art that uses other approaches? Do you run optimizations on your generated LLVM IR to clean it up and canonicalize it?

quic_bcain 3y ago

Alessandro presented revng at the 2016 llvm Bay area dev conf. video https://youtu.be/5CbuU4KwBCE?si=eUN_UGr7cjat3P8U https://youtu.be/5CbuU4KwBCE?si=eUN_UGr7cjat3P8U slides https://www.llvm.org/devmtg/2016-11/Slides/DiFederico-rev.ng.pdf https://www.llvm.org/devmtg/2016-11/Slides/DiFederico-rev.ng...

xvilka 3y ago

Rizin[1] is also able to uplift native code to the new RzIL, which is based on the BAP Core Theory[2] and is essentially an extension of SMT theories of bitvectors, bitvector-indexed arrays of bitvectors and effects[3]. [1] https://rizin.re/ https://rizin.re/ [2] https://binaryanalysisplatform.github.io/bap/api/master/bap-core-theory/Bap_core_theory/ https://binaryanalysisplatform.github.io/bap/api/master/bap-... [3] https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md https://github.com/rizinorg/rizin/blob/dev/doc/rzil.md

aleclm 3y ago
Cool projects, but let me make two observations: 1. Writing a binary analysis tool is an enormous task. Before getting to the useful and new part, you need a lot of infrastructure (e.g., a scalable intermediate representation). Rolling your own infrastructure is not a great idea IMO, there are great compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. We can literally run the LLVM -O2 optimization pipeline on our IR. 2. In binary analysis there's a trend to use SMT solvers. We absolutely do not, by design. They simply do not scale. Binary analysis theory is really just compiler theory. Have you ever seen a mainstream compiler use SMT solvers for the standard compilation pipeline? No, it does not scale. Depending on what you need to do, you go the old fashioned way and write a data-flow analysis that does what you need, instead of using an SMT solver that can solve much more complex problems but does not scale. At rev.ng, we're in love with Monotone Frameworks. If your analysis fits in the framework, it will run in linear time. It's more difficult than going "Z3 solve it for me", but it scales great. Suggested reading: https://link.springer.com/book/10.1007/978-3-662-03811-6 https://link.springer.com/book/10.1007/978-3-662-03811-6 Don't get me wrong, there are use cases where using an SMT solver makes sense, for instance looking for bugs, but SMT solvers should be a last resource measure for tasks where you're OK with saying "OK, I didn't find anything in 10 minutes, let's bail out". But that's not the case for simply lifting and even decompiling a binary. This said, I haven't been following closely the developments of rizin, but my comments are more on the general trend in binary analysis that I see.
- xvilka 3y ago
  > compiler frameworks (LLVM, but GCC would work too) that provide you with a lot of tools to start with. While I agree with this, the binary code to LLVM IR uplifting loses a lot of context and semantics information because LLVM IR was designed to do precisely the opposite. Moreover, the increasing popularity of using "middle level" IRs in LLVM-based compilers, like MIR and Swift IR, makes the "distance" between code and the IR representation even bigger. This is why IRs specifically designed to tackle RE tasks would perform better (RzIL and any relatively modern intermediate representation). > They simply do not scale. I agree with this notion, but you don't need to "solve" a whole program; it's impossible for any real-world software; you can selectively pick places to do so. Currently, existing binary analysis software already does that, but usually in a more "manual" way, e.g., by emulating pieces of the code sparsely to figure out indirect jumps and so on.

saagarjha 3y ago

How’s performance of the translated version?

aleclm 3y ago
Here you can find some data: https://rev.ng/downloads/bar-2019-slides.pdf https://rev.ng/downloads/bar-2019-slides.pdf