4 ms·

Wonder if you could use this with binfmt and a cache to create a kind of super-Rosetta for Linux. Any Linux binary not for the host architecture will be transla

by api 3y ago

Wonder if you could use this with binfmt and a cache to create a kind of super-Rosetta for Linux. Any Linux binary not for the host architecture will be translated, and translated binaries are cached somewhere by a hash of the original binary so the translation only has to happen once. All transparent to the user.

dmitrygr 3y ago
Just like Rosetta, that will be an incomplete solution, without also including a JIT. Nothing stops a binary from generating native code at runtime, loading it from a random file, and many such things. A static recompiler is not possible to write such that it covers ALL the possibilities. You need a runtime component
- Brian_K_White 3y ago
  True but does not seem to make it not worth doing. Incomplete? OK but how incomplete really? I have never written anything myself that relied on generating and then executing machine code at run time. Sure it exists, but so what if one app out of everything I ever use turns out to do it? Or even say ten? Complete but only by dint of including a jit? OK but does that imply there might as well just be the jit alone? I don't think so. The general concept of caching has been well proven to be overwhelmingly worth it in pretty much any context it's possible.
  - api 3y ago
    If you ran the transpiled binaries with a security flag prohibiting execution of generated code, that would cause anything with a jit to crash neatly and possibly with an informative error message. Docs for this system would have to be clear that jits will not work and must be native. I think Linux has some kind of no-exec flag like that. I know macOS does.
    - dmitrygr 3y ago
      that would not save you from someone doing things like: ((void (*)(void))(((uintptr_t)&myfunc) + 1))(); and there you go - on x86 this could plausibly be a way to obfuscate code since it has variable-length instrs. And your static analysis may not catch this. This does not require writeable executable pages or JIT. Hell, i can receive a uint32 from the network, cast to a func pointer, and jump to it. And for all you know, my protocol relies on a lack of ASLR and sends me a valid entry point addr in my executable, which will have changed in the translation you do. (Fun story: long ago, AIM's anti-protocol-cloning code did that - server would send a VA and ask for a few bytes at that VA. Unless you had a current copy of the aim executable in memory running you had no way to correctly reply) The long tail of valid but freaky shit is ....very long, and disallowing W&X pages will not protect you from it all
      - akoboldfrying 3y ago
        Correct, your incremented function pointer won't be caught -- but now we're really getting down to the parts-per-million of programs that use this. Demoscene code, sometimes, maybe? I would say W^X protection will protect you from all non-adversarial programs.
        dmitrygr 3y ago
        Due to anti piracy and anti cheat mechanisms, games are also likely to pull shit like that
  - saagarjha 3y ago
    Have you ever written Java code? Or Lua? Or (as of recently) Python, Ruby, …?
aleclm 3y ago
You can already do that with QEMU: https://wiki.debian.org/QemuUserEmulation https://wiki.debian.org/QemuUserEmulation However, the code we emit is faster than the one emitted by QEMU. We have a secret plan to use LLVM in QEMU as a tier 2 JIT engine to optimize hot code paths more aggressively. This has already been done, but never upstreamed. Adding an on-disk cache would also be quite useful.
- ris 3y ago
  > This has already been done, but never upstreamed Assume you're referring to hqemu http://csl.iis.sinica.edu.tw/hqemu/ http://csl.iis.sinica.edu.tw/hqemu/ > Adding an on-disk cache would also be quite useful. +100
- aengelke 3y ago
  > [LLVM in QEMU] has already been done, but never upstreamed. Yeah, that's a long history, and it never worked great, the architectures are too different. LLVM is great for compiling functions while QEMU compiles basic blocks, which is somewhat required for correctness. LLVM for binary translation requires longer superblocks (HQEMU) or functions (Instrew). I don't see _correct_ binary translation with LLVM as viable anytime soon, though: every memory access can cause a synchronous signal (SIGSEGV/SIGBUS), and I'd expect recovering the full register state at the prior instruction boundary from the LLVM-compiled code to come at severe performance penalties. > Adding an on-disk cache would also be quite useful. I implemented a persistent cache in Instrew to work around LLVM's high compile times. Cache access is done by decoding the code-to-be-translated (very fast) and using a hash of that code for lookup. So shared libraries and even the same code in different binaries will hit the cache, regardless of their address or path name. If translation is fast (QEMU/TCG), caching is probably not very useful, though.
  - aleclm 3y ago
    > every memory access can cause a synchronous signal (SIGSEGV/SIGBUS) AFAIU you have all the same problems as soon as you translate more than one instruction at a time and allow merging them, which QEMU does, even if just at basic block level. IIRC post-SIGSEGV state is not 100% correct in QEMU, but I'd need to investigate some more. We built a dynamic binary translator for a customer dealing with mainframes using ORC JIT where we translate large amounts of code, handle self-modifying code and caching things on disk (similar approach to the one you suggest). Compile time is a problem, but then we the performance results are impressive. Giving visibility over loops to LLVM helps a lot. But the real reason why nothing like this is upstream is that it's difficult to get things upstream, it requires a lot of effort compared to putting together a PoC. :)

by api 3y ago

dmitrygr 3y ago

Just like Rosetta, that will be an incomplete solution, without also including a JIT. Nothing stops a binary from generating native code at runtime, loading it from a random file, and many such things. A static recompiler is not possible to write such that it covers ALL the possibilities. You need a runtime component

Brian_K_White 3y ago
True but does not seem to make it not worth doing. Incomplete? OK but how incomplete really? I have never written anything myself that relied on generating and then executing machine code at run time. Sure it exists, but so what if one app out of everything I ever use turns out to do it? Or even say ten? Complete but only by dint of including a jit? OK but does that imply there might as well just be the jit alone? I don't think so. The general concept of caching has been well proven to be overwhelmingly worth it in pretty much any context it's possible.
- api 3y ago
  If you ran the transpiled binaries with a security flag prohibiting execution of generated code, that would cause anything with a jit to crash neatly and possibly with an informative error message. Docs for this system would have to be clear that jits will not work and must be native. I think Linux has some kind of no-exec flag like that. I know macOS does.
  - dmitrygr 3y ago
    that would not save you from someone doing things like: ((void (*)(void))(((uintptr_t)&myfunc) + 1))(); and there you go - on x86 this could plausibly be a way to obfuscate code since it has variable-length instrs. And your static analysis may not catch this. This does not require writeable executable pages or JIT. Hell, i can receive a uint32 from the network, cast to a func pointer, and jump to it. And for all you know, my protocol relies on a lack of ASLR and sends me a valid entry point addr in my executable, which will have changed in the translation you do. (Fun story: long ago, AIM's anti-protocol-cloning code did that - server would send a VA and ask for a few bytes at that VA. Unless you had a current copy of the aim executable in memory running you had no way to correctly reply) The long tail of valid but freaky shit is ....very long, and disallowing W&X pages will not protect you from it all
    - akoboldfrying 3y ago
      Correct, your incremented function pointer won't be caught -- but now we're really getting down to the parts-per-million of programs that use this. Demoscene code, sometimes, maybe? I would say W^X protection will protect you from all non-adversarial programs.
      - dmitrygr 3y ago
        Due to anti piracy and anti cheat mechanisms, games are also likely to pull shit like that
- saagarjha 3y ago
  Have you ever written Java code? Or Lua? Or (as of recently) Python, Ruby, …?

aleclm 3y ago

You can already do that with QEMU: https://wiki.debian.org/QemuUserEmulation https://wiki.debian.org/QemuUserEmulation However, the code we emit is faster than the one emitted by QEMU. We have a secret plan to use LLVM in QEMU as a tier 2 JIT engine to optimize hot code paths more aggressively. This has already been done, but never upstreamed. Adding an on-disk cache would also be quite useful.

ris 3y ago
> This has already been done, but never upstreamed Assume you're referring to hqemu http://csl.iis.sinica.edu.tw/hqemu/ http://csl.iis.sinica.edu.tw/hqemu/ > Adding an on-disk cache would also be quite useful. +100
aengelke 3y ago
> [LLVM in QEMU] has already been done, but never upstreamed. Yeah, that's a long history, and it never worked great, the architectures are too different. LLVM is great for compiling functions while QEMU compiles basic blocks, which is somewhat required for correctness. LLVM for binary translation requires longer superblocks (HQEMU) or functions (Instrew). I don't see _correct_ binary translation with LLVM as viable anytime soon, though: every memory access can cause a synchronous signal (SIGSEGV/SIGBUS), and I'd expect recovering the full register state at the prior instruction boundary from the LLVM-compiled code to come at severe performance penalties. > Adding an on-disk cache would also be quite useful. I implemented a persistent cache in Instrew to work around LLVM's high compile times. Cache access is done by decoding the code-to-be-translated (very fast) and using a hash of that code for lookup. So shared libraries and even the same code in different binaries will hit the cache, regardless of their address or path name. If translation is fast (QEMU/TCG), caching is probably not very useful, though.
- aleclm 3y ago
  > every memory access can cause a synchronous signal (SIGSEGV/SIGBUS) AFAIU you have all the same problems as soon as you translate more than one instruction at a time and allow merging them, which QEMU does, even if just at basic block level. IIRC post-SIGSEGV state is not 100% correct in QEMU, but I'd need to investigate some more. We built a dynamic binary translator for a customer dealing with mainframes using ORC JIT where we translate large amounts of code, handle self-modifying code and caching things on disk (similar approach to the one you suggest). Compile time is a problem, but then we the performance results are impressive. Giving visibility over loops to LLVM helps a lot. But the real reason why nothing like this is upstream is that it's difficult to get things upstream, it requires a lot of effort compared to putting together a PoC. :)