4 ms·

Usercorn – a versatile kernel+system+userspace emulator

37 points by omn1 11y ago

brudgers 11y ago
Unicorn engine home: http://www.unicorn-engine.org/ http://www.unicorn-engine.org/
- cm3 11y ago
  Is Unicorn a fork which will eventually grow into another Qemu or just a fork to remove stuff and make it into something else that doesn't have the same goals. I'm confused because the website on the one hand says it's less functionality but then also that it does more than qemu. So which is it? Or is it just that they provide bindings and an API which they tout as "more than qemu".
  - lunixbochs 11y ago
    Unicorn is a massive patch on top of QEMU 2.2 [1] that approximately does the following: - Removes a couple of architectures (because they're not supported yet for hooks/register access). - Removes hardware support, as well as the qemu-system and qemu-user targets. - Modifies the JIT and all supporting APIs to not use globals. This allows you to compile JIT support for more than one guest architecture into the same binary, as well as run more than one instance of the JIT at once (so you can emulate two CPUs in parallel, even of different architectures). - Adds a hook infrastructure to call callback function(s) with metadata on many different events, including memory access, memory fault, interrupt, basic block entry, instruction execution (so like per instruction address). These calls are inserted into the JIT stream only when you add a hook, so they're reasonably efficient. - Adds register read/write APIs (this is actually a ton of code in the form of a switch/case table for each arch) - Exposes all of this using a simple C library / API: uc_open(arch, mode) uc_mem_map() uc_mem_write/read() uc_reg_write/read() uc_emu_start/stop() uc_hook_add/del() [1] I'm currently porting it to QEMU 2.5 because I need one new ARM NEON instruction. This is probably overkill, but it's a good cause.
    - cm3 11y ago
      Thanks for the detailed explanation. Does that mean it's not a replacement for QEMU and not intended to be? Regarding qemu-user and qemu-system, I've had bad luck with it all the time. In 2.5-rc2 I got qemu-system-x86_64 but when I build 2.5 with the same configure flags, I don't get qemu-system-x86_64 at all. You wouldn't happen to know how to work around that, would you? As I use it just for kvm functionality for vms, I configure it like this, but qemu-2.5 final's build system seems to be broken. --target-list=x86_64-linux-user --enable-pie --enable-seccomp --enable-curses The last known version that was able to build qemu-system-x86_64 is 2.4.92. Any idea?
      - lunixbochs 11y ago
        1) It really depends on the features of QEMU you use. Unicorn and friends could eventually support every feature of QEMU, but currently are nowhere near that. Usercorn should be able to replace qemu-user eventualy. 2) You should really go to the QEMU team for support on that.
        cm3 11y ago
        2) Seems like the configure (and/or build) script was broken at some point. Letting it build all targets does not omit the amd64 kvm executable. 1) But I don't really understand the end goal if you're always forking of the next qemu relese to patch stuff back in. Is there no interest in qemu upstream for those features? I've seen it for the first time 2.6-rc, so is qapi some kind of API you might adapt or is it too limited?
        lunixbochs 11y ago
        At this point Unicorn is a hard fork with a very specific goal. It accomplishes this goal much better than upstream QEMU. I believe combined TCG / no-globals should be upstreamed (I think I actually saw commits in 2.5 about supporting multiple architectures in one QEMU binary?), but I don't know if anyone working on Unicorn has talked to anyone upstream about this. I believe Unicorn also contains QEMU bugfixes. To me, in an ideal world, we might have something like this: libqemu-cpu - Basically Unicorn. C library/API to easily create/hook/manipulate one or more CPUs. libqemu-system - All hardware/peripheral support from QEMU. Attach any component(s) to a supported CPU at runtime. libqemu-user - The current qemu-user, except as a library with syscall/whatever hooking support. There'd probably be a good amount of code generation here.
lunixbochs 11y ago
Author here. Usercorn at a basic level is like qemu-user in that it can load a userspace binary like ELF, MachO, etc into a CPU emulator and provide a syscall emulator interface. It's built as an API that allows instrumentation of the resulting binary. You can hook instructions, basic blocks, syscalls, memory access... You get information like stacktraces in a cross-platform manner. There are helpers to compute and display diffs of memory and registers throughout execution. There are many differences from qemu-user: -- Usercorn attempts to emulate several OS/syscall ABIs all from one binary (Linux, BSD, XNU, CGC, more to come like Redox, maybe weird stuff like Windows and DOS). qemu-user just translates arguments to your host kernel so to run a Linux binary you need a Linux host. qemu-user also has a different binary for each arch, like qemu-i386 and qemu-arm. The primary usercorn binary can emulate every single supported OS/arch combination and aims automatically detect the correct arch/kernel when you run a program. Usercorn is being built as a framework around the Unicorn Engine. It's not just a way to run a binary on the command line. It can be loaded as a library (`NewUsercorn("binary").Run(args, env)` will run an arbitrary supported app). It allows hooking many things and I've been building a lot of features around static trace/analysis. I'm extremely excited about some of the features and supporting tools I have planned. This twitter thread shows examples of a tool called imgtrace I built with Usercorn. imgtrace can visualize memory writes done by a program. The Hilbert curve view of MD5 is nice, as it makes the memory containing the internal MD5 state as well as the iterators very obvious at a glance: https://twitter.com/lunixbochs/status/717945725024280578 https://twitter.com/lunixbochs/status/717945725024280578 The underlying code of imgtrace is also extremely simple (basically all of it is to create the image data), which shows off the power of Usercorn's simplicity: https://github.com/lunixbochs/usercorn/tree/master/go/cmd/imgtrace https://github.com/lunixbochs/usercorn/tree/master/go/cmd/im... Usercorn is much less mature than qemu-user. It supports around ~50/400 Posix syscalls and is missing many architecture-specific features. Some architectures still need work on memory segmentation and thread-local storage. I've split up functionality internally in a way that makes sense, but I'm still building the base functionality and haven't designed the public API at all yet (even though it's possible to use Usercorn from external code, that's just happenstance due to the functions/types I exposed while building it). ---- Architecture support is tied to Unicorn. Usercorn supports x86_64 best, and has various levels of support for ARM, MIPS, sparc, and m68k. It's still very much WIP, but can run many binaries at this point, even Linux binaries dynamically linked to glibc. Host support is best on OS X and Linux. Guest support is best on Linux.
tetrep 11y ago
One of the many practical neat things you can do when you're emulating all the software of a machine, is visually analyzing memory relatively easily: https://www.youtube.com/watch?v=25nLk1Z8wd0 https://www.youtube.com/watch?v=25nLk1Z8wd0

37 points by omn1 11y ago

brudgers 11y ago

Unicorn engine home: http://www.unicorn-engine.org/ http://www.unicorn-engine.org/

cm3 11y ago
Is Unicorn a fork which will eventually grow into another Qemu or just a fork to remove stuff and make it into something else that doesn't have the same goals. I'm confused because the website on the one hand says it's less functionality but then also that it does more than qemu. So which is it? Or is it just that they provide bindings and an API which they tout as "more than qemu".
- lunixbochs 11y ago
  Unicorn is a massive patch on top of QEMU 2.2 [1] that approximately does the following: - Removes a couple of architectures (because they're not supported yet for hooks/register access). - Removes hardware support, as well as the qemu-system and qemu-user targets. - Modifies the JIT and all supporting APIs to not use globals. This allows you to compile JIT support for more than one guest architecture into the same binary, as well as run more than one instance of the JIT at once (so you can emulate two CPUs in parallel, even of different architectures). - Adds a hook infrastructure to call callback function(s) with metadata on many different events, including memory access, memory fault, interrupt, basic block entry, instruction execution (so like per instruction address). These calls are inserted into the JIT stream only when you add a hook, so they're reasonably efficient. - Adds register read/write APIs (this is actually a ton of code in the form of a switch/case table for each arch) - Exposes all of this using a simple C library / API: uc_open(arch, mode) uc_mem_map() uc_mem_write/read() uc_reg_write/read() uc_emu_start/stop() uc_hook_add/del() [1] I'm currently porting it to QEMU 2.5 because I need one new ARM NEON instruction. This is probably overkill, but it's a good cause.
  - cm3 11y ago
    Thanks for the detailed explanation. Does that mean it's not a replacement for QEMU and not intended to be? Regarding qemu-user and qemu-system, I've had bad luck with it all the time. In 2.5-rc2 I got qemu-system-x86_64 but when I build 2.5 with the same configure flags, I don't get qemu-system-x86_64 at all. You wouldn't happen to know how to work around that, would you? As I use it just for kvm functionality for vms, I configure it like this, but qemu-2.5 final's build system seems to be broken. --target-list=x86_64-linux-user --enable-pie --enable-seccomp --enable-curses The last known version that was able to build qemu-system-x86_64 is 2.4.92. Any idea?
    - lunixbochs 11y ago
      1) It really depends on the features of QEMU you use. Unicorn and friends could eventually support every feature of QEMU, but currently are nowhere near that. Usercorn should be able to replace qemu-user eventualy. 2) You should really go to the QEMU team for support on that.
      - cm3 11y ago
        2) Seems like the configure (and/or build) script was broken at some point. Letting it build all targets does not omit the amd64 kvm executable. 1) But I don't really understand the end goal if you're always forking of the next qemu relese to patch stuff back in. Is there no interest in qemu upstream for those features? I've seen it for the first time 2.6-rc, so is qapi some kind of API you might adapt or is it too limited?
        lunixbochs 11y ago
        At this point Unicorn is a hard fork with a very specific goal. It accomplishes this goal much better than upstream QEMU. I believe combined TCG / no-globals should be upstreamed (I think I actually saw commits in 2.5 about supporting multiple architectures in one QEMU binary?), but I don't know if anyone working on Unicorn has talked to anyone upstream about this. I believe Unicorn also contains QEMU bugfixes. To me, in an ideal world, we might have something like this: libqemu-cpu - Basically Unicorn. C library/API to easily create/hook/manipulate one or more CPUs. libqemu-system - All hardware/peripheral support from QEMU. Attach any component(s) to a supported CPU at runtime. libqemu-user - The current qemu-user, except as a library with syscall/whatever hooking support. There'd probably be a good amount of code generation here.

lunixbochs 11y ago

Author here. Usercorn at a basic level is like qemu-user in that it can load a userspace binary like ELF, MachO, etc into a CPU emulator and provide a syscall emulator interface. It's built as an API that allows instrumentation of the resulting binary. You can hook instructions, basic blocks, syscalls, memory access... You get information like stacktraces in a cross-platform manner. There are helpers to compute and display diffs of memory and registers throughout execution. There are many differences from qemu-user: -- Usercorn attempts to emulate several OS/syscall ABIs all from one binary (Linux, BSD, XNU, CGC, more to come like Redox, maybe weird stuff like Windows and DOS). qemu-user just translates arguments to your host kernel so to run a Linux binary you need a Linux host. qemu-user also has a different binary for each arch, like qemu-i386 and qemu-arm. The primary usercorn binary can emulate every single supported OS/arch combination and aims automatically detect the correct arch/kernel when you run a program. Usercorn is being built as a framework around the Unicorn Engine. It's not just a way to run a binary on the command line. It can be loaded as a library (`NewUsercorn("binary").Run(args, env)` will run an arbitrary supported app). It allows hooking many things and I've been building a lot of features around static trace/analysis. I'm extremely excited about some of the features and supporting tools I have planned. This twitter thread shows examples of a tool called imgtrace I built with Usercorn. imgtrace can visualize memory writes done by a program. The Hilbert curve view of MD5 is nice, as it makes the memory containing the internal MD5 state as well as the iterators very obvious at a glance: https://twitter.com/lunixbochs/status/717945725024280578 https://twitter.com/lunixbochs/status/717945725024280578 The underlying code of imgtrace is also extremely simple (basically all of it is to create the image data), which shows off the power of Usercorn's simplicity: https://github.com/lunixbochs/usercorn/tree/master/go/cmd/imgtrace https://github.com/lunixbochs/usercorn/tree/master/go/cmd/im... Usercorn is much less mature than qemu-user. It supports around ~50/400 Posix syscalls and is missing many architecture-specific features. Some architectures still need work on memory segmentation and thread-local storage. I've split up functionality internally in a way that makes sense, but I'm still building the base functionality and haven't designed the public API at all yet (even though it's possible to use Usercorn from external code, that's just happenstance due to the functions/types I exposed while building it). ---- Architecture support is tied to Unicorn. Usercorn supports x86_64 best, and has various levels of support for ARM, MIPS, sparc, and m68k. It's still very much WIP, but can run many binaries at this point, even Linux binaries dynamically linked to glibc. Host support is best on OS X and Linux. Guest support is best on Linux.

tetrep 11y ago

One of the many practical neat things you can do when you're emulating all the software of a machine, is visually analyzing memory relatively easily: https://www.youtube.com/watch?v=25nLk1Z8wd0 https://www.youtube.com/watch?v=25nLk1Z8wd0