6 ms·

Transparently running binaries from any architecture in Linux (2018)

104 points by billwashere 4y ago

ranger_danger 4y ago
You can also do this with docker containers for other architectures + the binfmt qemu-user trick, which may be easier to work with pre-existing rootfs images and other software.
hsnewman 4y ago
This is exaggeration, I doubt that it can run AS/400 binaries.
- stingraycharles 4y ago
  Well looks like qemu does have support S390X, which is pretty impressive. https://wiki.qemu.org/Documentation/Platforms/S390X https://wiki.qemu.org/Documentation/Platforms/S390X
  - Teknoman117 4y ago
    I've actually used that a few times. I've never seen a mainframe (in person), nor do I think I ever will without a career change, but I did need to make some changes to a build system in an open source project that had wide architecture support. I was using qemu to check that all of the binaries for the various supported architectures actually ran in some capacity.
    - pabs3 4y ago
      BTW, there are a few different ways to get access to mainframe hardware rather than using qemu. In short, the Debian porterboxen, the Deb-o-matic service and the IBM LinuxOne cloud. https://wiki.debian.org/Hardware/Wanted#Available_hardware https://wiki.debian.org/Hardware/Wanted#Available_hardware
  - rurban 4y ago
    The only remaining supported Big Ending architecture today, to test BE quirks.
snvzz 4y ago
I have found this useful in practice when debootstrapping for a different architecture.
fargle 4y ago
Toward the end of the article they use chroot to run an entire rootfs as sort of a user-level system emulation. The next step is to do the same thing except using containers/namespaces. I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc. A very handy technique for testing and it does run much faster than full-system emulation.
- spyremeown 4y ago
  >I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc I'd love to try that! Any pointers? :)
  - fargle 4y ago
    very, very, roughly: - build a rootfs using the poky reference distro (but do it for your arm target). - https://docs.yoctoproject.org/ https://docs.yoctoproject.org/ - https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html https://docs.yoctoproject.org/brief-yoctoprojectqs/index.htm... - you'll need to make or get a layer for your machine type. for example, for rpi you'll want: https://github.com/agherzan/meta-raspberrypi https://github.com/agherzan/meta-raspberrypi bitbake core-image-minimal (or whatever the machine layer wants you to do) - find the unpacked root image (or unpack the final image). should be like build/tmp/work/<machine>/<blah>-image/1.0-r0/rootfs - run (note: in the past I used LXC, but lets try podman today) apt install qemu-user-static podman podman run -it --rootfs <rootfsloc> /sbin/init
- Twirrim 4y ago
  I used this approach to start iterating on some Arm builds before I got access to any of the Arm servers they were introducing at work (Oracle Cloud Infrastructure). I'd started out using a full emulation VM, and it was alright, but the cost of emulation was crippling for parts of the build process. IIRC one part of the build process was pulling in python libraries that didn't have arm wheels, and that took a bit of work to compile even on native architecture. Add in the overhead of full system emulation and it really hurt the iteration process. Especially as I worked my way from "Finally got it to build!" through to "Got the build repeatable from scratch!" The binfmt / container approach dramatically reduced the amount of emulation being done, resulting in phenomenally faster build times. Then I finally got access to an actual Arm instance and the entire process took even a fraction of that time.
mkoubaa 4y ago
What exactly does "transparently" mean in this context. I've seen that term used in a dozen different ways within software engineering
- seba_dos1 4y ago
  It's just a process in your system that acts the same way as native one.
- robonerd 4y ago
  In a manner that hides the nitty gritty details from the user. I don't know why it's called transparent instead of opaque.
sgtnoodle 4y ago
I've been using this technique to manipulate raspberry pi OS images for use in embedded system prototypes. It's very easy to set up. It's also nice to be able to use the image's embedded toolchain rather than set up a proper cross-compiler. It's slow to compile stuff due to the emulation, but relatively foolproof. Of course, the best long term solution is to use something like yocto or buildroot, but that takes considerable time and knowledge to do properly.
skykooler 4y ago
Does this work for graphical programs?
- jeroenhd 4y ago
  I don't see why not, as long as those programs don't need platform-specific drivers. In the end, the ARM programs will all speak X11 or Wayland, and those calls should transfer back to the native display renderer without any issues. I've had more challenges getting the necessary dependencies installed for foreign architectures than I've had issues running them through qemu. It's honestly surprising how easy it is and it makes me wonder why Windows doesn't have something similar.
giomasce 4y ago
First: you don't necessarily need an alternative chroot, Debian allows installing packages of foreign architectures in the same main tree. It has some hiccups, but it should mostly work. Second: of you like playing with foreign architectures, I have a collection of ready to boot Debian images of many architectures, that you can promptly boot with QEMU. Command line included. It is mostly aimed at full system emulation, though (but if you look through the cogs you can also download chroots). https://people.debian.org/~gio/dqib/ https://people.debian.org/~gio/dqib/
quag 4y ago
For those on NixOS add boot.binfmt.emulatedSystems = ["aarch64-linux"]; to /etc/nixos/configuration.nix and it will enable running ARM binaries. https://search.nixos.org/options?channel=21.11&show=boot.binfmt.emulatedSystems&from=0&size=50&sort=relevance&type=packages&query=binfmt https://search.nixos.org/options?channel=21.11&show=boot.bin...
- metadat 4y ago
  Does Nix use qemu or a different technique to pull off this one-liner magic?
  - zimbatm 4y ago
    There is one emulator mapped per target system. Depending on the target system it will use qemu, wine, wasmtime or mmixware. https://github.com/NixOS/nixpkgs/blob/111839dcf6e9a8bac6972e234b8aeb88e5c2269e/lib/systems/default.nix#L144-L172 https://github.com/NixOS/nixpkgs/blob/111839dcf6e9a8bac6972e...
  - deleted 4y ago
    [deleted]
anonymousDan 4y ago
I presume it is not safe to do this with malware?
FooBarWidget 4y ago
How does qemu-user's performance compare to Rosetta 2? The latter is marketed as nearly native performance because it performs binary translation. But I read that qemu-user also performs binary translation.
lproven 4y ago
All that work to recreate what Inferno (Unix 3) did out of the box over 25 years ago. FOSS Unix: 42 steps forward, 42 steps back. :-/

104 points by billwashere 4y ago

You can also do this with docker containers for other architectures + the binfmt qemu-user trick, which may be easier to work with pre-existing rootfs images and other software.

hsnewman 4y ago

This is exaggeration, I doubt that it can run AS/400 binaries.

stingraycharles 4y ago
Well looks like qemu does have support S390X, which is pretty impressive. https://wiki.qemu.org/Documentation/Platforms/S390X https://wiki.qemu.org/Documentation/Platforms/S390X
- Teknoman117 4y ago
  I've actually used that a few times. I've never seen a mainframe (in person), nor do I think I ever will without a career change, but I did need to make some changes to a build system in an open source project that had wide architecture support. I was using qemu to check that all of the binaries for the various supported architectures actually ran in some capacity.
  - pabs3 4y ago
    BTW, there are a few different ways to get access to mainframe hardware rather than using qemu. In short, the Debian porterboxen, the Deb-o-matic service and the IBM LinuxOne cloud. https://wiki.debian.org/Hardware/Wanted#Available_hardware https://wiki.debian.org/Hardware/Wanted#Available_hardware
- rurban 4y ago
  The only remaining supported Big Ending architecture today, to test BE quirks.

snvzz 4y ago

I have found this useful in practice when debootstrapping for a different architecture.

fargle 4y ago

Toward the end of the article they use chroot to run an entire rootfs as sort of a user-level system emulation. The next step is to do the same thing except using containers/namespaces. I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc. A very handy technique for testing and it does run much faster than full-system emulation.

spyremeown 4y ago
>I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc I'd love to try that! Any pointers? :)
- fargle 4y ago
  very, very, roughly: - build a rootfs using the poky reference distro (but do it for your arm target). - https://docs.yoctoproject.org/ https://docs.yoctoproject.org/ - https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html https://docs.yoctoproject.org/brief-yoctoprojectqs/index.htm... - you'll need to make or get a layer for your machine type. for example, for rpi you'll want: https://github.com/agherzan/meta-raspberrypi https://github.com/agherzan/meta-raspberrypi bitbake core-image-minimal (or whatever the machine layer wants you to do) - find the unpacked root image (or unpack the final image). should be like build/tmp/work/<machine>/<blah>-image/1.0-r0/rootfs - run (note: in the past I used LXC, but lets try podman today) apt install qemu-user-static podman podman run -it --rootfs <rootfsloc> /sbin/init
Twirrim 4y ago
I used this approach to start iterating on some Arm builds before I got access to any of the Arm servers they were introducing at work (Oracle Cloud Infrastructure). I'd started out using a full emulation VM, and it was alright, but the cost of emulation was crippling for parts of the build process. IIRC one part of the build process was pulling in python libraries that didn't have arm wheels, and that took a bit of work to compile even on native architecture. Add in the overhead of full system emulation and it really hurt the iteration process. Especially as I worked my way from "Finally got it to build!" through to "Got the build repeatable from scratch!" The binfmt / container approach dramatically reduced the amount of emulation being done, resulting in phenomenally faster build times. Then I finally got access to an actual Arm instance and the entire process took even a fraction of that time.

mkoubaa 4y ago

What exactly does "transparently" mean in this context. I've seen that term used in a dozen different ways within software engineering

seba_dos1 4y ago
It's just a process in your system that acts the same way as native one.
robonerd 4y ago
In a manner that hides the nitty gritty details from the user. I don't know why it's called transparent instead of opaque.

sgtnoodle 4y ago

I've been using this technique to manipulate raspberry pi OS images for use in embedded system prototypes. It's very easy to set up. It's also nice to be able to use the image's embedded toolchain rather than set up a proper cross-compiler. It's slow to compile stuff due to the emulation, but relatively foolproof. Of course, the best long term solution is to use something like yocto or buildroot, but that takes considerable time and knowledge to do properly.

skykooler 4y ago

Does this work for graphical programs?

jeroenhd 4y ago
I don't see why not, as long as those programs don't need platform-specific drivers. In the end, the ARM programs will all speak X11 or Wayland, and those calls should transfer back to the native display renderer without any issues. I've had more challenges getting the necessary dependencies installed for foreign architectures than I've had issues running them through qemu. It's honestly surprising how easy it is and it makes me wonder why Windows doesn't have something similar.

giomasce 4y ago

First: you don't necessarily need an alternative chroot, Debian allows installing packages of foreign architectures in the same main tree. It has some hiccups, but it should mostly work. Second: of you like playing with foreign architectures, I have a collection of ready to boot Debian images of many architectures, that you can promptly boot with QEMU. Command line included. It is mostly aimed at full system emulation, though (but if you look through the cogs you can also download chroots). https://people.debian.org/~gio/dqib/ https://people.debian.org/~gio/dqib/

quag 4y ago

For those on NixOS add boot.binfmt.emulatedSystems = ["aarch64-linux"]; to /etc/nixos/configuration.nix and it will enable running ARM binaries. https://search.nixos.org/options?channel=21.11&show=boot.binfmt.emulatedSystems&from=0&size=50&sort=relevance&type=packages&query=binfmt https://search.nixos.org/options?channel=21.11&show=boot.bin...

metadat 4y ago
Does Nix use qemu or a different technique to pull off this one-liner magic?
- zimbatm 4y ago
  There is one emulator mapped per target system. Depending on the target system it will use qemu, wine, wasmtime or mmixware. https://github.com/NixOS/nixpkgs/blob/111839dcf6e9a8bac6972e234b8aeb88e5c2269e/lib/systems/default.nix#L144-L172 https://github.com/NixOS/nixpkgs/blob/111839dcf6e9a8bac6972e...
- deleted 4y ago
  [deleted]

anonymousDan 4y ago

I presume it is not safe to do this with malware?

FooBarWidget 4y ago

How does qemu-user's performance compare to Rosetta 2? The latter is marketed as nearly native performance because it performs binary translation. But I read that qemu-user also performs binary translation.

lproven 4y ago

All that work to recreate what Inferno (Unix 3) did out of the box over 25 years ago. FOSS Unix: 42 steps forward, 42 steps back. :-/