6 ms·

Is there any practical purpose for APE? Like who is shipping single executables that need to run on N OSes?

Timon3 3y ago
I'm trying to release future side projects that way. I even build a Redbean executable for any client-only SPAs. The idea of this stuff just working on so many systems is amazing.
- nicoburns 3y ago
  Problem with this is that self-modifying code will likely trip security blocks on most platforms, which is probably more annoying for users than having to download separate executables.
  - ElectricalUnion 3y ago
    If the person needs to run with "security blocks" up, it's statistically unlikely that the same person can randomly download the specific and potentially outdated version of a dependency you need for the code to run in the first place.
jjnoakes 3y ago
Storing executables on file systems shared across operating systems or architectures gets a lot easier if you don't have to have special paths for each os+arch combination.
- saagarjha 3y ago
  Doesn’t the executable get overwritten after first invocation?
  - jjnoakes 3y ago
    I don't think so.
  - ElectricalUnion 3y ago
    Only if you specifically run it with the `--assimilate` flag.
- dataangel 3y ago
  I've seen this solved before by giving each combo a different mount with the same path.
jcelerier 3y ago
i've been asked this hundreds of times in my life, by customers, bosses, etc...
- mhh__ 3y ago
  But they don't know or care the difference between the code being in the same binary or not
  - ElectricalUnion 3y ago
    But they know and care that "it doesn't run" even if they don't know how and why.
  - jcelerier 3y ago
    I mean, yes they care, they want to be able to put the executable in a usb key or an email and that it runs on any computer they would plug it into
lmm 3y ago
I can imagine it making it slightly easier to distribute executables from e.g. maven repositories (where having multiple artifacts for the same id/version combination is cumbersome). I think PyPI might have a similar issue? Also cross-compiling is painful; a lot of small OSS projects that don't have a build farm with all of the different BSDs etc. to hand simply don't ship binaries for those. Even e.g. kubectl has a binary for Linux but not for any of the BSDs (unless you count OSX). So this seems like an easy win for those.
corysama 3y ago
Java/Closure/Kotlin programmers. .Net programmers. JS, Python, Ruby, Erlang, every VM-based language programmers. Statically-compiled programs are under the hood of everything, but at the application level in the minority. APE is a big “Who needs a VM if an AOT compiled language as low level as C can compile once and run universally?”
- up2isomorphism 3y ago
  Portability is not the reason you see VMs everywhere.
  - londons_explore 3y ago
    I think it is a big part of it.
- pjmlp 3y ago
  If libc is the only leaf dependency.
  - flohofwoe 3y ago
    Which is true for a lot of useful tools that run in a terminal (for many coders this is pretty much everything except an IDE and a browser).
    - pjmlp 3y ago
      If their world is ISO C and nothing else, quite a boring one actually. Not even a bit of colour, as terminal escape sequences assume a POSIX host environment, and will throw up gibberish if not.
      - flohofwoe 3y ago
        Nah, terminal colors via escape sequences work fine on Windows, even on the "traditional" cmd.exe. I use that all the time in cross-platform Python and Typescript cmdline tools (cmd.exe is probably limited to 16 colors though, but that's just how it is).
tstack 3y ago
I use an APE executable as an agent for communicating with remote hosts in the Logfile Navigator (https://lnav.org https://lnav.org). While lnav itself is not built as an APE, the agent built into itself is. That agent is transferred to the remote when the user wants to read logs on that host. This way, there is no extra step to determine the type of OS or building in multiple versions of the executable. Here's a short blog post on the subject: https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html
- electroly 3y ago
  My biggest problem with APE: not portable enough, in the sense that when people adopt APE, they lock themselves into less portability than they might have supported otherwise. I live in an ARM world and because of APE you only offer Intel x64 binaries. All new Macs are ARM64, Raspberry Pi models are a mix of ARM32 and ARM64 (mine is ARM64), and I use exclusively Graviton instances in AWS. Running x64 binaries with QEMU/Rosetta emulation on an ARM system is a cute trick but not something I want to do for a piece of infrastructure. Raspberry Pis are barely fast enough to run native binaries. Consider offering APE for x64 but then still producing ARM binaries the old fashioned way. APE is pushing the idea that x64 is a good bytecode in a world that is moving from x64 to ARM. I'd like to see the project accept that ARM exists and produce an ARM version of APE, maybe with some jart flourish--can we make a cross-platform and cross-architecture fat binary?
  - cesarb 3y ago
    > APE is pushing the idea that x64 is a good bytecode And x86-64 is a particularly bad bytecode, for one main reason: its strong memory ordering rules, compared to nearly every other architecture. IIRC, simple things like doing a store in x86-64 have an implicit release barrier, while other architectures can freely reorder the stores unless there's an explicit barrier. This means that a x86-64 emulator on for instance ARM has to either be single-threaded (and would still have issues with shared mmap regions), have special hardware support for x86-compatible memory ordering (as found for instance on recent Apple ARM CPUs, and used for their x86-64 emulation, but not common elsewhere), or add explicit barriers everywhere (which kills the performance). The only real advantage of x86-64 as a bytecode would be that it has less registers than most other architectures (only 16 registers, while other 64-bit architectures usually have around 32 registers), which allows a 1:1 mapping of the registers on the emulator while still leaving plenty of them free for temporary use.
flohofwoe 3y ago
I have a shader compiler which would benefit from a platform agnostic, ready-to-run executable format. It's too big to be build ad-hoc on the user's machine (contains various big 3rd party C++ dependencies, and takes between 2 and 20 minutes to build, depending on the hardware). Needs to run on macOS x86-64+ARM, Linux x86-64 (ARM would be nice too of course), and Windows x86-64. I'm currently looking into WASM, but that needs an installed WASI runtime.
efrecon 3y ago
In vscode extensions that rely on an external binary to do most of the job?
nonameiguess 3y ago
It's much more useful for greenfield software. Mad props to whoever wrote the gcc patch just to get something like this work some of the time, but rebuilding existing software is probably a fool's errand. There's a good reason you see things like the rpm package format include not only kernel but distro. Running on "Linux," BSD, Windows, or MacOS is one thing, but Linux is a bunch of different OSes, not one. A whole lot of software out there relies upon assuming you're using a particular DNS provider, filesystem hierarchy that may or may not be FHS-compliant. When I went down a rabbit hole a few years back trying to extend Linux from Scratch to include a bunch more packages, I found that GHC won't build if the file "/etc/os-release" doesn't exist. The file can be empty, but it has to exist. Someone mentioned busybox elsewhere, which sure, will build with this if you disable a whole bunch of stuff, but busybox is supposed to be the minimal utilities needed to run something close to POSIX-compliant, which obviously assumes your system is a POSIX system. It also includes most of the stuff usually provided by util-linux, which obviously won't run except on Linux. If your software is doing something like trying to read from particular device files to get hardware info from the kernel, sometimes that will work on both Linux and BSD (and even Mac since it's kind of a BSD), but it definitely won't work on Windows. For software that uses the network, how do you query DNS? If you're calling getaddrinfo, I guess Cosmo libc will do some magic for you to make sure that works everywhere, but a lot of software not written in C is doing something not quite that easily made universal, like just assuming /etc/resolv.conf exists and reading it, which definitely won't work on Windows, or using the native platform APIs, which will only work on one platform. If you distribute desktop software, I would say good luck. Are you going to comply with the XDG? Well, now I guess it will "work" on Windows, but you're not complying with Windows conventions where you're supposed to put stuff in App\RoamingLow or whatever the hell that I would never remember without looking it up every time. Don't try storing data in the registry, because now it will only work on Windows. Does your software? It better be present on all systems then. Do you query any environment variables? I don't think LC_TIME and LC_COLLATE are provided anywhere but Linux usually. The XDG_ variables are only Linux. Does Windows have PAGER? I don't even know. I'm pretty sure PATH, USER, and HOME work everywhere. A truly actually portable executable (TAPE) would require far more than an executable file format that can successfully load into memory and execute its first instruction on any platform. There's a reason vendors just write software that uses a browser's JavaScript engine as a platform instead of the OS's native runtime. EDIT: I don't know if this is ironic, but I guess consider your choice of programming language, too. Go became popular in part because of the static linking, single executable thing, but it inherently can't be cross-platform since it doesn't use the C library and tries to make all calls into the kernel on its own by keeping track of the syscall table provided by every OS it supports. Well, they're still different on every OS, so an executable that works on Linux will not work anywhere else and vice versa.
regularfry 3y ago
A long time ago I wanted an Ansible equivalent that worked by injecting a bootstrap VM, then sending configuration code to run on the remote node, rather than chatting back and forth over an SSH tunnel. A lot of the slowness of applying config was that chattiness, and it would have been a lot faster to avoid it. I have no idea if there is a tool that actually works like this - I remember an exploit tool that did something similar, but it wasn't quite right - but making that initial bootstrap VM and a useful stdlib an APE executable would also have avoided the "there needs to be a working Python at the other end" issue which persisted in tripping me up for far longer than it should have.

I'm trying to release future side projects that way. I even build a Redbean executable for any client-only SPAs. The idea of this stuff just working on so many systems is amazing.

nicoburns 3y ago
Problem with this is that self-modifying code will likely trip security blocks on most platforms, which is probably more annoying for users than having to download separate executables.
- ElectricalUnion 3y ago
  If the person needs to run with "security blocks" up, it's statistically unlikely that the same person can randomly download the specific and potentially outdated version of a dependency you need for the code to run in the first place.

jjnoakes 3y ago

Storing executables on file systems shared across operating systems or architectures gets a lot easier if you don't have to have special paths for each os+arch combination.

saagarjha 3y ago
Doesn’t the executable get overwritten after first invocation?
- jjnoakes 3y ago
  I don't think so.
- ElectricalUnion 3y ago
  Only if you specifically run it with the `--assimilate` flag.
dataangel 3y ago
I've seen this solved before by giving each combo a different mount with the same path.

jcelerier 3y ago

i've been asked this hundreds of times in my life, by customers, bosses, etc...

mhh__ 3y ago
But they don't know or care the difference between the code being in the same binary or not
- ElectricalUnion 3y ago
  But they know and care that "it doesn't run" even if they don't know how and why.
- jcelerier 3y ago
  I mean, yes they care, they want to be able to put the executable in a usb key or an email and that it runs on any computer they would plug it into

lmm 3y ago

I can imagine it making it slightly easier to distribute executables from e.g. maven repositories (where having multiple artifacts for the same id/version combination is cumbersome). I think PyPI might have a similar issue? Also cross-compiling is painful; a lot of small OSS projects that don't have a build farm with all of the different BSDs etc. to hand simply don't ship binaries for those. Even e.g. kubectl has a binary for Linux but not for any of the BSDs (unless you count OSX). So this seems like an easy win for those.

corysama 3y ago

Java/Closure/Kotlin programmers. .Net programmers. JS, Python, Ruby, Erlang, every VM-based language programmers. Statically-compiled programs are under the hood of everything, but at the application level in the minority. APE is a big “Who needs a VM if an AOT compiled language as low level as C can compile once and run universally?”

up2isomorphism 3y ago
Portability is not the reason you see VMs everywhere.
- londons_explore 3y ago
  I think it is a big part of it.
pjmlp 3y ago
If libc is the only leaf dependency.
- flohofwoe 3y ago
  Which is true for a lot of useful tools that run in a terminal (for many coders this is pretty much everything except an IDE and a browser).
  - pjmlp 3y ago
    If their world is ISO C and nothing else, quite a boring one actually. Not even a bit of colour, as terminal escape sequences assume a POSIX host environment, and will throw up gibberish if not.
    - flohofwoe 3y ago
      Nah, terminal colors via escape sequences work fine on Windows, even on the "traditional" cmd.exe. I use that all the time in cross-platform Python and Typescript cmdline tools (cmd.exe is probably limited to 16 colors though, but that's just how it is).

tstack 3y ago

I use an APE executable as an agent for communicating with remote hosts in the Logfile Navigator (https://lnav.org https://lnav.org). While lnav itself is not built as an APE, the agent built into itself is. That agent is transferred to the remote when the user wants to read logs on that host. This way, there is no extra step to determine the type of OS or building in multiple versions of the executable. Here's a short blog post on the subject: https://lnav.org/2021/05/03/tailing-remote-files.html https://lnav.org/2021/05/03/tailing-remote-files.html

electroly 3y ago
My biggest problem with APE: not portable enough, in the sense that when people adopt APE, they lock themselves into less portability than they might have supported otherwise. I live in an ARM world and because of APE you only offer Intel x64 binaries. All new Macs are ARM64, Raspberry Pi models are a mix of ARM32 and ARM64 (mine is ARM64), and I use exclusively Graviton instances in AWS. Running x64 binaries with QEMU/Rosetta emulation on an ARM system is a cute trick but not something I want to do for a piece of infrastructure. Raspberry Pis are barely fast enough to run native binaries. Consider offering APE for x64 but then still producing ARM binaries the old fashioned way. APE is pushing the idea that x64 is a good bytecode in a world that is moving from x64 to ARM. I'd like to see the project accept that ARM exists and produce an ARM version of APE, maybe with some jart flourish--can we make a cross-platform and cross-architecture fat binary?
- cesarb 3y ago
  > APE is pushing the idea that x64 is a good bytecode And x86-64 is a particularly bad bytecode, for one main reason: its strong memory ordering rules, compared to nearly every other architecture. IIRC, simple things like doing a store in x86-64 have an implicit release barrier, while other architectures can freely reorder the stores unless there's an explicit barrier. This means that a x86-64 emulator on for instance ARM has to either be single-threaded (and would still have issues with shared mmap regions), have special hardware support for x86-compatible memory ordering (as found for instance on recent Apple ARM CPUs, and used for their x86-64 emulation, but not common elsewhere), or add explicit barriers everywhere (which kills the performance). The only real advantage of x86-64 as a bytecode would be that it has less registers than most other architectures (only 16 registers, while other 64-bit architectures usually have around 32 registers), which allows a 1:1 mapping of the registers on the emulator while still leaving plenty of them free for temporary use.

flohofwoe 3y ago

I have a shader compiler which would benefit from a platform agnostic, ready-to-run executable format. It's too big to be build ad-hoc on the user's machine (contains various big 3rd party C++ dependencies, and takes between 2 and 20 minutes to build, depending on the hardware). Needs to run on macOS x86-64+ARM, Linux x86-64 (ARM would be nice too of course), and Windows x86-64. I'm currently looking into WASM, but that needs an installed WASI runtime.

efrecon 3y ago

In vscode extensions that rely on an external binary to do most of the job?

nonameiguess 3y ago

It's much more useful for greenfield software. Mad props to whoever wrote the gcc patch just to get something like this work some of the time, but rebuilding existing software is probably a fool's errand. There's a good reason you see things like the rpm package format include not only kernel but distro. Running on "Linux," BSD, Windows, or MacOS is one thing, but Linux is a bunch of different OSes, not one. A whole lot of software out there relies upon assuming you're using a particular DNS provider, filesystem hierarchy that may or may not be FHS-compliant. When I went down a rabbit hole a few years back trying to extend Linux from Scratch to include a bunch more packages, I found that GHC won't build if the file "/etc/os-release" doesn't exist. The file can be empty, but it has to exist. Someone mentioned busybox elsewhere, which sure, will build with this if you disable a whole bunch of stuff, but busybox is supposed to be the minimal utilities needed to run something close to POSIX-compliant, which obviously assumes your system is a POSIX system. It also includes most of the stuff usually provided by util-linux, which obviously won't run except on Linux. If your software is doing something like trying to read from particular device files to get hardware info from the kernel, sometimes that will work on both Linux and BSD (and even Mac since it's kind of a BSD), but it definitely won't work on Windows. For software that uses the network, how do you query DNS? If you're calling getaddrinfo, I guess Cosmo libc will do some magic for you to make sure that works everywhere, but a lot of software not written in C is doing something not quite that easily made universal, like just assuming /etc/resolv.conf exists and reading it, which definitely won't work on Windows, or using the native platform APIs, which will only work on one platform. If you distribute desktop software, I would say good luck. Are you going to comply with the XDG? Well, now I guess it will "work" on Windows, but you're not complying with Windows conventions where you're supposed to put stuff in App\RoamingLow or whatever the hell that I would never remember without looking it up every time. Don't try storing data in the registry, because now it will only work on Windows. Does your software? It better be present on all systems then. Do you query any environment variables? I don't think LC_TIME and LC_COLLATE are provided anywhere but Linux usually. The XDG_ variables are only Linux. Does Windows have PAGER? I don't even know. I'm pretty sure PATH, USER, and HOME work everywhere. A truly actually portable executable (TAPE) would require far more than an executable file format that can successfully load into memory and execute its first instruction on any platform. There's a reason vendors just write software that uses a browser's JavaScript engine as a platform instead of the OS's native runtime. EDIT: I don't know if this is ironic, but I guess consider your choice of programming language, too. Go became popular in part because of the static linking, single executable thing, but it inherently can't be cross-platform since it doesn't use the C library and tries to make all calls into the kernel on its own by keeping track of the syscall table provided by every OS it supports. Well, they're still different on every OS, so an executable that works on Linux will not work anywhere else and vice versa.

regularfry 3y ago

A long time ago I wanted an Ansible equivalent that worked by injecting a bootstrap VM, then sending configuration code to run on the remote node, rather than chatting back and forth over an SSH tunnel. A lot of the slowness of applying config was that chattiness, and it would have been a lot faster to avoid it. I have no idea if there is a tool that actually works like this - I remember an exploit tool that did something similar, but it wasn't quite right - but making that initial bootstrap VM and a useful stdlib an APE executable would also have avoided the "there needs to be a working Python at the other end" issue which persisted in tripping me up for far longer than it should have.