5 ms·

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

by amelius 10mo ago

mdavid626 10mo ago
You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.
- fc417fc802 10mo ago
  Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.
  - mdavid626 10mo ago
    Yes, that's true. But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. These are just strange and confusing from the end users' perspective.
    - toast0 10mo ago
      > But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. Why would you include most of your dynamic libraries but not your libc? You could still run into problems if you (or your libraries) want to use syscalls that weren't available on older kernels or whatever.
      - mdavid626 10mo ago
        You can include it, but - either you use chroot, proot or similar to make /lib path contain your executable’s loader - or you hardcode different loader path into your executable Both are difficult for an end user.
        toast0 10mo ago
        This isn't that hard (that's not to say this is easy, it is tricky). Your executable should be a statically linked stub loader with an awful lot of data, the stub loader dynamically links your real executable (and libraries, including libc) from the data and runs it.
        fc417fc802 10mo ago
        To add to this, in case of any remaining confusion. You can implement your own execve in userspace. [0] But the kernel's execve is a piece of machinery that invokes the loader so obviously it follows that you're free to make any changes you'd like to the overall process. Bonus points if you add compression or encryption and manage to trip a virus scanner or three. [1] [0] https://grugq.github.io/docs/ul_exec.txt https://grugq.github.io/docs/ul_exec.txt [1] https://blackhat.com/presentations/bh-usa-07/Yason/Whitepaper/bh-usa-07-yason-WP.pdf#page=17 https://blackhat.com/presentations/bh-usa-07/Yason/Whitepape...
  - X-Ryl669 10mo ago
    No you can't. dlopen signature takes a file path, not a memory range. And if you start to save the libraries to the filesystem before opening them, there's no difference to shipping an archive directly and skip the trouble of your own archive code.
- geocar 10mo ago
  > But you can't take .so files and make one "static" binary out of them. Yes you can! This is more-or-less what unexec does - https://news.ycombinator.com/item?id=21394916 https://news.ycombinator.com/item?id=21394916 For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds. But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in one big file ahead of time! [1]: ASLR would be one of those things...
  - mdavid626 10mo ago
    What if the library you use calls dlopen later? That’ll fail. There is no universal, working way to do it. Only some hacks which work in some special cases.
    - geocar 10mo ago
      > What if the library you use calls dlopen later? That’ll fail. Nonsense. xemacs could absolutely call dlopen. > There is no universal, working way to do it. Only some hacks which work in some special cases. So you say, but I remember not too long ago you weren't even aware it was possible, and you clearly didn't check one of the most prominent users of this technique, so maybe you should also explain why I or anyone else should give a fuck about what you think is a "hack"?
marksugar 10mo ago
[dead]
TheDong 10mo ago
There are things like this. The things I know of and can think of off the top of my head are: 1. appimage https://appimage.org/ https://appimage.org/ 2. nix-bundle https://github.com/nix-community/nix-bundle https://github.com/nix-community/nix-bundle 3. guix via guix pack 4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc https://github.com/NilsIrl/dockerc ) 5. A docker image (a package that runs everywhere, assuming a docker runtime is available) 6. https://flatpak.org/ https://flatpak.org/ 7. https://en.wikipedia.org/wiki/Snap_(software) https://en.wikipedia.org/wiki/Snap_(software) AppImage is the closest to what you want I think.
- amelius 10mo ago
  AppImage looks like what I need, thanks. I wonder though, if I package say a .so file from nVidia, is that allowed by the license?
  - c0balt 10mo ago
    Depends on the license and the specific piece of software. Redistribution of commercial software is may be restricted or require explicit approval. You generally still also have to abide by license obligations for OSS too, e. G., GPL. To be specific for the exampls, Nvidia has historically been quite restrictive (only on approval) here. Firmware has only recently been opened up a bit and drivers continue to be an issue iirc.
  - mdavid626 10mo ago
    Don't forget - AppImage won't work if you package something with glibc, but run on musl/uclibc.
  - direwolf20 10mo ago
    No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.
    - amelius 10mo ago
aa-jv 10mo ago
AppImage comes close to fulfilling this need: https://appimage.github.io/appimagetool/ https://appimage.github.io/appimagetool/ Myself, I've committed to using Lua for all my cross-platform development needs, and in that regard I find luastatic very, very useful ..
formerly_proven 10mo ago
I don't think you can link shared objects into a static binary because you'd have to patch all instances where the code reads the PLT/GOT, but this can be arbitrarily mangled by the optimizer, and turn them back into relocations for the linker to then resolve them. You can change the rpath though, which is sort of like an LD_LIBRARY_PATH baked into the object, which makes it relatively easy to bundle everything but libc with your binary. edit: Mild correction, there is this: https://sourceforge.net/projects/statifier/ https://sourceforge.net/projects/statifier/ But the way this works is that it has the dynamic linker load everything (without ASLR / in a compact layout, presumably) and then dumps an image of the process. Everything else is just increasingly fancy ways of copying shared objects around and making ld.so prefer the bundled libraries.
- deleted 10mo ago
  [deleted]
fieu 10mo ago
Ermine: https://www.magicermine.com/ https://www.magicermine.com/ It works surprisingly well but their pricing is hidden and last time I contacted them as a student it was upwards of $350/year
secure 10mo ago
https://github.com/gokrazy/freeze https://github.com/gokrazy/freeze is a minimal take on this
alas44 10mo ago
There is this project "actually portable executable"/cosmopolitan libc https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan that allows a compile once execute anywhere style type of C++ binary
lizknope 10mo ago
15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.
- mkoubaa 10mo ago
  Sounds painful. Better to distrib a separate bundle per platform and use RPATH
- ValdikSS 10mo ago
  It you know what you're doing and scrupulous enough, you can package the software in a way that it works 25 years later. https://xcancel.com/ValdikSS/status/1843044963443253678 https://xcancel.com/ValdikSS/status/1843044963443253678
ryan-c 10mo ago
(not an endorsement, I do not use it, but I know of it) https://www.magicermine.com/ https://www.magicermine.com/
saghm 10mo ago
I don't think it's as simple as "run this one thing to package it", so if the process rather than the format is what you're looking for, this won't work, but that sounds a lot like how AppImages work from the user perspective. My understanding is that an AppImage is basically a static binary paired with a small filesystem image containing the "root" for the application (including the expected libraries under /usr/lib or wherever they belong). I don't line everything about the format, but overall it feels a lot less prescriptive than other "universal" packages like flatpak or snap, and the fact that you can easily extract it and pick out the pieces you want to repackage without needing any external tools (there are built-in flags on the binary like --appimage-extract) in helps a lot.
jcalvinowens 10mo ago
mkdir chroot cd chroot for lib in $(ldd ${executable} | grep -oE '/\S+'); do tgt="$(dirname ${lib})" mkdir -p .${tgt} cp ${lib} .${tgt} done mkdir -p .$(dirname ${executable}) cp ${executable} .${executable} tar cf ../chroot-run-anywhere.tgz .
- saidinesh5 10mo ago
  You're supposed to do this recursively for all the libs no? Eg. Your App might just depend on libqt5gui.so but that libqt5gui.so might depend on some libxml etc... Not to mention all the files from /usr/share etc... That your application might indirectly depend on.
  - jcalvinowens 10mo ago
    > You're supposed to do this recursively ldd works recursively. > Not to mention all the files from /usr/share Well yeah, there obviously cannot be a generic way to enumerate all the files a program might open...
Conscat 10mo ago
Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64. https://github.com/mgord9518/shappimage https://github.com/mgord9518/shappimage
- sekh60 10mo ago
  So autotools but for execution instead of compilation?
fdgdd 10mo ago
Exodus (https://github.com/intoli/exodus https://github.com/intoli/exodus) used to be good for this but is giving me a Python error these days.
- MrDrMcCoy 10mo ago
  A couple similar projects: https://github.com/sigurd-dev/mkblob https://github.com/sigurd-dev/mkblob https://github.com/tweag/clodl https://github.com/tweag/clodl

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

by amelius 10mo ago

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

mdavid626 10mo ago

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.

fc417fc802 10mo ago
Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.
- mdavid626 10mo ago
  Yes, that's true. But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. These are just strange and confusing from the end users' perspective.
  - toast0 10mo ago
    > But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. Why would you include most of your dynamic libraries but not your libc? You could still run into problems if you (or your libraries) want to use syscalls that weren't available on older kernels or whatever.
    - mdavid626 10mo ago
      You can include it, but - either you use chroot, proot or similar to make /lib path contain your executable’s loader - or you hardcode different loader path into your executable Both are difficult for an end user.
      - toast0 10mo ago
        This isn't that hard (that's not to say this is easy, it is tricky). Your executable should be a statically linked stub loader with an awful lot of data, the stub loader dynamically links your real executable (and libraries, including libc) from the data and runs it.
        fc417fc802 10mo ago
        To add to this, in case of any remaining confusion. You can implement your own execve in userspace. [0] But the kernel's execve is a piece of machinery that invokes the loader so obviously it follows that you're free to make any changes you'd like to the overall process. Bonus points if you add compression or encryption and manage to trip a virus scanner or three. [1] [0] https://grugq.github.io/docs/ul_exec.txt https://grugq.github.io/docs/ul_exec.txt [1] https://blackhat.com/presentations/bh-usa-07/Yason/Whitepaper/bh-usa-07-yason-WP.pdf#page=17 https://blackhat.com/presentations/bh-usa-07/Yason/Whitepape...
- X-Ryl669 10mo ago
  No you can't. dlopen signature takes a file path, not a memory range. And if you start to save the libraries to the filesystem before opening them, there's no difference to shipping an archive directly and skip the trouble of your own archive code.
geocar 10mo ago
> But you can't take .so files and make one "static" binary out of them. Yes you can! This is more-or-less what unexec does - https://news.ycombinator.com/item?id=21394916 https://news.ycombinator.com/item?id=21394916 For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds. But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in one big file ahead of time! [1]: ASLR would be one of those things...
- mdavid626 10mo ago
  What if the library you use calls dlopen later? That’ll fail. There is no universal, working way to do it. Only some hacks which work in some special cases.
  - geocar 10mo ago
    > What if the library you use calls dlopen later? That’ll fail. Nonsense. xemacs could absolutely call dlopen. > There is no universal, working way to do it. Only some hacks which work in some special cases. So you say, but I remember not too long ago you weren't even aware it was possible, and you clearly didn't check one of the most prominent users of this technique, so maybe you should also explain why I or anyone else should give a fuck about what you think is a "hack"?

marksugar 10mo ago

[dead]

TheDong 10mo ago

There are things like this. The things I know of and can think of off the top of my head are: 1. appimage https://appimage.org/ https://appimage.org/ 2. nix-bundle https://github.com/nix-community/nix-bundle https://github.com/nix-community/nix-bundle 3. guix via guix pack 4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc https://github.com/NilsIrl/dockerc ) 5. A docker image (a package that runs everywhere, assuming a docker runtime is available) 6. https://flatpak.org/ https://flatpak.org/ 7. https://en.wikipedia.org/wiki/Snap_(software) https://en.wikipedia.org/wiki/Snap_(software) AppImage is the closest to what you want I think.

amelius 10mo ago
AppImage looks like what I need, thanks. I wonder though, if I package say a .so file from nVidia, is that allowed by the license?
- c0balt 10mo ago
  Depends on the license and the specific piece of software. Redistribution of commercial software is may be restricted or require explicit approval. You generally still also have to abide by license obligations for OSS too, e. G., GPL. To be specific for the exampls, Nvidia has historically been quite restrictive (only on approval) here. Firmware has only recently been opened up a bit and drivers continue to be an issue iirc.
- mdavid626 10mo ago
  Don't forget - AppImage won't work if you package something with glibc, but run on musl/uclibc.
- direwolf20 10mo ago
  No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.
  - amelius 10mo ago

aa-jv 10mo ago

AppImage comes close to fulfilling this need: https://appimage.github.io/appimagetool/ https://appimage.github.io/appimagetool/ Myself, I've committed to using Lua for all my cross-platform development needs, and in that regard I find luastatic very, very useful ..

formerly_proven 10mo ago

I don't think you can link shared objects into a static binary because you'd have to patch all instances where the code reads the PLT/GOT, but this can be arbitrarily mangled by the optimizer, and turn them back into relocations for the linker to then resolve them. You can change the rpath though, which is sort of like an LD_LIBRARY_PATH baked into the object, which makes it relatively easy to bundle everything but libc with your binary. edit: Mild correction, there is this: https://sourceforge.net/projects/statifier/ https://sourceforge.net/projects/statifier/ But the way this works is that it has the dynamic linker load everything (without ASLR / in a compact layout, presumably) and then dumps an image of the process. Everything else is just increasingly fancy ways of copying shared objects around and making ld.so prefer the bundled libraries.

deleted 10mo ago
[deleted]

fieu 10mo ago

Ermine: https://www.magicermine.com/ https://www.magicermine.com/ It works surprisingly well but their pricing is hidden and last time I contacted them as a student it was upwards of $350/year

secure 10mo ago

https://github.com/gokrazy/freeze https://github.com/gokrazy/freeze is a minimal take on this

alas44 10mo ago

There is this project "actually portable executable"/cosmopolitan libc https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan that allows a compile once execute anywhere style type of C++ binary

lizknope 10mo ago

15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.

mkoubaa 10mo ago
Sounds painful. Better to distrib a separate bundle per platform and use RPATH
ValdikSS 10mo ago
It you know what you're doing and scrupulous enough, you can package the software in a way that it works 25 years later. https://xcancel.com/ValdikSS/status/1843044963443253678 https://xcancel.com/ValdikSS/status/1843044963443253678

ryan-c 10mo ago

(not an endorsement, I do not use it, but I know of it) https://www.magicermine.com/ https://www.magicermine.com/

saghm 10mo ago

I don't think it's as simple as "run this one thing to package it", so if the process rather than the format is what you're looking for, this won't work, but that sounds a lot like how AppImages work from the user perspective. My understanding is that an AppImage is basically a static binary paired with a small filesystem image containing the "root" for the application (including the expected libraries under /usr/lib or wherever they belong). I don't line everything about the format, but overall it feels a lot less prescriptive than other "universal" packages like flatpak or snap, and the fact that you can easily extract it and pick out the pieces you want to repackage without needing any external tools (there are built-in flags on the binary like --appimage-extract) in helps a lot.

jcalvinowens 10mo ago

mkdir chroot cd chroot for lib in $(ldd ${executable} | grep -oE '/\S+'); do tgt="$(dirname ${lib})" mkdir -p .${tgt} cp ${lib} .${tgt} done mkdir -p .$(dirname ${executable}) cp ${executable} .${executable} tar cf ../chroot-run-anywhere.tgz .

saidinesh5 10mo ago
You're supposed to do this recursively for all the libs no? Eg. Your App might just depend on libqt5gui.so but that libqt5gui.so might depend on some libxml etc... Not to mention all the files from /usr/share etc... That your application might indirectly depend on.
- jcalvinowens 10mo ago
  > You're supposed to do this recursively ldd works recursively. > Not to mention all the files from /usr/share Well yeah, there obviously cannot be a generic way to enumerate all the files a program might open...

Conscat 10mo ago

Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64. https://github.com/mgord9518/shappimage https://github.com/mgord9518/shappimage

sekh60 10mo ago
So autotools but for execution instead of compilation?

fdgdd 10mo ago

Exodus (https://github.com/intoli/exodus https://github.com/intoli/exodus) used to be good for this but is giving me a Python error these days.

MrDrMcCoy 10mo ago
A couple similar projects: https://github.com/sigurd-dev/mkblob https://github.com/sigurd-dev/mkblob https://github.com/tweag/clodl https://github.com/tweag/clodl