6 ms·

FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ http://icculus.org/fatelf/ fail to persuade me to want to pay any ad

by holdenk 17y ago

FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ http://icculus.org/fatelf/ fail to persuade me to want to pay any additional cost.

bad_user 17y ago
Oh, so having future commercial packages that many people depend on, like Adobe Photoshop or 3D Studio Max ... packaged and distributed as a universal binary for every major distribution and every architecture ... is of no importance whatsoever? This is a major roadblock for commercial third-parties that would like to distribute ports of their software for Linux.
- prodigal_erik 17y ago
  If a proprietary vendor is willing to do all the work to cross-compile, package, and test on a variety of architectures and distributions, I can't believe they would be deterred by the need to link a binary for each package. With FatELF they would have to do that anyway, and then glue all those binaries together before building all their packages.
  - pohl 17y ago
    Yes, and NeXT could have asked its ISVs to do the same thing back when they ran on 040, x86, SPARC, and PA-RISC. The motivation, though, was not just to make things easier for the ISVs, but also to make things easy for the ISVs' customers. Simplifying the process of buying and installing software was the real thrust of the endeavor. [1] There is real economic value in making things easy for customers. Why, for example, would web sites offer 1-Click ordering, when entering a credit card number is just a few more keystrokes? Because simplicity makes money flow, and is good for a market. The Year of the Linux Desktop will never come until these issues become important to the community. Which is to say that it will never happen, because the community, as a whole, has no economic incentive to lower the bar for customers like this. Ryan Gordan probably could have figured that out sooner, but I'm glad he was an optimist for a little while, at least. [1] There was also a secondary benefit for system administrators. You could install a single copy of a 4-way-fat-binary on an NFS share and a workstation running NeXTstep of any architecture could launch it over the network.
    - lil_cain 17y ago
      Except, linux users don't use four platforms on the desktop. They use (nearly entirely) 1. Those use x86-64 are almost entirely clued in enough to grab the correct architecture. And even if they weren't, you could get around this simply by making your installer a shell script, that choose the right dpkg based on the output of uname -m. Hardly overly complex.
      - pohl 17y ago
        I guess that's fine, as long as nobody has ambitions for Linux adoption to grow beyond the relatively small already-clued-in demographic. There's nothing wrong with wanting to stay small. FWIW, I used Debian/PPC at home for years. But, then, I was also among the small set of people who actually ran NeXTSTEP on a "gecko" PA-RISC machine and a SPARC laptop made by Tadpole. That's the story behind my perspective, anyway.
- thingie 17y ago
  You can do this without fatelf. Put everything you have into a single package and add "shell scripts and flakey logic to pick the right binary and libraries to load". (He fails to mention, that fatelf is nothing more than this, only hardwired into kernel.)
  - antonovka 17y ago
    How are shell scripts and flakey logic an improvement on Apple's cross-platform development/usage model? gcc -arch i386 -arch ppc -arch x86_64 -isysroot <SDK path> example.c -o example Done. > file example example: Mach-O universal binary with 3 architectures example (for architecture i386): Mach-O executable i386 example (for architecture ppc7400): Mach-O executable ppc example (for architecture x86_64): Mach-O 64-bit executable x86_64 I can even transparently run PPC binaries on my x86 Mac. Universal binaries are a core component of this easy-to-use (for developers and users alike) architecture independence.
    - vidarh 17y ago
      What you are demonstrating is the userland toolchain. What was suggested above is an improvement by not requiring meddling with the kernel. Nothing prevents a userland toolchain for Linux from supporting what you are describing above without kernel modifications.
      - antonovka 17y ago
        What you are demonstrating is the userland toolchain. No, what I'm demonstrating is an end-to-end architecture-transparent platform, which includes a complete userspace toolchain, a universal binary format, and necessary kernel and dynamic linker support. What was suggested above is an improvement by not requiring meddling with the kernel I'm not sure I understand how "meddling" in the kernel is a bad thing when it provides for user-transparent execution of multi-architecture binaries, including transparent emulation of binaries that lack support for the host architecture. It's not as if it's complex or dangerous to parse the Mach-O or FatELF formats, and if you take a page from Mac OS X or qemu, you can even do transparent emulation in userspace. Nothing prevents a userland toolchain for Linux from supporting what you are describing above without kernel modifications. How is it useful to build an easy-to-use multi-architecture binary if the kernel can't actually execute it? Why are you afraid of doing simple parsing[1] of the binary? The kernel already does this -- how do you think ELF loading and shell script execution works? I can imagine this conversation 30 years ago[2] -- "Why should we add shell script shebang parsing/execution support to the kernel? Why not just glue together loader executables that load the shell script" [1] The dead simple Mach-O universal binary loading code: http://fxr.watson.org/fxr/source/bsd/kern/kern_exec.c?v=xnu-1228#L563 http://fxr.watson.org/fxr/source/bsd/kern/kern_exec.c?v=xnu-... [2] The introduction of shebang in UNIX: http://www.in-ulm.de/~mascheck/various/shebang/sys1.c.html http://www.in-ulm.de/~mascheck/various/shebang/sys1.c.html
- Tuna-Fish 17y ago
  It would be of plenty of importance, but fat_elf does not actually help solve the problem in any way. You can produce all important functionality of the fat_elf patches with a single simple shell script.
  - pohl 17y ago
    This may be the achilles heel of the freenix world: since almost anything can be worked-around with some scripting, people actively argue against putting facilities like this at the right place in the system's architecture. Add 1 to the Cathedral's score.
    - Tuna-Fish 17y ago
      ... Except why exactly is kernel and libc the right place to put this in? Remember that stuff in kernel works outside memory protection. Moving complex stuff away from the kernel is a win pretty much always when there are no pressing performance reasons to do otherwise. This code will only run once every time a program is started, so performance is certainly not a good reason to put it in kernel. What I really want to reiterate here is that using fat_elf:s would in absolutely no way make shipping stuff to multiple different linux platforms easier. The reason it is hard is because when someone says linux, pretty much the only guarantee you have about the system is that it runs a linux kernel, and even that can potentially be so old or be so strange that you can't trust anything about it. All fat_elf gives us is stuffing multiple binaries in a single file, and we can do this already without much fuss. It does not in any way give us true "Universal Binaries", because to do so, we would have to either agree on a common subset of libraries a linux system should always ship (and agree on indefinite binary compability for those libraries), or ship a meaningful portion of the entire platform in every binary. If you want what fat_elf gives, you can get it with a 20-line shell script you concatenate into a bunch of binaries, and it could be argued that it is the cleaner and better place to put it in, considering it's only an ugly hack anyway. It's just that people look at fat_elf and see something that isn't there.
      - pohl 17y ago
        The architectural location that I was alluding to is some fat binary format itself, because this has the greatest impact on customer and user experience. How you parcel out the supporting code between kernel and userland would flow from that invariant...if I were Linus for a day. Or, perhaps even better, adopt a small, architecture-neutral IR like LLVM-BC and a workstation of any architecture could choose to compile it just-in-time at launch, or at install time. Either way, the greatest tragedy here is small thinking. I would love to see Linux rise to the level where users don't need to know what an instruction-set architecture is.
- lil_cain 17y ago
  It doesn't necessarily solve the 'every distribution' thing. And most commercial developers don't support multiple architecture s in the first place.
oddgodd 17y ago
It really is just a bad idea. Or at least one that is working against ideas central to the way Linux is currently used. Universal/fat binaries made sense on the Macintosh because there is no concept of program installation on these systems. While I think that eschewing installation is generally a better design, one drawback is that if you want to be able to support multiple architectures in one application you have to do the architecture check when the program is loaded. Central to Linux and Windows is the idea of program installation, either through packages or installer programs. No one is interested in making it so that you can drag and drop items in Program Files or /usr/bin between systems and expect them to run, which is the only thing that using fat binaries really gets you over other solutions. Nearly all of the commercial binary-only software I have seen in Linux (and other Unixes) uses an installer program, just like windows. There is no technical reason why such an installer couldn’t determine the architecture to install.
- FooBarWidget 17y ago
  Not quite. Current Linux packaging formats encourage the developer to build one package per architecture. This means presenting several download choices for the user which can be confusing. The user doesn't always know what his architecture is. The problem can be solved in two ways: 1) Distributing through a repository and have the package manager auto-select the architecture. However this is highly distribution-specific. If you want to build a single tar.gz that works on the majority of the Linux distros then you're out of luck. 2) Compile binaries for multiple architectures and bundle everything into the same package, and have a shell script or whatever select the correct binary. While (2) is entirely doable and does not confuse the end user, it does make the developer's job harder. He has to compile binaries multiple times and spend a lot of effort on packaging. Having support for universal binaries throughout the entire system, including the developer toolchain, solves not only confusion for the end user but also hassle for the developer. On OS X I can type "gcc -arch i386 -arch ppc" and it'll generate a universal binary with two architectures. I don't have to spend time setting up a PPC virtual machine with a development environment in it, or to setup a cross compiler, just to compile binaries for PPC; everything is right there. I think the ultimate point is not to make impossible things possible, but to make already possible things easier, for both end users and app developers.

by holdenk 17y ago

FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ http://icculus.org/fatelf/ fail to persuade me to want to pay any additional cost.

bad_user 17y ago

Oh, so having future commercial packages that many people depend on, like Adobe Photoshop or 3D Studio Max ... packaged and distributed as a universal binary for every major distribution and every architecture ... is of no importance whatsoever? This is a major roadblock for commercial third-parties that would like to distribute ports of their software for Linux.

prodigal_erik 17y ago
If a proprietary vendor is willing to do all the work to cross-compile, package, and test on a variety of architectures and distributions, I can't believe they would be deterred by the need to link a binary for each package. With FatELF they would have to do that anyway, and then glue all those binaries together before building all their packages.
- pohl 17y ago
  Yes, and NeXT could have asked its ISVs to do the same thing back when they ran on 040, x86, SPARC, and PA-RISC. The motivation, though, was not just to make things easier for the ISVs, but also to make things easy for the ISVs' customers. Simplifying the process of buying and installing software was the real thrust of the endeavor. [1] There is real economic value in making things easy for customers. Why, for example, would web sites offer 1-Click ordering, when entering a credit card number is just a few more keystrokes? Because simplicity makes money flow, and is good for a market. The Year of the Linux Desktop will never come until these issues become important to the community. Which is to say that it will never happen, because the community, as a whole, has no economic incentive to lower the bar for customers like this. Ryan Gordan probably could have figured that out sooner, but I'm glad he was an optimist for a little while, at least. [1] There was also a secondary benefit for system administrators. You could install a single copy of a 4-way-fat-binary on an NFS share and a workstation running NeXTstep of any architecture could launch it over the network.
  - lil_cain 17y ago
    Except, linux users don't use four platforms on the desktop. They use (nearly entirely) 1. Those use x86-64 are almost entirely clued in enough to grab the correct architecture. And even if they weren't, you could get around this simply by making your installer a shell script, that choose the right dpkg based on the output of uname -m. Hardly overly complex.
    - pohl 17y ago
      I guess that's fine, as long as nobody has ambitions for Linux adoption to grow beyond the relatively small already-clued-in demographic. There's nothing wrong with wanting to stay small. FWIW, I used Debian/PPC at home for years. But, then, I was also among the small set of people who actually ran NeXTSTEP on a "gecko" PA-RISC machine and a SPARC laptop made by Tadpole. That's the story behind my perspective, anyway.
thingie 17y ago
You can do this without fatelf. Put everything you have into a single package and add "shell scripts and flakey logic to pick the right binary and libraries to load". (He fails to mention, that fatelf is nothing more than this, only hardwired into kernel.)
- antonovka 17y ago
  How are shell scripts and flakey logic an improvement on Apple's cross-platform development/usage model? gcc -arch i386 -arch ppc -arch x86_64 -isysroot <SDK path> example.c -o example Done. > file example example: Mach-O universal binary with 3 architectures example (for architecture i386): Mach-O executable i386 example (for architecture ppc7400): Mach-O executable ppc example (for architecture x86_64): Mach-O 64-bit executable x86_64 I can even transparently run PPC binaries on my x86 Mac. Universal binaries are a core component of this easy-to-use (for developers and users alike) architecture independence.
  - vidarh 17y ago
    What you are demonstrating is the userland toolchain. What was suggested above is an improvement by not requiring meddling with the kernel. Nothing prevents a userland toolchain for Linux from supporting what you are describing above without kernel modifications.
    - antonovka 17y ago
      What you are demonstrating is the userland toolchain. No, what I'm demonstrating is an end-to-end architecture-transparent platform, which includes a complete userspace toolchain, a universal binary format, and necessary kernel and dynamic linker support. What was suggested above is an improvement by not requiring meddling with the kernel I'm not sure I understand how "meddling" in the kernel is a bad thing when it provides for user-transparent execution of multi-architecture binaries, including transparent emulation of binaries that lack support for the host architecture. It's not as if it's complex or dangerous to parse the Mach-O or FatELF formats, and if you take a page from Mac OS X or qemu, you can even do transparent emulation in userspace. Nothing prevents a userland toolchain for Linux from supporting what you are describing above without kernel modifications. How is it useful to build an easy-to-use multi-architecture binary if the kernel can't actually execute it? Why are you afraid of doing simple parsing[1] of the binary? The kernel already does this -- how do you think ELF loading and shell script execution works? I can imagine this conversation 30 years ago[2] -- "Why should we add shell script shebang parsing/execution support to the kernel? Why not just glue together loader executables that load the shell script" [1] The dead simple Mach-O universal binary loading code: http://fxr.watson.org/fxr/source/bsd/kern/kern_exec.c?v=xnu-1228#L563 http://fxr.watson.org/fxr/source/bsd/kern/kern_exec.c?v=xnu-... [2] The introduction of shebang in UNIX: http://www.in-ulm.de/~mascheck/various/shebang/sys1.c.html http://www.in-ulm.de/~mascheck/various/shebang/sys1.c.html
Tuna-Fish 17y ago
It would be of plenty of importance, but fat_elf does not actually help solve the problem in any way. You can produce all important functionality of the fat_elf patches with a single simple shell script.
- pohl 17y ago
  This may be the achilles heel of the freenix world: since almost anything can be worked-around with some scripting, people actively argue against putting facilities like this at the right place in the system's architecture. Add 1 to the Cathedral's score.
  - Tuna-Fish 17y ago
    ... Except why exactly is kernel and libc the right place to put this in? Remember that stuff in kernel works outside memory protection. Moving complex stuff away from the kernel is a win pretty much always when there are no pressing performance reasons to do otherwise. This code will only run once every time a program is started, so performance is certainly not a good reason to put it in kernel. What I really want to reiterate here is that using fat_elf:s would in absolutely no way make shipping stuff to multiple different linux platforms easier. The reason it is hard is because when someone says linux, pretty much the only guarantee you have about the system is that it runs a linux kernel, and even that can potentially be so old or be so strange that you can't trust anything about it. All fat_elf gives us is stuffing multiple binaries in a single file, and we can do this already without much fuss. It does not in any way give us true "Universal Binaries", because to do so, we would have to either agree on a common subset of libraries a linux system should always ship (and agree on indefinite binary compability for those libraries), or ship a meaningful portion of the entire platform in every binary. If you want what fat_elf gives, you can get it with a 20-line shell script you concatenate into a bunch of binaries, and it could be argued that it is the cleaner and better place to put it in, considering it's only an ugly hack anyway. It's just that people look at fat_elf and see something that isn't there.
    - pohl 17y ago
      The architectural location that I was alluding to is some fat binary format itself, because this has the greatest impact on customer and user experience. How you parcel out the supporting code between kernel and userland would flow from that invariant...if I were Linus for a day. Or, perhaps even better, adopt a small, architecture-neutral IR like LLVM-BC and a workstation of any architecture could choose to compile it just-in-time at launch, or at install time. Either way, the greatest tragedy here is small thinking. I would love to see Linux rise to the level where users don't need to know what an instruction-set architecture is.
lil_cain 17y ago
It doesn't necessarily solve the 'every distribution' thing. And most commercial developers don't support multiple architecture s in the first place.

oddgodd 17y ago

It really is just a bad idea. Or at least one that is working against ideas central to the way Linux is currently used. Universal/fat binaries made sense on the Macintosh because there is no concept of program installation on these systems. While I think that eschewing installation is generally a better design, one drawback is that if you want to be able to support multiple architectures in one application you have to do the architecture check when the program is loaded. Central to Linux and Windows is the idea of program installation, either through packages or installer programs. No one is interested in making it so that you can drag and drop items in Program Files or /usr/bin between systems and expect them to run, which is the only thing that using fat binaries really gets you over other solutions. Nearly all of the commercial binary-only software I have seen in Linux (and other Unixes) uses an installer program, just like windows. There is no technical reason why such an installer couldn’t determine the architecture to install.

FooBarWidget 17y ago
Not quite. Current Linux packaging formats encourage the developer to build one package per architecture. This means presenting several download choices for the user which can be confusing. The user doesn't always know what his architecture is. The problem can be solved in two ways: 1) Distributing through a repository and have the package manager auto-select the architecture. However this is highly distribution-specific. If you want to build a single tar.gz that works on the majority of the Linux distros then you're out of luck. 2) Compile binaries for multiple architectures and bundle everything into the same package, and have a shell script or whatever select the correct binary. While (2) is entirely doable and does not confuse the end user, it does make the developer's job harder. He has to compile binaries multiple times and spend a lot of effort on packaging. Having support for universal binaries throughout the entire system, including the developer toolchain, solves not only confusion for the end user but also hassle for the developer. On OS X I can type "gcc -arch i386 -arch ppc" and it'll generate a universal binary with two architectures. I don't have to spend time setting up a PPC virtual machine with a development environment in it, or to setup a cross compiler, just to compile binaries for PPC; everything is right there. I think the ultimate point is not to make impossible things possible, but to make already possible things easier, for both end users and app developers.