8 ms·

FatELF is no more. Linux developer community owned Ryan Gordon.

59 points by jeff18 17y ago

chanux 17y ago
I don't believe this is the way it should end, leaving one enthusiast all broken. Anyway I'm not really aware about background details of the matter.
- mahmud 17y ago
  Enthusiasm is not a substitute for pragmatics; all the other contributors are just as enthusiastic, but they know their patches are not entitled to merger with the main tree willy nilly, without there being a very good reason.
- orph 17y ago
  This sort of soul-crush usually means more quality employee fodder for Google & Facebook.
- ableal 17y ago
  This reminds me of P.G.Wodehouse's stories, where you'll occasionally find a victim being dragooned into judging a 'Bonny Baby' contest.
spitfire 17y ago
This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out. Open source can foster innovation, as long as ego's don't get in the way. Sadly there are a lot of ego's involved in open source. NB: I think the idea of fatelf is a very good one. Mainly for binary applications though. Which OSS guys tend to dislike.
- tvon 17y ago
  > Mainly for binary applications though. Which OSS guys tend to dislike. Huh?
  - dlsspy 17y ago
    Where binary == closed source. Imagine a binary-only distribution of some application that runs on all hardware platforms (because we get that a lot in OS X and we used to get it on NeXTSTEP as well).
    - nailer 17y ago
      The parent's correct though - 'binary' does not mean 'binary only. They have very distinct meanings, and reading the sentence as written is confusing.
      - tvon 17y ago
        Indeed, I took it to mean that OSS guys only build from source, or something along those lines. Proprietary software uses binary formats, yes, but interchanging 'proprietary' with 'binary' is a bit confusing.
- thingie 17y ago
  He could write the patch, he could try it and persuade mainline kernel (and glibc and gcc, and...) developers to include it. If the feature is really that useful, he can even maintain it separately, or some Linux distribution can. All of which would be impossible with closed software. Innovation in the oss way, sadly, means, that there are lots of ideas, and most of them thrown out. Hostile reaction from kernel developers? Well, maybe it's little sad, they could trash the patch more politely or something, but eventually they have to. Fatelf adds stuff to kernel, glibc and few other parts of system without really solving much. He could try something with binfmt_misc and completely userspace, not add junk into kernel and glibc itself. (With junk I mean something unnecessary that can be done in userspace. (or not at all))
  - cschwarm 17y ago
    > without really solving much On a German Linux news site (prolinux.de), a commercial Linux game developer -- Ankh2, IIRC -- once commented that they could not get their game running on about 2-3% of their customers' Linux computers, despite their best efforts. I dunno whether FatElf would have solved problems like this in the long run, but there are problems. You may not be affected but this doesn't mean others aren't.
    - thingie 17y ago
      No, I don't think it will solve this kind of problem. It can do the only thing: pack two different binaries into one file, which is also binary, and the kernel has some extra logic to choose, so it is 'transparent'. While this makes some thing easier (if you really don't know whether you want to download that strange 'i386' file or even stranger 'x86_64' file, you download one larger 'fatelf' containing both), it doesn't solve the real problems. Distributions spent quite a lot of effort to make both 32 and 64-bit binaries run on a single system. This is just a hack, nothing more. And it can make the mess even worse. Do I have x86_64 binary in this? Is there a correct elf packed somewhere? How do I know.
      - catzaa 17y ago
        > it doesn't solve the real problems. Well, it would sure make running software on a mixed cluster a lot easier. But in any case - the proliferation of package "managers" is a bigger problem. I really hope a few Linux distributions die so that we can at least have standardization in an evolutionary manner.
- deleted 17y ago
  [deleted]
- davidw 17y ago
  I think it has more to do with one open source project in particular, the kernel, which has a lot of history. The one time I tried to get a patch in, it was very difficult and I ultimately let it be, despite people regularly saying they thought it was a sensible idea. Indeed, there was never any real technical opposition, just a sort of inertia to ignore stuff.
- jdminhbg 17y ago
  I don't think this is specific to open source; anyone who's worked at a closed-source company can certainly tell the same story about stuff being quashed by egos higher-up. At least in the open source community, if you feel strongly enough, you can fork it and go off without the egos.
- shpxnvz 17y ago
  This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out. I may be missing something, but it seems to be fostering innovation in this example as well. He has managed to create something that at least a few people consider innovative, thanks in part to the open licenses of the sources this work is built on, and he is free to distribute it however he likes. Or are you arguing that it's not innovation unless some other group can be forced to take up work on your behalf, by merging and/or maintaining code they don't want or agree with?
wmf 17y ago
Look on the bright side: they saved him a lot of time. Usually the Linux developers tell you to redesign and rewrite your patch from scratch. :-/
holdenk 17y ago
FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ http://icculus.org/fatelf/ fail to persuade me to want to pay any additional cost.
- bad_user 17y ago
  Oh, so having future commercial packages that many people depend on, like Adobe Photoshop or 3D Studio Max ... packaged and distributed as a universal binary for every major distribution and every architecture ... is of no importance whatsoever? This is a major roadblock for commercial third-parties that would like to distribute ports of their software for Linux.
  - prodigal_erik 17y ago
    If a proprietary vendor is willing to do all the work to cross-compile, package, and test on a variety of architectures and distributions, I can't believe they would be deterred by the need to link a binary for each package. With FatELF they would have to do that anyway, and then glue all those binaries together before building all their packages.
    - pohl 17y ago
      Yes, and NeXT could have asked its ISVs to do the same thing back when they ran on 040, x86, SPARC, and PA-RISC. The motivation, though, was not just to make things easier for the ISVs, but also to make things easy for the ISVs' customers. Simplifying the process of buying and installing software was the real thrust of the endeavor. [1] There is real economic value in making things easy for customers. Why, for example, would web sites offer 1-Click ordering, when entering a credit card number is just a few more keystrokes? Because simplicity makes money flow, and is good for a market. The Year of the Linux Desktop will never come until these issues become important to the community. Which is to say that it will never happen, because the community, as a whole, has no economic incentive to lower the bar for customers like this. Ryan Gordan probably could have figured that out sooner, but I'm glad he was an optimist for a little while, at least. [1] There was also a secondary benefit for system administrators. You could install a single copy of a 4-way-fat-binary on an NFS share and a workstation running NeXTstep of any architecture could launch it over the network.
jrockway 17y ago
I don't see what's stopping him from forking Linux and glibc. If this is really better, everyone will just use his fork. (It worked for my fork of cperl-mode, anyway...)
- FooBarWidget 17y ago
  The point is to get patches accepted without forking. Linux and glibc aren't exactly small. Do you honestly see a single man maintaining millions of lines of forked code as well as handling packaging and stuff?
  - tamas 17y ago
    It's not as bad as it sounds, basically great many of the http://git.kernel.org/ http://git.kernel.org/ repos are actual forks of the kernel, regularly pulling changes from vanilla, resolving the occasional conflicts and maintaining/developing their own patches.
    - FooBarWidget 17y ago
      Actually it is as bad as it sounds. Although git makes it easier to merge with mainline, in the end it still requires an active maintainer who manually merges and tests stuff once in a while. Maintaining a fork requires one's constant attention and diverts one from doing other - maybe more useful - things.
  - etherealG 17y ago
    don't deny the fork. the fork is good. use the source luke.
    - gjm11 17y ago
      "There is no fork."
      - rjurney 17y ago
        The fork is just bacteria all up in your hard drive.
coliveira 17y ago
I think this guy is trying to solve a problem that doesn't exist in Linux. In Linux, every piece of software is compile for the architecture you want, so there is no need for a fat binary. Mac OS X needed this because they Mac users mostly run commercial software, so it is better to have a single binary for all platforms.
djcapelis 17y ago
As someone who has actually built binaries to run across multiple distros, this doesn't address even half the issues. Very few people are concerned about binaries that run on more than one architecture when distributing binaries that run on more than one distro is a harder problem. This solution doesn't solve the hard problems, but solves an easy one in an uninspired fashion. Rejecting these patches wasn't a case of holding back innovation, but merely holding back solutions that more experienced developers felt were not appropriate for the platform. There was a good reason for that and playing armchair critic after reading a sympathetic story from a developer who is understandably hurt after getting his feature rejected just doesn't help anyone. The kernel is more well managed than most people give it credit for, and it's exactly because of this that hacky incremental solutions get rejected, even when they might legitimately produce a small benefit. Eventually the overall solution to the problem FatElf is trying to solve is going to have to be something different. I feel sorry for the developer, no one likes getting a feature rejected. But it was the wrong solution and the technical merits must trump other concerns.
- antonovka 17y ago
  But it was the wrong solution and the technical merits must trump other concerns. Your post is very long, but I'm curious to hear why universal binaries are the wrong solution? On the Mac, it's the cornerstone of their transparent multi-architecture support. Apple (and previously NeXT) inarguably demonstrated its value and ease-of-use over the past 20 years. Producing x86-32, x86-64, armv6, armv7, and PPC binaries all from the same source code -- and having them work the first time -- is incredibly simple on Mac OS X. NeXTSTEP (the OS) ran on 68k, x86, PA-RISC, and Sparc processors, and it was equally easy to build software that ran on those machines.
  - djcapelis 17y ago
    OS X has a unified set of libraries to work with, Linux does not. This means the problem is much more complex. Creating a binary that simply runs on different distributions all on the same architecture is a fundamentally hard problem. Simply stacking together binaries for multiple architectures together doesn't result in a universal binary. At worst you have to stack together different binaries for different combinations of libraries and different architectures and if you assume even just the latest versions only of the top 5 distributions running on the top 5 supported architectures for Linux you may well end up with 25 different binaries that FatELF would glue together resulting in every program being a cool 2500% larger. This is unacceptable in terms of space and unacceptable in that it still isn't universal after all that as not all arches are supported and you only get the latest versions of the popular distros. This is why I claim FatELF does not solve the hard problems with universal binaries. The only thing it might solve is someone could distribute a binary for a particular distro that could run on multiple architectures for that distro, but they can already do this with shell scripts and this is an uncommon case and most people would just rather download and store the one appropriate binary. (This last is easy with a functional package management system. You should keep in mind that OS X does not have a functional complete package management scheme that allows users to discover, acquire and manage 3rd party packages. Not all Linux distros do either, but those that do are less in need of universal binaries. This is really a separate discussion though that gets to the heart of how software ought to be distributed. There are many blind spots in OS X's software strategy some of which Sparkle is making a half-baked (not sparkle's fault) attempt to fix.)[1] At the end of the day you can't simply assume that the solution for OS X is the right one for Linux and you'd make a much bigger mistake assuming that the developers who rejected these patches weren't both entirely aware of OS X's excellent use of this technique and were doing this for anything but sound technical reasons. (I.E. It's not just NIH, they would happily adopt this if this was a reasonable solution to Linux's unique[2] challenges with binary compatibility across "Linux") [1] There are many things wrong with this and most package management systems do not incorporate third party repos/packages as well as they should, which IMO is the larger problem here. Not that corporations should have to manage repos for every distro, but that becomes a separate mess. [2] Perhaps read as special. Aside #432: As you noticed my replies can often be anything but concise. I apologize if that makes things hard to follow.

59 points by jeff18 17y ago

I don't believe this is the way it should end, leaving one enthusiast all broken. Anyway I'm not really aware about background details of the matter.

mahmud 17y ago
Enthusiasm is not a substitute for pragmatics; all the other contributors are just as enthusiastic, but they know their patches are not entitled to merger with the main tree willy nilly, without there being a very good reason.
orph 17y ago
This sort of soul-crush usually means more quality employee fodder for Google & Facebook.
ableal 17y ago
This reminds me of P.G.Wodehouse's stories, where you'll occasionally find a victim being dragooned into judging a 'Bonny Baby' contest.

spitfire 17y ago

This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out. Open source can foster innovation, as long as ego's don't get in the way. Sadly there are a lot of ego's involved in open source. NB: I think the idea of fatelf is a very good one. Mainly for binary applications though. Which OSS guys tend to dislike.

tvon 17y ago
> Mainly for binary applications though. Which OSS guys tend to dislike. Huh?
- dlsspy 17y ago
  Where binary == closed source. Imagine a binary-only distribution of some application that runs on all hardware platforms (because we get that a lot in OS X and we used to get it on NeXTSTEP as well).
  - nailer 17y ago
    The parent's correct though - 'binary' does not mean 'binary only. They have very distinct meanings, and reading the sentence as written is confusing.
    - tvon 17y ago
      Indeed, I took it to mean that OSS guys only build from source, or something along those lines. Proprietary software uses binary formats, yes, but interchanging 'proprietary' with 'binary' is a bit confusing.
thingie 17y ago
He could write the patch, he could try it and persuade mainline kernel (and glibc and gcc, and...) developers to include it. If the feature is really that useful, he can even maintain it separately, or some Linux distribution can. All of which would be impossible with closed software. Innovation in the oss way, sadly, means, that there are lots of ideas, and most of them thrown out. Hostile reaction from kernel developers? Well, maybe it's little sad, they could trash the patch more politely or something, but eventually they have to. Fatelf adds stuff to kernel, glibc and few other parts of system without really solving much. He could try something with binfmt_misc and completely userspace, not add junk into kernel and glibc itself. (With junk I mean something unnecessary that can be done in userspace. (or not at all))
- cschwarm 17y ago
  > without really solving much On a German Linux news site (prolinux.de), a commercial Linux game developer -- Ankh2, IIRC -- once commented that they could not get their game running on about 2-3% of their customers' Linux computers, despite their best efforts. I dunno whether FatElf would have solved problems like this in the long run, but there are problems. You may not be affected but this doesn't mean others aren't.
  - thingie 17y ago
    No, I don't think it will solve this kind of problem. It can do the only thing: pack two different binaries into one file, which is also binary, and the kernel has some extra logic to choose, so it is 'transparent'. While this makes some thing easier (if you really don't know whether you want to download that strange 'i386' file or even stranger 'x86_64' file, you download one larger 'fatelf' containing both), it doesn't solve the real problems. Distributions spent quite a lot of effort to make both 32 and 64-bit binaries run on a single system. This is just a hack, nothing more. And it can make the mess even worse. Do I have x86_64 binary in this? Is there a correct elf packed somewhere? How do I know.
    - catzaa 17y ago
      > it doesn't solve the real problems. Well, it would sure make running software on a mixed cluster a lot easier. But in any case - the proliferation of package "managers" is a bigger problem. I really hope a few Linux distributions die so that we can at least have standardization in an evolutionary manner.
deleted 17y ago
[deleted]
davidw 17y ago
I think it has more to do with one open source project in particular, the kernel, which has a lot of history. The one time I tried to get a patch in, it was very difficult and I ultimately let it be, despite people regularly saying they thought it was a sensible idea. Indeed, there was never any real technical opposition, just a sort of inertia to ignore stuff.
jdminhbg 17y ago
I don't think this is specific to open source; anyone who's worked at a closed-source company can certainly tell the same story about stuff being quashed by egos higher-up. At least in the open source community, if you feel strongly enough, you can fork it and go off without the egos.
shpxnvz 17y ago
This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out. I may be missing something, but it seems to be fostering innovation in this example as well. He has managed to create something that at least a few people consider innovative, thanks in part to the open licenses of the sources this work is built on, and he is free to distribute it however he likes. Or are you arguing that it's not innovation unless some other group can be forced to take up work on your behalf, by merging and/or maintaining code they don't want or agree with?

wmf 17y ago

Look on the bright side: they saved him a lot of time. Usually the Linux developers tell you to redesign and rewrite your patch from scratch. :-/

holdenk 17y ago

FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ http://icculus.org/fatelf/ fail to persuade me to want to pay any additional cost.

bad_user 17y ago
Oh, so having future commercial packages that many people depend on, like Adobe Photoshop or 3D Studio Max ... packaged and distributed as a universal binary for every major distribution and every architecture ... is of no importance whatsoever? This is a major roadblock for commercial third-parties that would like to distribute ports of their software for Linux.
- prodigal_erik 17y ago
  If a proprietary vendor is willing to do all the work to cross-compile, package, and test on a variety of architectures and distributions, I can't believe they would be deterred by the need to link a binary for each package. With FatELF they would have to do that anyway, and then glue all those binaries together before building all their packages.
  - pohl 17y ago
    Yes, and NeXT could have asked its ISVs to do the same thing back when they ran on 040, x86, SPARC, and PA-RISC. The motivation, though, was not just to make things easier for the ISVs, but also to make things easy for the ISVs' customers. Simplifying the process of buying and installing software was the real thrust of the endeavor. [1] There is real economic value in making things easy for customers. Why, for example, would web sites offer 1-Click ordering, when entering a credit card number is just a few more keystrokes? Because simplicity makes money flow, and is good for a market. The Year of the Linux Desktop will never come until these issues become important to the community. Which is to say that it will never happen, because the community, as a whole, has no economic incentive to lower the bar for customers like this. Ryan Gordan probably could have figured that out sooner, but I'm glad he was an optimist for a little while, at least. [1] There was also a secondary benefit for system administrators. You could install a single copy of a 4-way-fat-binary on an NFS share and a workstation running NeXTstep of any architecture could launch it over the network.

jrockway 17y ago

I don't see what's stopping him from forking Linux and glibc. If this is really better, everyone will just use his fork. (It worked for my fork of cperl-mode, anyway...)

FooBarWidget 17y ago
The point is to get patches accepted without forking. Linux and glibc aren't exactly small. Do you honestly see a single man maintaining millions of lines of forked code as well as handling packaging and stuff?
- tamas 17y ago
  It's not as bad as it sounds, basically great many of the http://git.kernel.org/ http://git.kernel.org/ repos are actual forks of the kernel, regularly pulling changes from vanilla, resolving the occasional conflicts and maintaining/developing their own patches.
  - FooBarWidget 17y ago
    Actually it is as bad as it sounds. Although git makes it easier to merge with mainline, in the end it still requires an active maintainer who manually merges and tests stuff once in a while. Maintaining a fork requires one's constant attention and diverts one from doing other - maybe more useful - things.
- etherealG 17y ago
  don't deny the fork. the fork is good. use the source luke.
  - gjm11 17y ago
    "There is no fork."
    - rjurney 17y ago
      The fork is just bacteria all up in your hard drive.

coliveira 17y ago

I think this guy is trying to solve a problem that doesn't exist in Linux. In Linux, every piece of software is compile for the architecture you want, so there is no need for a fat binary. Mac OS X needed this because they Mac users mostly run commercial software, so it is better to have a single binary for all platforms.

djcapelis 17y ago

As someone who has actually built binaries to run across multiple distros, this doesn't address even half the issues. Very few people are concerned about binaries that run on more than one architecture when distributing binaries that run on more than one distro is a harder problem. This solution doesn't solve the hard problems, but solves an easy one in an uninspired fashion. Rejecting these patches wasn't a case of holding back innovation, but merely holding back solutions that more experienced developers felt were not appropriate for the platform. There was a good reason for that and playing armchair critic after reading a sympathetic story from a developer who is understandably hurt after getting his feature rejected just doesn't help anyone. The kernel is more well managed than most people give it credit for, and it's exactly because of this that hacky incremental solutions get rejected, even when they might legitimately produce a small benefit. Eventually the overall solution to the problem FatElf is trying to solve is going to have to be something different. I feel sorry for the developer, no one likes getting a feature rejected. But it was the wrong solution and the technical merits must trump other concerns.

antonovka 17y ago
But it was the wrong solution and the technical merits must trump other concerns. Your post is very long, but I'm curious to hear why universal binaries are the wrong solution? On the Mac, it's the cornerstone of their transparent multi-architecture support. Apple (and previously NeXT) inarguably demonstrated its value and ease-of-use over the past 20 years. Producing x86-32, x86-64, armv6, armv7, and PPC binaries all from the same source code -- and having them work the first time -- is incredibly simple on Mac OS X. NeXTSTEP (the OS) ran on 68k, x86, PA-RISC, and Sparc processors, and it was equally easy to build software that ran on those machines.
- djcapelis 17y ago
  OS X has a unified set of libraries to work with, Linux does not. This means the problem is much more complex. Creating a binary that simply runs on different distributions all on the same architecture is a fundamentally hard problem. Simply stacking together binaries for multiple architectures together doesn't result in a universal binary. At worst you have to stack together different binaries for different combinations of libraries and different architectures and if you assume even just the latest versions only of the top 5 distributions running on the top 5 supported architectures for Linux you may well end up with 25 different binaries that FatELF would glue together resulting in every program being a cool 2500% larger. This is unacceptable in terms of space and unacceptable in that it still isn't universal after all that as not all arches are supported and you only get the latest versions of the popular distros. This is why I claim FatELF does not solve the hard problems with universal binaries. The only thing it might solve is someone could distribute a binary for a particular distro that could run on multiple architectures for that distro, but they can already do this with shell scripts and this is an uncommon case and most people would just rather download and store the one appropriate binary. (This last is easy with a functional package management system. You should keep in mind that OS X does not have a functional complete package management scheme that allows users to discover, acquire and manage 3rd party packages. Not all Linux distros do either, but those that do are less in need of universal binaries. This is really a separate discussion though that gets to the heart of how software ought to be distributed. There are many blind spots in OS X's software strategy some of which Sparkle is making a half-baked (not sparkle's fault) attempt to fix.)[1] At the end of the day you can't simply assume that the solution for OS X is the right one for Linux and you'd make a much bigger mistake assuming that the developers who rejected these patches weren't both entirely aware of OS X's excellent use of this technique and were doing this for anything but sound technical reasons. (I.E. It's not just NIH, they would happily adopt this if this was a reasonable solution to Linux's unique[2] challenges with binary compatibility across "Linux") [1] There are many things wrong with this and most package management systems do not incorporate third party repos/packages as well as they should, which IMO is the larger problem here. Not that corporations should have to manage repos for every distro, but that becomes a separate mess. [2] Perhaps read as special. Aside #432: As you noticed my replies can often be anything but concise. I apologize if that makes things hard to follow.