6 ms·

It's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In

It's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In fact, some of them could generate self-contained binaries that allowed you to bundle app assets like icons and images inside the executable - no additional add-ons needed. And they were native executables, not complicated wrappers with layer upon layer of different technologies. This is not to take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before.

weberc2 8y ago
> The programming field often feels like it's in a giant loop constantly re-discovering what went before. It might feel that way, but this is driven by economics. Basically disk, memory, and bandwidth is far cheaper than it used to be so who cares if you waste a GB or so copying the same libraries all over if you don't have to solve for dependency hell?
- Koshkin 8y ago
  > disk is far cheaper Not in the cloud.
  - weberc2 8y ago
    To be clear, you're asserting that the price of disk in the cloud is comparable to what it was decades ago?
    - Koshkin 8y ago
      Things are more complicated than that. (Depending on the pricing scheme, you might end up paying per gigabyte per hour, for example; which may or may not include transfer/storage of actual data; a faster "drive" could be much more expensive than a slower one, etc.)
      - weberc2 8y ago
        Sorry, I'm not following. Granted pricing schemes have changed, but adjusting for that, are you claiming that disk is not dramatically cheaper than it was ~20 years ago? Not trying to be a dick, just not following your point.
        Koshkin 8y ago
        Point is, "disk" in the cloud is not the same thing as the disk in your PC.
        ecnahc515 8y ago
        No, the point is it doesn't matter where the disk is, the overall price has gone down.
        weberc2 8y ago
        Of course it's not, but the cost of both has declined.
  - GordonS 8y ago
    Disagree, cloud storage is cheap - bandwidth is expensive.
  - solarkraft 8y ago
    1) It is 2) Application space on the server is vanishingly tiny compared to the data you're processing, so it is a far more generous trade-off.
- eeZah7Ux 8y ago
  Vendorizing libraries and static linking has become a security hell. Just an example: https://blog.acolyer.org/2017/04/03/a-study-of-security-vulnerabilities-on-docker-hub/ https://blog.acolyer.org/2017/04/03/a-study-of-security-vuln...
  - weberc2 8y ago
    Your link cites "little thought to the inclusion of dependencies", _not_ static linkage, as the cause of this "hell".
    - wahern 8y ago
      The implication is that statically linked binaries make it easier to throw in dependencies because you neither need to worry about how to ship them nor bother with re-using the shared libraries on the system. If there's a bug in Go's TLS implementation, you need to identify, recompile, and redistribute every Go application you use. Whereas if it were using the system's OpenSSL library everything would magically be fixed by a simple `apt-get upgrade` or `yum update`.
      - jjnoakes 8y ago
        Is this always true? If the shared library you linked to uses symbol versions and the bug fix requires a new version of a symbol, the application has to be relinked anyway. Same as static linking, except now it is not obvious you need to relink to get the bug fix.
        rkeene2 8y ago
        Only if the fix involves breaking the old semantics entirely. It's possible in some cases to support old semantics in a safe way at the cost of performance, or in the case that it's not you can disable that call in such a way that the program aborts if it is called, thus indicating that the user should relink that program. This is why integration projects such as Linux distributions are hard, and useful.
      - 8y ago
- shaan7 8y ago
  > so who cares if you waste a GB or so I wonder if that sounds arrogant to just me. As a user I DO CARE. Its frustrating to find applications shipping their own copy of libraries wasting precious disk space. When I was using a Mac, I had at least 4 applications shipping their own copy of Qt5.4 ... aaaarrgh Back on the Linux desktop, package managers handle this beautifully. And for the cases where you need to skip the distro, Flatpak still lets you use system dependencies. Everyone wins. And for whoever claims storage is cheap, please link me the magical place you're buying yours from. Thanks.
  - AnIdiotOnTheNet 8y ago
    Package managers come with their own limitations. You can't have multiple versions of the same application, packages are often out of date, packages sometimes conflict, you can't choose where applications are installed, etc. And storage is ridiculously cheap. You can get an 8TB HDD for $150[0]. Or a 1TB SSD for the same price[1]. [0]https://www.amazon.com/Seagate-Expansion-Desktop-External-STEB8000100/dp/B01HAPGEIE https://www.amazon.com/Seagate-Expansion-Desktop-External-ST... [1]https://www.amazon.com/Blue-NAND-1TB-SSD-WDS100T2B0A/dp/B073SBQMCX https://www.amazon.com/Blue-NAND-1TB-SSD-WDS100T2B0A/dp/B073...
    - rkeene2 8y ago
      > You can't have multiple versions of the same application In many package managers you can have multiple versions of the same application without a big deal -- in most cases you don't want to SUPPORT multiple versions of the same application, so there's always a trade-off.
      - AnIdiotOnTheNet 8y ago
        > In many package managers you can have multiple versions of the same application without a big deal In my experience, only if the distro has specifically set things up to allow that (Python2/3), or you find a PPA maintained by someone. Otherwise, time to compile from source I guess.
IanCal 8y ago
> This is not take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before. I think part of this comes from various tradeoffs becoming more or less important at different times. Processing power goes up so you move to fat clients, then people move to battery powered devices and heavy lifting makes more sense on servers again.
- tomatocracy 8y ago
  It's not really processing power so much as storage and RAM that used to be seen as the disadvantages of static binaries. The advantages of the static binary model are that it avoids "DLL hell" and makes sourcing dependencies a bit better, which are indeed laudable aims. But nowadays it has terrible security (and to some extent reliability) trade-offs - if everything is a static binary and there's a security issue in a library, you need to upgrade every single binary depending on it, which is much less reliable and more prone to error/oversight. As security and reliability have increased in importance, so the "cost" of the static binary route has actually gone up significantly in my view compared to 20-30 years ago. So it's a bit odd to me to see this model (and closely related ideas like flatpak etc) gaining currency. There must be better ways to manage the "DLL hell" issues than that.
  - jungler 8y ago
    From the POV of folks working around massive server farms, all the servers are disposable, so having one binary is convenient, and the short lifespans mean they avoid the downsides. The concern has more relevance to client applications, but then the cost becomes related to update strategy, and if every app is tied to an update manager process(whether it's an app store type of thing or a custom one) then the thinking becomes similar to the cloud server setup: Why bother, I'm going to blow it all away in the next update. When the dependency management is exposed to users, there's a much bigger surface area for troubleshooting and support. It makes sense that devs favor static if they expect the whole environment to be disposable. It isn't like it was in the 90's, where patching was occasional but major new versions were essentially new products.
  - 8y ago
mayank 8y ago
It's understandable due to RAM, disk, and network being way less of a constraint than it used to be (which necessitated dynamic linking and shared libraries), and the modern need to ship around binaries to hundreds or thousands of servers and have them run predictably everywhere.
- v_lisivka 8y ago
  Maybe it better to use DLL and ship to tens of thousands of servers instead.
mLuby 8y ago
That trend suggests that previous efforts are either not discoverable or not sufficiently extensible.
pjmlp 8y ago
Yes, I remember when it was such a big deal that we could finally use dynamic libraries on Amiga OS, MS-DOS protected mode and Windows 3.x. Even GNU/Linux made such a big deal of moving from a.out to ELF binary format, due to easier support for dynamic linking, which quite cumbersome with a.out. Now almost 30 years later it is a big deal to support static linking, go figure.
- artursapek 8y ago
  Technology goes in cycles, like humans do :)
  - pjmlp 8y ago
    Yeah, like renaissance rediscovering Roman and Greek tech.
jcelerier 8y ago
it's not really a loop, but there are really two categories of users : * those who want applications that take the least space on the drive, and share most of their data / libraries across themselves - shared electron runtime, steam runtime shared with the os * those who want applications to conform exactly to what the developper used since the developer can guarantee a "working" configuration Ideally it should be up to the user to choose what configuration he wants, but the problem is that it's up to the developer to make that choice. Thus, the eternal loop : applications released as a blob -> this takes too much disk space, we should use shared libraries ! -> applications released as shared libraries -> DLL hell -> applications released as a blob -> ...
interfixus 8y ago
Typing me some Nim and shaking my head. Passing -static to the linker, I get neat, robust, non-dynamic executables, typically in the low MB or even kB size-range. const varname = staticRead"filename" embeds a given resource at compile time - to make the project completely selfcontained if need be, and just for the hell of it. I know you can do this in several other languages, but the Nim way is - unsurprisingly - the most ready-made, obvious and transparent one I've come across.
- rkeene2 8y ago
  Tcl beats it in terms of both transparency and ready-made-ness. In Tcl, all filesystem access goes through the Tcl Virtual Filesystem (VFS) layer. This is a layer of indirection that allows for the interpreter, including at the script-level, to intercept filesystem calls and replace them with their own data. The default (and catch-all) driver is called "native" that just passes things to the native OS, but you can write other drivers and over the decades people have done so. The most popular one is Starkit/Starpack which uses the Metakit database (which is a heirarchial database) and appends that archive to a loader (either a script, as a stub, or to an executable which includes the stub with the ability to open itself). This database can be modified externally using the starkit tools (sdx), and when you pack all your resources into it, it appears from within the interpreter that all the files are just normal files even though only the database is being accessed.
  - interfixus 8y ago
    Tcl is still a breath of fresh air, here 30 years after it first appeared. Such a pity all the minshare it has lost.
    - rkeene2 8y ago
      And it can now be compiled to machine code (TclQuadCode), an amazing feat which should have further information presented at this year's Tcl conference (already underway).
      - 8y ago
api 8y ago
We passed through a period where shared libraries were a great way to save storage and memory, but now dependency management has become such an incredibly headache that it's driving us back to monolithic binaries.
tannhaeuser 8y ago
The root of all problems are incompatible file system layouts, library version, and package formats across Linuxes. I don't see how adding yet another package format is solving this.

weberc2 8y ago

> The programming field often feels like it's in a giant loop constantly re-discovering what went before. It might feel that way, but this is driven by economics. Basically disk, memory, and bandwidth is far cheaper than it used to be so who cares if you waste a GB or so copying the same libraries all over if you don't have to solve for dependency hell?

Koshkin 8y ago
> disk is far cheaper Not in the cloud.
- weberc2 8y ago
  To be clear, you're asserting that the price of disk in the cloud is comparable to what it was decades ago?
  - Koshkin 8y ago
    Things are more complicated than that. (Depending on the pricing scheme, you might end up paying per gigabyte per hour, for example; which may or may not include transfer/storage of actual data; a faster "drive" could be much more expensive than a slower one, etc.)
    - weberc2 8y ago
      Sorry, I'm not following. Granted pricing schemes have changed, but adjusting for that, are you claiming that disk is not dramatically cheaper than it was ~20 years ago? Not trying to be a dick, just not following your point.
      - Koshkin 8y ago
        Point is, "disk" in the cloud is not the same thing as the disk in your PC.
        ecnahc515 8y ago
        No, the point is it doesn't matter where the disk is, the overall price has gone down.
        weberc2 8y ago
        Of course it's not, but the cost of both has declined.
- GordonS 8y ago
  Disagree, cloud storage is cheap - bandwidth is expensive.
- solarkraft 8y ago
  1) It is 2) Application space on the server is vanishingly tiny compared to the data you're processing, so it is a far more generous trade-off.
eeZah7Ux 8y ago
Vendorizing libraries and static linking has become a security hell. Just an example: https://blog.acolyer.org/2017/04/03/a-study-of-security-vulnerabilities-on-docker-hub/ https://blog.acolyer.org/2017/04/03/a-study-of-security-vuln...
- weberc2 8y ago
  Your link cites "little thought to the inclusion of dependencies", _not_ static linkage, as the cause of this "hell".
  - wahern 8y ago
    The implication is that statically linked binaries make it easier to throw in dependencies because you neither need to worry about how to ship them nor bother with re-using the shared libraries on the system. If there's a bug in Go's TLS implementation, you need to identify, recompile, and redistribute every Go application you use. Whereas if it were using the system's OpenSSL library everything would magically be fixed by a simple `apt-get upgrade` or `yum update`.
    - jjnoakes 8y ago
      Is this always true? If the shared library you linked to uses symbol versions and the bug fix requires a new version of a symbol, the application has to be relinked anyway. Same as static linking, except now it is not obvious you need to relink to get the bug fix.
      - rkeene2 8y ago
        Only if the fix involves breaking the old semantics entirely. It's possible in some cases to support old semantics in a safe way at the cost of performance, or in the case that it's not you can disable that call in such a way that the program aborts if it is called, thus indicating that the user should relink that program. This is why integration projects such as Linux distributions are hard, and useful.
    - 8y ago
shaan7 8y ago
> so who cares if you waste a GB or so I wonder if that sounds arrogant to just me. As a user I DO CARE. Its frustrating to find applications shipping their own copy of libraries wasting precious disk space. When I was using a Mac, I had at least 4 applications shipping their own copy of Qt5.4 ... aaaarrgh Back on the Linux desktop, package managers handle this beautifully. And for the cases where you need to skip the distro, Flatpak still lets you use system dependencies. Everyone wins. And for whoever claims storage is cheap, please link me the magical place you're buying yours from. Thanks.
- AnIdiotOnTheNet 8y ago
  Package managers come with their own limitations. You can't have multiple versions of the same application, packages are often out of date, packages sometimes conflict, you can't choose where applications are installed, etc. And storage is ridiculously cheap. You can get an 8TB HDD for $150[0]. Or a 1TB SSD for the same price[1]. [0]https://www.amazon.com/Seagate-Expansion-Desktop-External-STEB8000100/dp/B01HAPGEIE https://www.amazon.com/Seagate-Expansion-Desktop-External-ST... [1]https://www.amazon.com/Blue-NAND-1TB-SSD-WDS100T2B0A/dp/B073SBQMCX https://www.amazon.com/Blue-NAND-1TB-SSD-WDS100T2B0A/dp/B073...
  - rkeene2 8y ago
    > You can't have multiple versions of the same application In many package managers you can have multiple versions of the same application without a big deal -- in most cases you don't want to SUPPORT multiple versions of the same application, so there's always a trade-off.
    - AnIdiotOnTheNet 8y ago
      > In many package managers you can have multiple versions of the same application without a big deal In my experience, only if the distro has specifically set things up to allow that (Python2/3), or you find a PPA maintained by someone. Otherwise, time to compile from source I guess.

IanCal 8y ago

> This is not take anything away from this project, just making an observation. The programming field often feels like it's in a giant loop constantly re-discovering what went before. I think part of this comes from various tradeoffs becoming more or less important at different times. Processing power goes up so you move to fat clients, then people move to battery powered devices and heavy lifting makes more sense on servers again.

tomatocracy 8y ago
It's not really processing power so much as storage and RAM that used to be seen as the disadvantages of static binaries. The advantages of the static binary model are that it avoids "DLL hell" and makes sourcing dependencies a bit better, which are indeed laudable aims. But nowadays it has terrible security (and to some extent reliability) trade-offs - if everything is a static binary and there's a security issue in a library, you need to upgrade every single binary depending on it, which is much less reliable and more prone to error/oversight. As security and reliability have increased in importance, so the "cost" of the static binary route has actually gone up significantly in my view compared to 20-30 years ago. So it's a bit odd to me to see this model (and closely related ideas like flatpak etc) gaining currency. There must be better ways to manage the "DLL hell" issues than that.
- jungler 8y ago
  From the POV of folks working around massive server farms, all the servers are disposable, so having one binary is convenient, and the short lifespans mean they avoid the downsides. The concern has more relevance to client applications, but then the cost becomes related to update strategy, and if every app is tied to an update manager process(whether it's an app store type of thing or a custom one) then the thinking becomes similar to the cloud server setup: Why bother, I'm going to blow it all away in the next update. When the dependency management is exposed to users, there's a much bigger surface area for troubleshooting and support. It makes sense that devs favor static if they expect the whole environment to be disposable. It isn't like it was in the 90's, where patching was occasional but major new versions were essentially new products.
- 8y ago

mayank 8y ago

It's understandable due to RAM, disk, and network being way less of a constraint than it used to be (which necessitated dynamic linking and shared libraries), and the modern need to ship around binaries to hundreds or thousands of servers and have them run predictably everywhere.

v_lisivka 8y ago
Maybe it better to use DLL and ship to tens of thousands of servers instead.

mLuby 8y ago

That trend suggests that previous efforts are either not discoverable or not sufficiently extensible.

pjmlp 8y ago

Yes, I remember when it was such a big deal that we could finally use dynamic libraries on Amiga OS, MS-DOS protected mode and Windows 3.x. Even GNU/Linux made such a big deal of moving from a.out to ELF binary format, due to easier support for dynamic linking, which quite cumbersome with a.out. Now almost 30 years later it is a big deal to support static linking, go figure.

artursapek 8y ago
Technology goes in cycles, like humans do :)
- pjmlp 8y ago
  Yeah, like renaissance rediscovering Roman and Greek tech.

jcelerier 8y ago

it's not really a loop, but there are really two categories of users : * those who want applications that take the least space on the drive, and share most of their data / libraries across themselves - shared electron runtime, steam runtime shared with the os * those who want applications to conform exactly to what the developper used since the developer can guarantee a "working" configuration Ideally it should be up to the user to choose what configuration he wants, but the problem is that it's up to the developer to make that choice. Thus, the eternal loop : applications released as a blob -> this takes too much disk space, we should use shared libraries ! -> applications released as shared libraries -> DLL hell -> applications released as a blob -> ...

interfixus 8y ago

Typing me some Nim and shaking my head. Passing -static to the linker, I get neat, robust, non-dynamic executables, typically in the low MB or even kB size-range. const varname = staticRead"filename" embeds a given resource at compile time - to make the project completely selfcontained if need be, and just for the hell of it. I know you can do this in several other languages, but the Nim way is - unsurprisingly - the most ready-made, obvious and transparent one I've come across.

rkeene2 8y ago
Tcl beats it in terms of both transparency and ready-made-ness. In Tcl, all filesystem access goes through the Tcl Virtual Filesystem (VFS) layer. This is a layer of indirection that allows for the interpreter, including at the script-level, to intercept filesystem calls and replace them with their own data. The default (and catch-all) driver is called "native" that just passes things to the native OS, but you can write other drivers and over the decades people have done so. The most popular one is Starkit/Starpack which uses the Metakit database (which is a heirarchial database) and appends that archive to a loader (either a script, as a stub, or to an executable which includes the stub with the ability to open itself). This database can be modified externally using the starkit tools (sdx), and when you pack all your resources into it, it appears from within the interpreter that all the files are just normal files even though only the database is being accessed.
- interfixus 8y ago
  Tcl is still a breath of fresh air, here 30 years after it first appeared. Such a pity all the minshare it has lost.
  - rkeene2 8y ago
    And it can now be compiled to machine code (TclQuadCode), an amazing feat which should have further information presented at this year's Tcl conference (already underway).
    - 8y ago

api 8y ago

We passed through a period where shared libraries were a great way to save storage and memory, but now dependency management has become such an incredibly headache that it's driving us back to monolithic binaries.

tannhaeuser 8y ago

The root of all problems are incompatible file system layouts, library version, and package formats across Linuxes. I don't see how adding yet another package format is solving this.