3 ms·

Back when I programmed in CL and participated in comp.lang.lisp, the topic of Static Executables was a recurring thing. I never understood why people were so ob

by jwr 6y ago

Back when I programmed in CL and participated in comp.lang.lisp, the topic of Static Executables was a recurring thing. I never understood why people were so obsessed with it, but somehow this became the Major Showstopper for wider CL adoption, in the eyes of some users. Most people who actually used CL for building stuff were somewhat amused, and tried to find out what was so important about encapsulating everything into a single executable and why other languages didn't have to do it, but CL absolutely did. I thought this was a red herring, but indeed for some reason it did affect acceptability. Glad to see a solution that looks fairly good.

7thaccount 6y ago
Single executables make distribution really easy. Sure, a lot of languages like Python don't have this, but it's also easier to tell someone to download and install Python to run a script than the SBCL process. The entire thing from emacs/slime to the package manager and so forth was just very complicated to me. Things like Portacle didn't help much either.
- jwr 6y ago
  Yeah, I heard that argument — but seriously, how many people run or "install" single executables these days? What actually gets distributed as a single executable? I honestly can't think of anything, on any system, except perhaps ngrok which I use for development.
  - jgalt212 6y ago
    I think that's true for the technical crowd, but much less so for the general populace.
    - sidpatil 6y ago
      I don't see how, unless you're counting Windows application installers as single-file executables.
  - phone8675309 6y ago
    The previous version of the article (https://www.timmons.dev/posts/static-executables-with-sbcl.html https://www.timmons.dev/posts/static-executables-with-sbcl.h...) explains a few contexts where static binaries are desirable.
  - tumdum_ 6y ago
    In my case: $ find ~/bin /usr/bin/ /bin/ -executable | xargs -I XXX file XXX | rg "statically linked" | awk '{ print $1 }' /home/t/bin/floki: /home/t/bin/yj: /home/t/bin/helm: /home/t/bin/fzf: /home/t/bin/kubectl: /usr/bin/containerd-shim-runc-v1: /usr/bin/hyperfine: /usr/bin/docker-init: /usr/bin/containerd-shim: /usr/bin/containerd-shim-runc-v2: /bin/busybox:
  - yellowapple 6y ago
    > but seriously, how many people run or "install" single executables these days? Pretty much everyone who's ever used an AppImage on Linux, for one. This is also still the norm for macOS, from the user's point of view (yes, the application is technically a folder, but as far as most end users are concerned it's a single "thing" that gets dropped into their Applications folder). Still the norm for Windows, too, at least for installers (and "portable" apps, while not as common as ones installed to Program Files, certainly ain't obscure, even in this day and age).
  - qbasic_forever 6y ago
    Almost every single tool in the k8s ecosystem is a simple statically linked go-based executable. Core stuff like kubectl, helm, etc. Read any k8s guide and step 1 is usually "go download all theses exes from github releases" (or more in vogue now, install a global NPM package that has the executable in its bin dir).
  - 7thaccount 6y ago
    In my eyes, that's part of the problem. A single executable is a lot easier to deal with. Having someone download Anaconda Python (massive install) to run a script is a huge fail in my eyes.
  - aseipp 6y ago
    Chicken and egg. "many people" on Linux don't download executables and run them even if they're perfectly trusting of the source, because they'd often be linked against missing or incompatible dynamic libraries, which can lead to other problems during use. So then users are hesitant of them, so there's less of a desire to produce them in the first place (because users don't want it.) This is exacerbated by the fact static linking on Linux is complicated by historical factors and design decisions in some components like glibc. Go and Alpine Linux especially have caused a resurgence of interest in distributing static binaries. The vast majority of tools in the Go ecosystem encourage this, for instance. Really the whole conflation of and "executable application" as a "single file" is a historical mistake that we made but haven't fully let go of yet, and it's caused huge issues for users. For example, the distinction between "static linking" and "dynamic linking" largely isn't exposed to users on macOS, because they use .app bundles -- for all intents and purposes every macOS .app is "statically linked" because it can never resolve against other (possibly incompatible) dynamic libraries and it's a single file the user can manage. Then static/dynamic linking is just a technical aspect of how some amounts of code get loaded into the process, it's not a factor which impacts distribution or anything. At the end of the day, an "application" is NOT a file. It's an abstract object from the users perspective. Maybe there is a single file hiding behind the curtain, or it's a collection of files, or maybe something else entirely, but they aren't the same. Even Windows is also now moving in this direction (you can now have 'Runtimes' in Windows so that, say, a .py file and its data files can appear collectively like an .exe and be managed like a single file, with a single python runtime in the background), and Linux is too with things like AppImage, etc. Static linking is dead, long live static linking, etc etc.
- taeric 6y ago
  This is a standing trend that goes many ways. A common detour you will hit, is that the ease you had in distribution of the program can be erased by the complication in tracking what statically linked lib is active in that program. Such that static versus dynamic linking is just a show swinging of a trend back and forth.
TeMPOraL 6y ago
> why other languages didn't have to do it, but CL absolutely did Possibly a feedback loop on lack of popularity :). Other languages do have to do this in some way or another, with respect to their runtimes. The dependencies can be dynamically loaded, but the entry point - from the user's POV - still has to be invocable by double-clicking or by ./, or you'll lose audience. Also size considerations; people are now more willing to download a 100+ MB trivial application than they were just a few years ago. On the runtime end, C/C++ solve this by having the runtime being part of the system, Java by somehow managing to make itself required everywhere, so everybody has a JRE already. Rust also compiles down to regular executables. Python got popular enough that at least Linux users are likely to have several runtimes already installed - but then Windows users don't, and any serious app will ship the runtime with it, etc. In my experience, it always boiled down to this: how to ship a portable Lisp application in a single package (by "portable", I mean no installation step), where the user just has to double-click on something and have it run, and at no point they have to know or care it's written in Lisp? This, IIRC, was meaningfully non-trivial, especially if you wanted to interop with external dependencies by FFI. I remember that at one point, I shipped my code to a non-Lisper in form of a Dockerfile which handled installing SBCL for them. Not only they were grateful, they've managed to pass on that program to another friend who (back then) was not much of a coder at all, and they both used it. That's the qualitative difference easy deployment makes.
jchw 6y ago
With some languages the problem is solved by simply being so ubiquitous that the runtime is already on your machine. I think this is true of C, C++, Python, Java... The second best thing is a fully self-contained executable like you get with Go or Rust. (I do not think it is a coincidence that both Go and Rust have been massively successful, though I think statically linked binaries are only a minimal part of that.) The third best thing is shipping the binary alongside the app. The worst thing I can think of is just not shipping a runtime. (Of course when it comes to package maintainers, it may still be desirable to ship a separate runtime package that everything links to, so as long as the language provides ABI stability for the runtime... but I like to think of package maintainer use cases as being separate from normal usage.) And of course, for developers who primarily write apps that don't "ship" to external users, it would be normal to feel puzzled about this.
jimbokun 6y ago
Static executables seems like one of the key selling points driving adoption of Go, for example.
rjsw 6y ago
I help develop a CL application that is delivered as a static executable built using SBCL, it doesn't need any other shared libraries so save-lisp-and-die is enough.

by jwr 6y ago

7thaccount 6y ago

Single executables make distribution really easy. Sure, a lot of languages like Python don't have this, but it's also easier to tell someone to download and install Python to run a script than the SBCL process. The entire thing from emacs/slime to the package manager and so forth was just very complicated to me. Things like Portacle didn't help much either.

jwr 6y ago
Yeah, I heard that argument — but seriously, how many people run or "install" single executables these days? What actually gets distributed as a single executable? I honestly can't think of anything, on any system, except perhaps ngrok which I use for development.
- jgalt212 6y ago
  I think that's true for the technical crowd, but much less so for the general populace.
  - sidpatil 6y ago
    I don't see how, unless you're counting Windows application installers as single-file executables.
- phone8675309 6y ago
  The previous version of the article (https://www.timmons.dev/posts/static-executables-with-sbcl.html https://www.timmons.dev/posts/static-executables-with-sbcl.h...) explains a few contexts where static binaries are desirable.
- tumdum_ 6y ago
  In my case: $ find ~/bin /usr/bin/ /bin/ -executable | xargs -I XXX file XXX | rg "statically linked" | awk '{ print $1 }' /home/t/bin/floki: /home/t/bin/yj: /home/t/bin/helm: /home/t/bin/fzf: /home/t/bin/kubectl: /usr/bin/containerd-shim-runc-v1: /usr/bin/hyperfine: /usr/bin/docker-init: /usr/bin/containerd-shim: /usr/bin/containerd-shim-runc-v2: /bin/busybox:
- yellowapple 6y ago
  > but seriously, how many people run or "install" single executables these days? Pretty much everyone who's ever used an AppImage on Linux, for one. This is also still the norm for macOS, from the user's point of view (yes, the application is technically a folder, but as far as most end users are concerned it's a single "thing" that gets dropped into their Applications folder). Still the norm for Windows, too, at least for installers (and "portable" apps, while not as common as ones installed to Program Files, certainly ain't obscure, even in this day and age).
- qbasic_forever 6y ago
  Almost every single tool in the k8s ecosystem is a simple statically linked go-based executable. Core stuff like kubectl, helm, etc. Read any k8s guide and step 1 is usually "go download all theses exes from github releases" (or more in vogue now, install a global NPM package that has the executable in its bin dir).
- 7thaccount 6y ago
  In my eyes, that's part of the problem. A single executable is a lot easier to deal with. Having someone download Anaconda Python (massive install) to run a script is a huge fail in my eyes.
- aseipp 6y ago
  Chicken and egg. "many people" on Linux don't download executables and run them even if they're perfectly trusting of the source, because they'd often be linked against missing or incompatible dynamic libraries, which can lead to other problems during use. So then users are hesitant of them, so there's less of a desire to produce them in the first place (because users don't want it.) This is exacerbated by the fact static linking on Linux is complicated by historical factors and design decisions in some components like glibc. Go and Alpine Linux especially have caused a resurgence of interest in distributing static binaries. The vast majority of tools in the Go ecosystem encourage this, for instance. Really the whole conflation of and "executable application" as a "single file" is a historical mistake that we made but haven't fully let go of yet, and it's caused huge issues for users. For example, the distinction between "static linking" and "dynamic linking" largely isn't exposed to users on macOS, because they use .app bundles -- for all intents and purposes every macOS .app is "statically linked" because it can never resolve against other (possibly incompatible) dynamic libraries and it's a single file the user can manage. Then static/dynamic linking is just a technical aspect of how some amounts of code get loaded into the process, it's not a factor which impacts distribution or anything. At the end of the day, an "application" is NOT a file. It's an abstract object from the users perspective. Maybe there is a single file hiding behind the curtain, or it's a collection of files, or maybe something else entirely, but they aren't the same. Even Windows is also now moving in this direction (you can now have 'Runtimes' in Windows so that, say, a .py file and its data files can appear collectively like an .exe and be managed like a single file, with a single python runtime in the background), and Linux is too with things like AppImage, etc. Static linking is dead, long live static linking, etc etc.
taeric 6y ago
This is a standing trend that goes many ways. A common detour you will hit, is that the ease you had in distribution of the program can be erased by the complication in tracking what statically linked lib is active in that program. Such that static versus dynamic linking is just a show swinging of a trend back and forth.

TeMPOraL 6y ago

> why other languages didn't have to do it, but CL absolutely did Possibly a feedback loop on lack of popularity :). Other languages do have to do this in some way or another, with respect to their runtimes. The dependencies can be dynamically loaded, but the entry point - from the user's POV - still has to be invocable by double-clicking or by ./, or you'll lose audience. Also size considerations; people are now more willing to download a 100+ MB trivial application than they were just a few years ago. On the runtime end, C/C++ solve this by having the runtime being part of the system, Java by somehow managing to make itself required everywhere, so everybody has a JRE already. Rust also compiles down to regular executables. Python got popular enough that at least Linux users are likely to have several runtimes already installed - but then Windows users don't, and any serious app will ship the runtime with it, etc. In my experience, it always boiled down to this: how to ship a portable Lisp application in a single package (by "portable", I mean no installation step), where the user just has to double-click on something and have it run, and at no point they have to know or care it's written in Lisp? This, IIRC, was meaningfully non-trivial, especially if you wanted to interop with external dependencies by FFI. I remember that at one point, I shipped my code to a non-Lisper in form of a Dockerfile which handled installing SBCL for them. Not only they were grateful, they've managed to pass on that program to another friend who (back then) was not much of a coder at all, and they both used it. That's the qualitative difference easy deployment makes.

jchw 6y ago

With some languages the problem is solved by simply being so ubiquitous that the runtime is already on your machine. I think this is true of C, C++, Python, Java... The second best thing is a fully self-contained executable like you get with Go or Rust. (I do not think it is a coincidence that both Go and Rust have been massively successful, though I think statically linked binaries are only a minimal part of that.) The third best thing is shipping the binary alongside the app. The worst thing I can think of is just not shipping a runtime. (Of course when it comes to package maintainers, it may still be desirable to ship a separate runtime package that everything links to, so as long as the language provides ABI stability for the runtime... but I like to think of package maintainer use cases as being separate from normal usage.) And of course, for developers who primarily write apps that don't "ship" to external users, it would be normal to feel puzzled about this.

jimbokun 6y ago

Static executables seems like one of the key selling points driving adoption of Go, for example.

rjsw 6y ago

I help develop a CL application that is delivered as a static executable built using SBCL, it doesn't need any other shared libraries so save-lisp-and-die is enough.