7 ms·

Bakeware – Compile Elixir applications into single executable binaries

182 points by tempodox 6y ago

sajan45 6y ago
After using Go, I always felt the need for something like this in the Elixir world.
jdellinger 6y ago
In .NET we have a very similar tool for creating such packed binaries, dotnet-warp. I used it in one of my projects and quite liked it, since it's also quite easy to cross compile (cross-pack?) for the 3 major operating systems. I like the general idea, your independent of the system wide framework version and it still has this "one-click" install procedure (dropping the binary in your path). However, I guess this is also the negativ Part. Users don't expect that a single binary extracts itself to somewhere --> uninstalling the binary leaves traces on the system. Definitely looking forward to try it out for elixir, wondering how fast the erlang/elixir startup really is.
- crusso 6y ago
  From the page: ~0.5s startup times or better on our computers
- tasogare 6y ago
  The lack of single binary output will be fixed in dotnet 5.
- rhplus 6y ago
  Single-file, platform-specific executables, framework independence and dependency trimming are all available in .NET Core 3.0: https://www.hanselman.com/blog/MakingATinyNETCore30EntirelySelfcontainedSingleExecutable.aspx https://www.hanselman.com/blog/MakingATinyNETCore30EntirelyS... https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#compiledeploy https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotne...
- csdreamer7 6y ago
  Is there anything like this in Java? I am aware of launch4j, but curious if there is a single step approach for all 3 platforms.
  - madduci 6y ago
    Yes, since Java 9: it is called "jlink" and it's shipped with the OpenJDK. It assumes you have modularized jars. An alternative is the GraalVM, which can compile the code as a C++ binary or a shared application, using LLVM as backend. If you work with Webservices (especially JavaEE), you might also want to have a look at Quarkus, which also uses GraalVM to create single binary executables
    - vips7L 6y ago
      jlink/jpackage just isn't the same as the other things. It doesn't produce a single executable like .Net core or Bakeware which embed the vm into the binary. In .Net's case it self extracts on first run. With jlink/jpackage you either ship an installer or a zip file.
- at_a_remove 6y ago
  Complete n00b question -- you can do that? The last time I touched anything .NET was about a decade ago. My somewhat old-school superiors were unimpressed with the ability to come up with a plain .exe at the end of the process. My lack of familiarity with how Visual Studio had evolved certainly played into it; the IDE made me feel like a chimp dropped into an airliner cockpit. I had the worst time trying to figure out how to turn off the "of course you have an enterprise server dedicated to delivering upgrades!" setting. None of it seemed to, uh, scale down for our piddly purposes.
  - rhplus 6y ago
    .NET Core is a ground-up rewrite of the frameworks and tools. It's completely decoupled from Visual Studio. This would get you started on a toy project (one source file, one project file) now: dotnet new console --output sample1 dotnet publish --self-contained -r linux-x64 sample1 https://docs.microsoft.com/en-us/dotnet/core/get-started https://docs.microsoft.com/en-us/dotnet/core/get-started
arjan_sch 6y ago
Looks great! Did not look at it in detail, but I'm wondering about portability, eg how this works with system libraries, like OpenSSL. Is the resulting binary portable to systems with different C libraries than the ones the OTP system was linked against originally?
hauxir 6y ago
does it support hot code deployment?
- moocowtruck 6y ago
  you can't have your cake and eat it too
  - vindarel 6y ago
    Well in Common Lisp we can. We can build a self-contained binary, run it, start a Swank server, connect to it from home and code against the running program. We can even install new libraries without stopping anything. Not saying it's great for software updates, but it definitely is possible, and helps (like, update users' config etc).
    - dnautics 6y ago
      That's not hot code reloading in the erlang sense. Hot code reloading is having a NamedModule be replaced by NamedModule on the fly and have all of the processes executing functions in that module's (global) namespace respect the new binary code in accordance to the well-defined hot code reloading API that erlang has established, followed by kicking out the old namespace when it's dependent processes have drained of it. This is not trivial considering that different processes may be holding onto that code for different lifetimes, and all must be managed sanely.
- joshuakelly 6y ago
  Can vanilla Elixir releases, for that matter? Not from what I've seen - would love to know if I'm wrong.
  - albatross13 6y ago
    I believe so. I think this is what he was referring to: https://blog.appsignal.com/2018/10/16/elixir-alchemy-hot-code-reloading-in-elixir.html https://blog.appsignal.com/2018/10/16/elixir-alchemy-hot-cod...
crusso 6y ago
I love this idea. I've been using Elixir for years, on and off. The syntax is approachable, the actor model is solid and easy to reason about with many threads running. The Phoenix ecosystem is fantastic to work with, particularly now with LiveView making quick web UIs so effortless to create. My main practical problem with Elixir over the years has been handing the tools I've created to others who might find them useful. Bakeware looks like the right way to proceed.
victor106 6y ago
Wish something like this existed for Java
- ctas 6y ago
  I believe GraalVM[0] makes your wish come true. It allows you to compile your Java code into a single binary and offers other features which makes Java a feasible solution for CLI tools. [0] https://www.graalvm.org/ https://www.graalvm.org/
- madduci 6y ago
  There is: you can use jlink/jpackage or GraalVM compiler to produce compiled executables
zapnuk 6y ago
As a more or less elixir beginner, elixir/mix releases remain confusing to me. First of all, it took me way longer than it should be to figure out how to set the entrypoint of my application. More importantly the resulting releases are kinda confusing, at least to me. Why does it need to contain 17 (13 .exe + 4 .bat) executables? Why are there 55 c-header files in the directory of the erts? Why are there so many configuration files? I'm sure that they are there for a good reason, but the current solution is daunting, such that it lowers my enjoyment with the otherwise very elegant language. IMO there should be a release approach that is very simple + compact, and extendable. This would allow people like me to hit the compile button and distribute my application in very manageable parts. Though further configuration it should be possible to end up the very adjustable release that currently exists. This project seems to go into into the direction I would like. I hope the windows release is soonish such that can try it.
- brightball 6y ago
  The thing to remember about Elixir/Erlang is that it gives you a lot of deployment options. There's this mix of concerns with deployment tooling to both support all of those options and be simple at the same time, which is tough to achieve. Just as an example, with many Java App Servers you deploy your packaged java code and it automatically handles seamless deployment across the cluster with zero downtime. And there are lots of different server options. With languages using K8's and Docker, you're deploying a configured container across a system that will help manage the same thing. Distillery has been the standard for deployments with Elixir for a while now and supports everything. Releases are slowly rolling this functionality into the core, but don't currently support all of the options that you get with distillery. But with the BEAM you've got clustering built in. You've got hot reloading options which literally deploy your new code in the middle of running code without so much as dropping a socket connection. You get the ability to roll that back too. It's a little more complicated because it comes with a lot of very unique options that are built in instead of delegated to another system.
- rhizome31 6y ago
  My experience is that preparing a release is much more straightforward since Elixir 1.9, which added release support to the Elixir standard distribution. That said I tend to treat a release as a black box, I haven't tried to understand what all those files are and I just interact with bin/myapp. However I still have two complaints. First, the system on which the release is built and the system where it's going to run need to be similar (same version of the same Linux distribution). In practice this means that these days we build releases in containers matching the target host. Second, releases can't run mix commands so we have to write boilerplate code specifically for releases for things like database migrations, etc. It would be nice if writing app management commands could be a bit more DRY. I could be wrong but at first glance it doesn't seem that Bakeware addresses these issue.
abhijat 6y ago
Has anyone used elixir for building command line tools? It seems like this would be very useful for distributing them.
- zorbash 6y ago
  I have and blogged about it here https://zorbash.com/post/building-command-line-applications-with-elixir/ https://zorbash.com/post/building-command-line-applications-... I'll give bakeware a go and probably update the post with the results.
- nickjj 6y ago
  The readme file in the repo mentions a ~500ms start up time. Maybe it's just me, but I wouldn't want to wait half a second for a tool to start running. For context most popular Unix tools start up in 1-2ms.
  - pzmarzly 6y ago
    On my PC: time node -e "console.log('a')" # 0.377 total time ruby -e "p 'aaa'" # 0.084 total time python3 -c "print('a')" # 0.020 total time elixir -e "IO.puts 'a'" # 0.117 total time _build/prod/rel/bakeware/simple_script Hi # 0.205 total without zstd, 0.212 with Another question is which language loads modules the fastest (afaik Ruby is kind of slow when it comes to `require`).
    - YorickPeterse 6y ago
      > Another question is which language loads modules the fastest (afaik Ruby is kind of slow when it comes to `require`). I think both Python and Ruby loosely follow a similar model. The difference is that IIRC Python caches bytecode files, while Ruby compiles them from scratch every time. Not sure how big of a performance impact that has, especially for simple cases like this. And just to toot my own horn, using Inko (https://inko-lang.org/ https://inko-lang.org/), I get: inko /tmp/test.inko # 0.470 total Note that this invokes the horribly slow Ruby compiler currently in use. If you invoke the VM directly on the compiled bytecode, it only takes about 10 milliseconds to run. Using the `master` branch: vm/target/release/ivm /home/yorickpeterse/.cache/inko/bytecode/release/43/38021ddc9a3449ace13288a2fac894d1d3e2aaa.ibi # 0.008.8 total In this case all modules (including the standard library) are included in the bytecode file, meaning no disk IO is necessary, and modules can be parsed in parallel. This is something dynamic languages like Ruby and Python can't do, as loading modules is something that can be done anywhere at any time, so you have to process them one by one.
kungfooguru 6y ago
makeself (https://makeself.io/ https://makeself.io/) is a generic way to do this for Linux that works for any Erlang/Elixir release -- meaning no use of mix or rebar3: https://gist.github.com/tsloughter/d62aad6d67b263e69275376b1c80320d https://gist.github.com/tsloughter/d62aad6d67b263e69275376b1... Maybe something as simple is possible for Windows as well?

182 points by tempodox 6y ago

After using Go, I always felt the need for something like this in the Elixir world.

In .NET we have a very similar tool for creating such packed binaries, dotnet-warp. I used it in one of my projects and quite liked it, since it's also quite easy to cross compile (cross-pack?) for the 3 major operating systems. I like the general idea, your independent of the system wide framework version and it still has this "one-click" install procedure (dropping the binary in your path). However, I guess this is also the negativ Part. Users don't expect that a single binary extracts itself to somewhere --> uninstalling the binary leaves traces on the system. Definitely looking forward to try it out for elixir, wondering how fast the erlang/elixir startup really is.

crusso 6y ago
From the page: ~0.5s startup times or better on our computers
tasogare 6y ago
The lack of single binary output will be fixed in dotnet 5.
rhplus 6y ago
Single-file, platform-specific executables, framework independence and dependency trimming are all available in .NET Core 3.0: https://www.hanselman.com/blog/MakingATinyNETCore30EntirelySelfcontainedSingleExecutable.aspx https://www.hanselman.com/blog/MakingATinyNETCore30EntirelyS... https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#compiledeploy https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotne...
csdreamer7 6y ago
Is there anything like this in Java? I am aware of launch4j, but curious if there is a single step approach for all 3 platforms.
- madduci 6y ago
  Yes, since Java 9: it is called "jlink" and it's shipped with the OpenJDK. It assumes you have modularized jars. An alternative is the GraalVM, which can compile the code as a C++ binary or a shared application, using LLVM as backend. If you work with Webservices (especially JavaEE), you might also want to have a look at Quarkus, which also uses GraalVM to create single binary executables
  - vips7L 6y ago
    jlink/jpackage just isn't the same as the other things. It doesn't produce a single executable like .Net core or Bakeware which embed the vm into the binary. In .Net's case it self extracts on first run. With jlink/jpackage you either ship an installer or a zip file.
at_a_remove 6y ago
Complete n00b question -- you can do that? The last time I touched anything .NET was about a decade ago. My somewhat old-school superiors were unimpressed with the ability to come up with a plain .exe at the end of the process. My lack of familiarity with how Visual Studio had evolved certainly played into it; the IDE made me feel like a chimp dropped into an airliner cockpit. I had the worst time trying to figure out how to turn off the "of course you have an enterprise server dedicated to delivering upgrades!" setting. None of it seemed to, uh, scale down for our piddly purposes.
- rhplus 6y ago
  .NET Core is a ground-up rewrite of the frameworks and tools. It's completely decoupled from Visual Studio. This would get you started on a toy project (one source file, one project file) now: dotnet new console --output sample1 dotnet publish --self-contained -r linux-x64 sample1 https://docs.microsoft.com/en-us/dotnet/core/get-started https://docs.microsoft.com/en-us/dotnet/core/get-started

arjan_sch 6y ago

Looks great! Did not look at it in detail, but I'm wondering about portability, eg how this works with system libraries, like OpenSSL. Is the resulting binary portable to systems with different C libraries than the ones the OTP system was linked against originally?

hauxir 6y ago

does it support hot code deployment?

moocowtruck 6y ago
you can't have your cake and eat it too
- vindarel 6y ago
  Well in Common Lisp we can. We can build a self-contained binary, run it, start a Swank server, connect to it from home and code against the running program. We can even install new libraries without stopping anything. Not saying it's great for software updates, but it definitely is possible, and helps (like, update users' config etc).
  - dnautics 6y ago
    That's not hot code reloading in the erlang sense. Hot code reloading is having a NamedModule be replaced by NamedModule on the fly and have all of the processes executing functions in that module's (global) namespace respect the new binary code in accordance to the well-defined hot code reloading API that erlang has established, followed by kicking out the old namespace when it's dependent processes have drained of it. This is not trivial considering that different processes may be holding onto that code for different lifetimes, and all must be managed sanely.
joshuakelly 6y ago
Can vanilla Elixir releases, for that matter? Not from what I've seen - would love to know if I'm wrong.
- albatross13 6y ago
  I believe so. I think this is what he was referring to: https://blog.appsignal.com/2018/10/16/elixir-alchemy-hot-code-reloading-in-elixir.html https://blog.appsignal.com/2018/10/16/elixir-alchemy-hot-cod...

crusso 6y ago

I love this idea. I've been using Elixir for years, on and off. The syntax is approachable, the actor model is solid and easy to reason about with many threads running. The Phoenix ecosystem is fantastic to work with, particularly now with LiveView making quick web UIs so effortless to create. My main practical problem with Elixir over the years has been handing the tools I've created to others who might find them useful. Bakeware looks like the right way to proceed.

victor106 6y ago

Wish something like this existed for Java

ctas 6y ago
I believe GraalVM[0] makes your wish come true. It allows you to compile your Java code into a single binary and offers other features which makes Java a feasible solution for CLI tools. [0] https://www.graalvm.org/ https://www.graalvm.org/
madduci 6y ago
There is: you can use jlink/jpackage or GraalVM compiler to produce compiled executables

zapnuk 6y ago

As a more or less elixir beginner, elixir/mix releases remain confusing to me. First of all, it took me way longer than it should be to figure out how to set the entrypoint of my application. More importantly the resulting releases are kinda confusing, at least to me. Why does it need to contain 17 (13 .exe + 4 .bat) executables? Why are there 55 c-header files in the directory of the erts? Why are there so many configuration files? I'm sure that they are there for a good reason, but the current solution is daunting, such that it lowers my enjoyment with the otherwise very elegant language. IMO there should be a release approach that is very simple + compact, and extendable. This would allow people like me to hit the compile button and distribute my application in very manageable parts. Though further configuration it should be possible to end up the very adjustable release that currently exists. This project seems to go into into the direction I would like. I hope the windows release is soonish such that can try it.

brightball 6y ago
The thing to remember about Elixir/Erlang is that it gives you a lot of deployment options. There's this mix of concerns with deployment tooling to both support all of those options and be simple at the same time, which is tough to achieve. Just as an example, with many Java App Servers you deploy your packaged java code and it automatically handles seamless deployment across the cluster with zero downtime. And there are lots of different server options. With languages using K8's and Docker, you're deploying a configured container across a system that will help manage the same thing. Distillery has been the standard for deployments with Elixir for a while now and supports everything. Releases are slowly rolling this functionality into the core, but don't currently support all of the options that you get with distillery. But with the BEAM you've got clustering built in. You've got hot reloading options which literally deploy your new code in the middle of running code without so much as dropping a socket connection. You get the ability to roll that back too. It's a little more complicated because it comes with a lot of very unique options that are built in instead of delegated to another system.
rhizome31 6y ago
My experience is that preparing a release is much more straightforward since Elixir 1.9, which added release support to the Elixir standard distribution. That said I tend to treat a release as a black box, I haven't tried to understand what all those files are and I just interact with bin/myapp. However I still have two complaints. First, the system on which the release is built and the system where it's going to run need to be similar (same version of the same Linux distribution). In practice this means that these days we build releases in containers matching the target host. Second, releases can't run mix commands so we have to write boilerplate code specifically for releases for things like database migrations, etc. It would be nice if writing app management commands could be a bit more DRY. I could be wrong but at first glance it doesn't seem that Bakeware addresses these issue.

abhijat 6y ago

Has anyone used elixir for building command line tools? It seems like this would be very useful for distributing them.

zorbash 6y ago
I have and blogged about it here https://zorbash.com/post/building-command-line-applications-with-elixir/ https://zorbash.com/post/building-command-line-applications-... I'll give bakeware a go and probably update the post with the results.
nickjj 6y ago
The readme file in the repo mentions a ~500ms start up time. Maybe it's just me, but I wouldn't want to wait half a second for a tool to start running. For context most popular Unix tools start up in 1-2ms.
- pzmarzly 6y ago
  On my PC: time node -e "console.log('a')" # 0.377 total time ruby -e "p 'aaa'" # 0.084 total time python3 -c "print('a')" # 0.020 total time elixir -e "IO.puts 'a'" # 0.117 total time _build/prod/rel/bakeware/simple_script Hi # 0.205 total without zstd, 0.212 with Another question is which language loads modules the fastest (afaik Ruby is kind of slow when it comes to `require`).
  - YorickPeterse 6y ago
    > Another question is which language loads modules the fastest (afaik Ruby is kind of slow when it comes to `require`). I think both Python and Ruby loosely follow a similar model. The difference is that IIRC Python caches bytecode files, while Ruby compiles them from scratch every time. Not sure how big of a performance impact that has, especially for simple cases like this. And just to toot my own horn, using Inko (https://inko-lang.org/ https://inko-lang.org/), I get: inko /tmp/test.inko # 0.470 total Note that this invokes the horribly slow Ruby compiler currently in use. If you invoke the VM directly on the compiled bytecode, it only takes about 10 milliseconds to run. Using the `master` branch: vm/target/release/ivm /home/yorickpeterse/.cache/inko/bytecode/release/43/38021ddc9a3449ace13288a2fac894d1d3e2aaa.ibi # 0.008.8 total In this case all modules (including the standard library) are included in the bytecode file, meaning no disk IO is necessary, and modules can be parsed in parallel. This is something dynamic languages like Ruby and Python can't do, as loading modules is something that can be done anywhere at any time, so you have to process them one by one.

kungfooguru 6y ago

makeself (https://makeself.io/ https://makeself.io/) is a generic way to do this for Linux that works for any Erlang/Elixir release -- meaning no use of mix or rebar3: https://gist.github.com/tsloughter/d62aad6d67b263e69275376b1c80320d https://gist.github.com/tsloughter/d62aad6d67b263e69275376b1... Maybe something as simple is possible for Windows as well?