12 ms·

The simplicity of single-file Golang deployments

189 points by KingOfCoders 4y ago

andrewfromx 4y ago
Interesting. Are you saying for a big project run something to minimize the .go files into just one? Assuming there is just one package.
- neverartful 4y ago
  Probably referring to the fact that when you build a go executable there's just a single file to deploy (the executable).
  - jjtheblunt 4y ago
    Go has a facility for embedding build time files within the resulting binary such that they can be read as if in a runtime file system, because the Go file access routines know about this file system type.
    - PaulHoule 4y ago
      You can definitely pack a Java application into a single JAR file and skip the Docker. Java's xenophobia (allergy to linking libraries) is the real root of "write once run everywhere" so often all you need is the Java runtime.
      - thefounder 4y ago
        You still need jre so it's not really a "single file" like on Go
        creshal 4y ago
        Eh. From an ops perspective there isn't much difference between an executable file that Golang statically compiled all dependencies into and embedded a file system into, and a WAR archive that the Java compiler embedded a file system including dependencies into. Both are self-contained single files you can give to a completely different organization and expect to run on the first attempt with no complications. It's just that the latter needs Tomcat (not an issue, realistically) and has to be written as EnterpriseFactoryPatternFactorySingletonAbstractBaseFactorySingletonProvider that makes you feel dead inside just from looking at the documentation; while Golang (and similar newer languages) give you a lot more flexibility and better ergonomics on the developer side.
        PaulHoule 4y ago
        Try Guava or Spring. In either case the framework supplies you with a Factory, FactoryFactory, FactoryFactoryFactory, ... that does the transitive closure so you can just get a "single" object injected into your app where you need it.
        jjtheblunt 4y ago
        "lasciate ogne speranza, voi ch'intrate" (~ lose all hope, who enters) from Dante's Inferno is what comes to mind whenever someone mentions Spring.
        erichocean 4y ago
        You still need a bespoke systemd configuration for TFA's Go deploys so it's not really a "single file" deploy their either. And like the sibling comment noted, once you're allowed to set up the machine to support easy depolyment (e.g. JRE, Tomcat), a WAR becomes a single file deploy.
        logistark 4y ago
        You can use JLink to embed only the needed modules and classes of the JRE for your application to run
      - jjtheblunt 4y ago
        class loaders in java are jvm's dynamic linker/loaders, wouldn't you say?
mastax 4y ago
> Systemd holding connections and restarting the new binary. How does this work? Or does it just mean it stops new connections while it's restarting?
- christophilus 4y ago
  Systemd effectively acts as a proxy. I don't know that it's actually a proxy, but it keeps accepting connections from what I've seen. I use it for zero-downtime single-binary deploys, and it's great.
  - yencabulator 4y ago
    No, it's not a proxy, and it's not accepting connections (unless you're using the inetd emulation, but that's rare and inefficient). It's merely passing the listening socket as an already-open file descriptor to the spawned process. The "keeps accepting" part is just the listening socket backlog. Last I looked, systemd socket passing couldn't be used to do graceful shutdown, serving existing connections with the old version while having the new version receive new connections. Outside of that, it's very nice.
- hnarn 4y ago
  Could be what's described in this post: https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket...
RcouF1uZ4gsC 4y ago
I feel docker in many cases is a hack for languages and runtimes that don’t support single file static linked binaries. Often a single binary is a simpler and better option instead of a docker container.
- mariusmg 4y ago
  Often ? It's 100% scenario. Complex apps (like Gitea for example) delivered as a single binary is basically the pinnacle of deployment.
  - dizhn 4y ago
    gitea, caddy (which can update itself even with the same modules included), restic (again in-place updates), adguard home which embeds a dhcp and dns service etc etc. I really like the stuff the golang developers can put out. I even asked someone to produce a fresbsd binary please and they added one line to their github ci to make it available that day.
- ye-olde-sysrq 4y ago
  I more view it as us recognizing that there's more to "a system" than a binary. Kubernetes is this concept taken to its conclusion (since it defines everything in code, literally everything). But docker is often a super convenient middle ground where it's not nearly as stupidly verbose to just get a simple thing running, but still checks a lot of the boxes. I used to feel similarly with Java. "Why," I asked, "would you need this docker thing? Just build the shaded JAR and off you go." And to be sure, there are some systems - especially the kind people seem to build in go (network-only APIs that never touch the fs and use few libraries) - that do not need much more than their binary to work. But what of systems that call other CLI utilities? What of systems that create data locally that you'd like to scoot around or back up? Eventually nearly every system grows at least a few weird little things you need to do to set it up and make it comfy. Docker accommodates that. I do think there's a big kernel of truth to your sentiment though - I loved rails as a framework but hated, just hated deploying it, especially if you wanted 2 sites to share a linux box. Maybe I was just bad at it but it was really easy to break BOTH sites. Docker has totally solved this problem. Same for python stuff. I do think docker is also useful as a way to make deploying ~anything all look exactly the same. "Pull image, run container with these args". I actually think this is what I like the most about it - I wrote my own thing with the python docker SDK, basically a shitty puppet/ansible, except it's shitty in the exact way I want it to be. And this has been the best side effect - I pay very little in resource overhead and suddenly now all my software uses the exact same deployment system.
danwee 4y ago
How does one handle zero downtime deployments with single-file golang binaries? I remember I tried this setup some time ago and I couldn't successfully manage cleanly to accomplish no downtime when deploying a new version of my service. The reason was mainly port reuse. I couldn't have the old and the new version of my service running on the same port... so I started to hack together something and it became dirty pretty quickly. I'm talking about deployment of new version of service on the same machine/server as the old version was running.
- iamjackg 4y ago
  If you really don't want to use different ports you can handle it with Docker. Since each container has its own IP, they can all expose the same port. Otherwise, for non-containerized deployments you'll have to resort to two different ports. In either case, you will need a reverse proxy like Traefik/Nginx in front to smartly "balance" incoming requests to the two instances of the service.
- deleted 4y ago
  [deleted]
- hnarn 4y ago
  This doesn’t sound Go-specific, if you use something like haproxy targeting multiple nodes you can take them down one by one to perform a rolling upgrade.
- dividuum 4y ago
  Can't help with how to implement this, but just to be sure: You should be able to use the same port in multiple instances if you bind those with SO_REUSEPORT. A quick search points to https://github.com/libp2p/go-reuseport https://github.com/libp2p/go-reuseport for an implementation. Now you just need a mechanism to drain the old process.
  - joncfoo 4y ago
    Rough psuedocode to do this with the built-in http.Server where startServer(..) would use the reuseport library to create the listener so multiple servers can listen within the same process: func reloadConfig(config) { if newServer, err := startServer(config); err != nil { // gracefully shutdown previous server // no new connections will go to old server oldServer.Shutdown(...) oldServer = newServer } }
masto 4y ago
When I was stuck doing a web application in Java 15 years ago, I hated everything about it except for the deployment story, which boiled down to a single .war file being pushed to the server. When we upgraded to Perl, I liked that system so we designed deployment around "PAR" files in a similar way, bundling all of the dependencies together with the application in the CI build process, and I wrote a tiny bit of infrastructure that essentially moved a symlink to make the new version live. Google uses MPM and hermetic packaging of configuration with binaries: https://sre.google/sre-book/release-engineering/#packaging-8ksQTeum https://sre.google/sre-book/release-engineering/#packaging-8... The way I see it, Docker is basically this same thing, generalized to be independent of the language/application/platform. As a practical matter, it still fundamentally has the "one file" nature. I don't see what's special or better about compiling everything into a single binary, apart from fetishizing the executable format. In any system at scale, you still have to solve for the more important problems of managing the infrastructure. "I can deploy by scping the file from my workstation to the server" is kind of a late 90s throwback, but golang is a 70s throwback, so I guess it fits?
- marcosdumay 4y ago
  > I don't see what's special or better about compiling everything into a single binary, apart from fetishizing the executable format. When you distribute your software to other people, it cuts the step of installing the correct interpreter... at the cost of requiring the correct computer architecture. It is very likely a gain.
  - somehnguy 4y ago
    Exactly - I primarily write Java and fat-jars are great when developing apps for environments I control. But if I want to send an app to a friend it's a few additional steps to make sure they have the correct version of Java, paths are setup correct, etc. This isn't always trivial if they already have a different version of Java and want things to play nice side by side. Just bundle everything into a native executable, so many little annoyances just disappear. From what I understand Java does have facilities to bundle the runtime now but I haven't had the opportunity to really play with it yet.
zer00eyz 4y ago
From the article: "Standing here it looks like Docker was invented to manage dependencies for Python, Javascript and Java. It looks strange from a platform that deploys as one single binary." Let me say the quiet part out loud: Docker is covering up the fact that we don't write deployable software any more. Go isn't perfect either. The author isn't dealing with assets (images anyone?). I think there is plenty of room for innovation here, and were over due for some change.
- ElevenLathe 4y ago
  Anymore? When were we writing "deployable" software in the past?
  - marcosdumay 4y ago
    You used to buy a disk, put it on your computer and run the thing there.
    - ElevenLathe 4y ago
      Isn't that just "running" the software? To me, "deployment" implies some repeated process that actually wasn't especially valuable to automate or be very careful about when software was released once a year (and hence the reason it wasn't). Also, disks are a horrible way to deploy software. They have all the same problems of just distributing a random tarball: What operating system is it for? What version? Where do I copy the files? How do I get the OS to automatically start the service on startup? What version of make does it use? How about which libc and cc? You can say this stuff in the README (or printed docs) but isn't something more "deployable" when it's all machine-readable and can be reasoned about automatically? This is what package managers were invented for.
    - xyzzy4747 4y ago
      My father told me a story about when he was in college, he had to find a book in the library catalogue with the program he wanted, order it and check it out, and then type it up and test/debug it. After that all his colleagues and professors wanted to borrow it as well. This was in the 70s.
deleted 4y ago
[deleted]
sneak 4y ago
I still put the single file in a docker container because docker isn't complex.
- hsn915 4y ago
  This makes zero sense. I don't like cargo culting.
  - jerf 4y ago
    There are reasons. It's a reasonable security boundary. It integrates with other things that use Docker as the primary abstraction, and there's good odds I've got other docker things that aren't single binaries like databases and other tools. It puts it into a uniform control interface that works with other things as well. It doesn't cost much additional resources over simply running the binary directly because the real runtime cost of a docker container is the mini-OS they often bring up, not the target executable. It isn't necessary, but it's not nonsense.
  - sneak 4y ago
    I deploy everything everywhere with docker, because then the only thing installed on the system is dockerd. I can deploy identically on different distributions; I don't need to know anything about the host or keep track of files on the host. I can keep all of my build artifacts in a docker image repository with versions. I can deploy any version on any host without worrying about copying the version to the host. Whether your deploy is 1000000 files or 1, this system has clear advantages to copying things to the base server OS and turning it into a snowflake.
drewg123 4y ago
The fact that it produces a single static binary is one of the nicest things about golang. This used to be easy with C (on BSD & Linux) a long time ago, but then everything started to depend on various shared libs, who then depend on other libs, then things started to even dlopen libs behind your back so they didn't even show up in ldd, etc. Sigh.
- synergy20 4y ago
  very true, for Go though if you need CGO it's hard to make a single executable, otherwise it is great. I run a few Go apps, all are single executables, upgrading to new releases has never been easier. if you have a network oriented application, nothing beats golang as far as release|maintenance is concerned.
  - dilyevsky 4y ago
    > CGO_ENABLED=1 CC=musl-gcc go build --ldflags '-linkmode external -extldflags=-static' Unless you cant use musl for some reason, but that’s a glibc problem, not Go
  - aatd86 4y ago
    Some people are writing alternative via WebAssembly. For instance wazero which I'm really excited about. https://tetrate.io/blog/introducing-wazero-from-tetrate/?utm_campaign=Wazero%20Launch&utm_content=242066038&utm_medium=social&utm_source=twitter&hss_channel=tw-998918265177952259 https://tetrate.io/blog/introducing-wazero-from-tetrate/?utm...
  - cookiengineer 4y ago
    In the malware reverse engineering scene, there are a lot of forks of the upstream "debug" go library, because it allows loading, parsing, compiling and executing libraries from disk (rather than in-kernel or in-userspace) because it is independent of dlopen. And there's also "purego" as an implementation that directly generates shellcode. Maybe those will help you, too? I am just mentioning these because for my use cases those approaches worked perfectly, CGO free. [1] https://github.com/Binject/debug https://github.com/Binject/debug [2] https://github.com/ebitengine/purego https://github.com/ebitengine/purego
ethicalsmacker 4y ago
It's not always a static binary, if you use any os/config stdlib function calls or DNS look ups. In that case you need to specify CGO_ENABLED=0 to force static builds. I have been doing single binary full website deploys for about ~16 months in production. That includes all html, css and js embedded. It has been wonderful.
- adql 4y ago
  And with a little bit of code you can do switching between "use embedded files/use local files" on the app start easily and have convenience of not having to re-compile app to change some static files.
dicroce 4y ago
A single binary is nice... Perhaps second place is using the rpath origin linker option to create a relocatable application.
marcrosoft 4y ago
How is this news. Welcome to 10 years ago.
- adql 4y ago
  It become easier few years ago as tools to embed files are now nicely builtin into Go instead of external packages.
anderspitman 4y ago
If you're looking for a similar deployment experience, but can't use Golang, we've been using Apptainer (previously Singularity) for a couple years at work. It's really nice to be able to get the benefits of containers while retaining the simplicity of copying and running a single file. Only dependency is installing Apptainer, which is easy as well. [0]: https://apptainer.org/ https://apptainer.org/
avgcorrection 4y ago
> Fast-forward and we were automatically deploying Scala applications from CI bundled in Docker in the startup of my wife. > Last forward and I have deployed a Golang application to a cloud server. Editor hello?
twic 4y ago
I feel like i'm taking crazy pills (at a low dose) when i read this stuff. I deploy Java applications. In a runnable condition, they aren't a single file, but they aren't many - maybe a dozen jars plus some scripts. Our build process puts all that in a tarball. Deployment comprises copying the tarball to the server, then unpacking it [1]. That is one step more than deploying a single binary, but it's a trivial step, and both steps are done by a release script, so there is a single user-visible step. The additional pain associated with deploying a tarball rather than a single binary is negligible. It simply is not worth worrying about [2]. But Go enjoyers make such a big deal of this single binary! What am i missing? Now, this post does talk about Docker. If you use Docker to deploy, then yes, that is more of a headache. But Docker is not the only alternative to a single binary! You can just deploy a tarball! [1] We do deploy the JDK separately. We have a script which takes a local path to a JDK tarball and a hostname, and installs the JDK in the right place on the target machine. This is a bit caveman, and it might be better to use something like Ansible, or make custom OS packages for specific JDKs, or even use something like asdf. But we don't need to deploy JDKs very often, so the script works for us. [2] Although if you insist, it's pretty easy to make a self-expanding-and-running zip, so you could have a single file if you really want: https://github.com/vmware-archive/executable-dist-plugin https://github.com/vmware-archive/executable-dist-plugin
- simiones 4y ago
  Related to [1], I thought modern Java deployment style is to bundle the required modules of the JDK with your app, rather than any concept of a "deployed JDK". As it is, the difficulty of deploying a JDK + your app is much more than a single static binary, Go-style.
  - 5e92cb50239222b 4y ago
    It depends on your requirements and environment. With most things I am working on the commands to deploy both look exactly the same: $ rsync -a foo server:/opt/foo $ rsync -a bar server:/opt/bar Can you guess which one is a static binary written in Go, and which one is a directory with a slimmed down JRE produced by jlink + an application jar? For those using containers, there's no practical difference between the two.
lenkite 4y ago
Is this article from 2016? You can do all this with Java nowadays. I have observed a lot of folks on HN whose last knowledge about Java was from a decade plus ago pontificating about Java deficiencies that no longer exist today. Use the GraalVM native build tools https://graalvm.github.io/native-build-tools/latest/index.html https://graalvm.github.io/native-build-tools/latest/index.ht.... "Use Maven to Build a Native Executable from a Java Application" https://www.graalvm.org/22.2/reference-manual/native-image/guides/use-native-image-maven-plugin/ https://www.graalvm.org/22.2/reference-manual/native-image/g...
- Rapzid 4y ago
  .Net too, also same prob with decade old worldview about .Net One thing that hasn't changed is that C# > Java ;P
- cxr 4y ago
  You, and any other non-Golang programmer, could visit the golang.org site, download the latest release, untar it, write a hello-world service, and run "go build". It will take about that many steps and about 5 minutes, and you'll have your single-file binary ready for deployment. Can you compare doing the same thing with Java? How many more steps does it take, assuming that you don't already have a standard Java dev setup that's configured and ready-to-go on your machine? Now, let's let you put your thumb on the scale and assume that you do already the standard setup, but you want to pursue what this posts lays out and that you insist can be done with Java. How much more effort does it take just to go from "typical Java setup" to "setup that actually lets you do what this article describes"? If the answer is not zero but your position is still that there's nothing special here because "You can do all this with Java nowadays", then it's because you're not understanding what "here" and "this" actually are.
  - lenkite 4y ago
    "How much more effort does it take just to go from "typical Java setup" to "setup that actually lets you do what this article describes"? " 5-6 minutes extra. 3 more steps. You are making a mountain out of a mole-hill. Sorry to burst your bias-bubble, but Java deployment is damn easy nowadays. If you are leveraging a cloud build-pack, single file native exe deployments like Go's are un-optimal since they are slower - no intelligence to perform differential updates like the way you get for a Java fat-jar or tar-ball.
mirekrusin 4y ago
We're bundling b/e services from typescript monorepo in production as single bundle files - it works very well. Main reason was simply enforcing lockfile from monorepo.
neilv 4y ago
I'm often a fan of single source files, at the package level, including inline embedded API docs and unit tests.
- tinglymintyfrsh 4y ago
  That doesn't scale because it makes merging and file navigation nearly impossible, and the increased cognitive load is too much for anything real. Encapsulate functionality and break it into logical containers. That's what modules are. That's what files are. That's what functions are. That's what software engineering is.
  - neilv 4y ago
    Sometimes it does scale.
nathants 4y ago
(swirls fancy wine) pairs well with single file frontends.
avg_dev 4y ago
I feel like I agree with the general ethos of the project. And I am also a fan of pragmatism; I think Fred Brooks referred to our trade as “toolsmiths” and I feel it is an apt word. Our work product exists solely to fill a need or to enable things that were not previously possible. I feel like I work hard not to be an idealist or to view well-written code as an end to itself. But I must confess, > Systemd also restarts the app daily to make sure it works properly long term leaves me with a viscerally negative feeling. I feel like daemons should be able to run for years unless there is some kind of leak. Maybe I am wrong.
- layer8 4y ago
  They should, but you don’t know if they will, and, when they suddenly crash after two years, whether they will be able to restart. Restarting daily ensures that any problems will be caught early, and that the last known-good configuration is only a day ago and not two years ago.
  - whateveracct 4y ago
    Restarting daily also ensures you never find entire classes of lurking bugs involving memory leaks, in-memory caches, stuff like that.
    - layer8 4y ago
      On the other hand, they also never become relevant then.
- speed_spread 4y ago
  You can see the daily restart as a restartability test. Programs expecting to run forever may develop "bad habits"
galkk 4y ago
Just a note that you can do the same now with .net: https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview?tabs=cli https://learn.microsoft.com/en-us/dotnet/core/deploying/sing...
alexisread 4y ago
This can also be done with python and pyinstaller with the one file flag. It can bundle assets and C++ libs, as well as sign the executable (eg. signtool). Worked well for me for a zero-install bridge app. https://pyinstaller.org/en/stable/usage.html https://pyinstaller.org/en/stable/usage.html
osigurdson 4y ago
I’m jealous of this coming from C#. They kind of have it but not really.
ed25519FUUU 4y ago
Static binaries and automatic code formatting (no debates on code format whatsoever) are two incredible qualities of Go that should be copied to every new language but for whatever reason are mostly left out. And I’m not taking about “making a static binary in $LANG is easy, just follow these 7 steps…” trust me it’s nothing like Go then.

189 points by KingOfCoders 4y ago

Interesting. Are you saying for a big project run something to minimize the .go files into just one? Assuming there is just one package.

neverartful 4y ago
Probably referring to the fact that when you build a go executable there's just a single file to deploy (the executable).
- jjtheblunt 4y ago
  Go has a facility for embedding build time files within the resulting binary such that they can be read as if in a runtime file system, because the Go file access routines know about this file system type.
  - PaulHoule 4y ago
    You can definitely pack a Java application into a single JAR file and skip the Docker. Java's xenophobia (allergy to linking libraries) is the real root of "write once run everywhere" so often all you need is the Java runtime.
    - thefounder 4y ago
      You still need jre so it's not really a "single file" like on Go
      - creshal 4y ago
        Eh. From an ops perspective there isn't much difference between an executable file that Golang statically compiled all dependencies into and embedded a file system into, and a WAR archive that the Java compiler embedded a file system including dependencies into. Both are self-contained single files you can give to a completely different organization and expect to run on the first attempt with no complications. It's just that the latter needs Tomcat (not an issue, realistically) and has to be written as EnterpriseFactoryPatternFactorySingletonAbstractBaseFactorySingletonProvider that makes you feel dead inside just from looking at the documentation; while Golang (and similar newer languages) give you a lot more flexibility and better ergonomics on the developer side.
        PaulHoule 4y ago
        Try Guava or Spring. In either case the framework supplies you with a Factory, FactoryFactory, FactoryFactoryFactory, ... that does the transitive closure so you can just get a "single" object injected into your app where you need it.
        jjtheblunt 4y ago
        "lasciate ogne speranza, voi ch'intrate" (~ lose all hope, who enters) from Dante's Inferno is what comes to mind whenever someone mentions Spring.
      - erichocean 4y ago
        You still need a bespoke systemd configuration for TFA's Go deploys so it's not really a "single file" deploy their either. And like the sibling comment noted, once you're allowed to set up the machine to support easy depolyment (e.g. JRE, Tomcat), a WAR becomes a single file deploy.
      - logistark 4y ago
        You can use JLink to embed only the needed modules and classes of the JRE for your application to run
    - jjtheblunt 4y ago
      class loaders in java are jvm's dynamic linker/loaders, wouldn't you say?

mastax 4y ago

> Systemd holding connections and restarting the new binary. How does this work? Or does it just mean it stops new connections while it's restarting?

christophilus 4y ago
Systemd effectively acts as a proxy. I don't know that it's actually a proxy, but it keeps accepting connections from what I've seen. I use it for zero-downtime single-binary deploys, and it's great.
- yencabulator 4y ago
  No, it's not a proxy, and it's not accepting connections (unless you're using the inetd emulation, but that's rare and inefficient). It's merely passing the listening socket as an already-open file descriptor to the spawned process. The "keeps accepting" part is just the listening socket backlog. Last I looked, systemd socket passing couldn't be used to do graceful shutdown, serving existing connections with the old version while having the new version receive new connections. Outside of that, it's very nice.
hnarn 4y ago
Could be what's described in this post: https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket...

RcouF1uZ4gsC 4y ago

I feel docker in many cases is a hack for languages and runtimes that don’t support single file static linked binaries. Often a single binary is a simpler and better option instead of a docker container.

mariusmg 4y ago
Often ? It's 100% scenario. Complex apps (like Gitea for example) delivered as a single binary is basically the pinnacle of deployment.
- dizhn 4y ago
  gitea, caddy (which can update itself even with the same modules included), restic (again in-place updates), adguard home which embeds a dhcp and dns service etc etc. I really like the stuff the golang developers can put out. I even asked someone to produce a fresbsd binary please and they added one line to their github ci to make it available that day.
ye-olde-sysrq 4y ago
I more view it as us recognizing that there's more to "a system" than a binary. Kubernetes is this concept taken to its conclusion (since it defines everything in code, literally everything). But docker is often a super convenient middle ground where it's not nearly as stupidly verbose to just get a simple thing running, but still checks a lot of the boxes. I used to feel similarly with Java. "Why," I asked, "would you need this docker thing? Just build the shaded JAR and off you go." And to be sure, there are some systems - especially the kind people seem to build in go (network-only APIs that never touch the fs and use few libraries) - that do not need much more than their binary to work. But what of systems that call other CLI utilities? What of systems that create data locally that you'd like to scoot around or back up? Eventually nearly every system grows at least a few weird little things you need to do to set it up and make it comfy. Docker accommodates that. I do think there's a big kernel of truth to your sentiment though - I loved rails as a framework but hated, just hated deploying it, especially if you wanted 2 sites to share a linux box. Maybe I was just bad at it but it was really easy to break BOTH sites. Docker has totally solved this problem. Same for python stuff. I do think docker is also useful as a way to make deploying ~anything all look exactly the same. "Pull image, run container with these args". I actually think this is what I like the most about it - I wrote my own thing with the python docker SDK, basically a shitty puppet/ansible, except it's shitty in the exact way I want it to be. And this has been the best side effect - I pay very little in resource overhead and suddenly now all my software uses the exact same deployment system.

danwee 4y ago

How does one handle zero downtime deployments with single-file golang binaries? I remember I tried this setup some time ago and I couldn't successfully manage cleanly to accomplish no downtime when deploying a new version of my service. The reason was mainly port reuse. I couldn't have the old and the new version of my service running on the same port... so I started to hack together something and it became dirty pretty quickly. I'm talking about deployment of new version of service on the same machine/server as the old version was running.

iamjackg 4y ago
If you really don't want to use different ports you can handle it with Docker. Since each container has its own IP, they can all expose the same port. Otherwise, for non-containerized deployments you'll have to resort to two different ports. In either case, you will need a reverse proxy like Traefik/Nginx in front to smartly "balance" incoming requests to the two instances of the service.
deleted 4y ago
[deleted]
hnarn 4y ago
This doesn’t sound Go-specific, if you use something like haproxy targeting multiple nodes you can take them down one by one to perform a rolling upgrade.
dividuum 4y ago
Can't help with how to implement this, but just to be sure: You should be able to use the same port in multiple instances if you bind those with SO_REUSEPORT. A quick search points to https://github.com/libp2p/go-reuseport https://github.com/libp2p/go-reuseport for an implementation. Now you just need a mechanism to drain the old process.
- joncfoo 4y ago
  Rough psuedocode to do this with the built-in http.Server where startServer(..) would use the reuseport library to create the listener so multiple servers can listen within the same process: func reloadConfig(config) { if newServer, err := startServer(config); err != nil { // gracefully shutdown previous server // no new connections will go to old server oldServer.Shutdown(...) oldServer = newServer } }

masto 4y ago

When I was stuck doing a web application in Java 15 years ago, I hated everything about it except for the deployment story, which boiled down to a single .war file being pushed to the server. When we upgraded to Perl, I liked that system so we designed deployment around "PAR" files in a similar way, bundling all of the dependencies together with the application in the CI build process, and I wrote a tiny bit of infrastructure that essentially moved a symlink to make the new version live. Google uses MPM and hermetic packaging of configuration with binaries: https://sre.google/sre-book/release-engineering/#packaging-8ksQTeum https://sre.google/sre-book/release-engineering/#packaging-8... The way I see it, Docker is basically this same thing, generalized to be independent of the language/application/platform. As a practical matter, it still fundamentally has the "one file" nature. I don't see what's special or better about compiling everything into a single binary, apart from fetishizing the executable format. In any system at scale, you still have to solve for the more important problems of managing the infrastructure. "I can deploy by scping the file from my workstation to the server" is kind of a late 90s throwback, but golang is a 70s throwback, so I guess it fits?

marcosdumay 4y ago
> I don't see what's special or better about compiling everything into a single binary, apart from fetishizing the executable format. When you distribute your software to other people, it cuts the step of installing the correct interpreter... at the cost of requiring the correct computer architecture. It is very likely a gain.
- somehnguy 4y ago
  Exactly - I primarily write Java and fat-jars are great when developing apps for environments I control. But if I want to send an app to a friend it's a few additional steps to make sure they have the correct version of Java, paths are setup correct, etc. This isn't always trivial if they already have a different version of Java and want things to play nice side by side. Just bundle everything into a native executable, so many little annoyances just disappear. From what I understand Java does have facilities to bundle the runtime now but I haven't had the opportunity to really play with it yet.

zer00eyz 4y ago

From the article: "Standing here it looks like Docker was invented to manage dependencies for Python, Javascript and Java. It looks strange from a platform that deploys as one single binary." Let me say the quiet part out loud: Docker is covering up the fact that we don't write deployable software any more. Go isn't perfect either. The author isn't dealing with assets (images anyone?). I think there is plenty of room for innovation here, and were over due for some change.

ElevenLathe 4y ago
Anymore? When were we writing "deployable" software in the past?
- marcosdumay 4y ago
  You used to buy a disk, put it on your computer and run the thing there.
  - ElevenLathe 4y ago
    Isn't that just "running" the software? To me, "deployment" implies some repeated process that actually wasn't especially valuable to automate or be very careful about when software was released once a year (and hence the reason it wasn't). Also, disks are a horrible way to deploy software. They have all the same problems of just distributing a random tarball: What operating system is it for? What version? Where do I copy the files? How do I get the OS to automatically start the service on startup? What version of make does it use? How about which libc and cc? You can say this stuff in the README (or printed docs) but isn't something more "deployable" when it's all machine-readable and can be reasoned about automatically? This is what package managers were invented for.
  - xyzzy4747 4y ago
    My father told me a story about when he was in college, he had to find a book in the library catalogue with the program he wanted, order it and check it out, and then type it up and test/debug it. After that all his colleagues and professors wanted to borrow it as well. This was in the 70s.

deleted 4y ago

[deleted]

sneak 4y ago

I still put the single file in a docker container because docker isn't complex.

hsn915 4y ago
This makes zero sense. I don't like cargo culting.
- jerf 4y ago
  There are reasons. It's a reasonable security boundary. It integrates with other things that use Docker as the primary abstraction, and there's good odds I've got other docker things that aren't single binaries like databases and other tools. It puts it into a uniform control interface that works with other things as well. It doesn't cost much additional resources over simply running the binary directly because the real runtime cost of a docker container is the mini-OS they often bring up, not the target executable. It isn't necessary, but it's not nonsense.
- sneak 4y ago
  I deploy everything everywhere with docker, because then the only thing installed on the system is dockerd. I can deploy identically on different distributions; I don't need to know anything about the host or keep track of files on the host. I can keep all of my build artifacts in a docker image repository with versions. I can deploy any version on any host without worrying about copying the version to the host. Whether your deploy is 1000000 files or 1, this system has clear advantages to copying things to the base server OS and turning it into a snowflake.

drewg123 4y ago

The fact that it produces a single static binary is one of the nicest things about golang. This used to be easy with C (on BSD & Linux) a long time ago, but then everything started to depend on various shared libs, who then depend on other libs, then things started to even dlopen libs behind your back so they didn't even show up in ldd, etc. Sigh.

synergy20 4y ago
very true, for Go though if you need CGO it's hard to make a single executable, otherwise it is great. I run a few Go apps, all are single executables, upgrading to new releases has never been easier. if you have a network oriented application, nothing beats golang as far as release|maintenance is concerned.
- dilyevsky 4y ago
  > CGO_ENABLED=1 CC=musl-gcc go build --ldflags '-linkmode external -extldflags=-static' Unless you cant use musl for some reason, but that’s a glibc problem, not Go
- aatd86 4y ago
  Some people are writing alternative via WebAssembly. For instance wazero which I'm really excited about. https://tetrate.io/blog/introducing-wazero-from-tetrate/?utm_campaign=Wazero%20Launch&utm_content=242066038&utm_medium=social&utm_source=twitter&hss_channel=tw-998918265177952259 https://tetrate.io/blog/introducing-wazero-from-tetrate/?utm...
- cookiengineer 4y ago
  In the malware reverse engineering scene, there are a lot of forks of the upstream "debug" go library, because it allows loading, parsing, compiling and executing libraries from disk (rather than in-kernel or in-userspace) because it is independent of dlopen. And there's also "purego" as an implementation that directly generates shellcode. Maybe those will help you, too? I am just mentioning these because for my use cases those approaches worked perfectly, CGO free. [1] https://github.com/Binject/debug https://github.com/Binject/debug [2] https://github.com/ebitengine/purego https://github.com/ebitengine/purego

ethicalsmacker 4y ago

It's not always a static binary, if you use any os/config stdlib function calls or DNS look ups. In that case you need to specify CGO_ENABLED=0 to force static builds. I have been doing single binary full website deploys for about ~16 months in production. That includes all html, css and js embedded. It has been wonderful.

adql 4y ago
And with a little bit of code you can do switching between "use embedded files/use local files" on the app start easily and have convenience of not having to re-compile app to change some static files.

dicroce 4y ago

A single binary is nice... Perhaps second place is using the rpath origin linker option to create a relocatable application.

marcrosoft 4y ago

How is this news. Welcome to 10 years ago.

adql 4y ago
It become easier few years ago as tools to embed files are now nicely builtin into Go instead of external packages.

anderspitman 4y ago

If you're looking for a similar deployment experience, but can't use Golang, we've been using Apptainer (previously Singularity) for a couple years at work. It's really nice to be able to get the benefits of containers while retaining the simplicity of copying and running a single file. Only dependency is installing Apptainer, which is easy as well. [0]: https://apptainer.org/ https://apptainer.org/

avgcorrection 4y ago

> Fast-forward and we were automatically deploying Scala applications from CI bundled in Docker in the startup of my wife. > Last forward and I have deployed a Golang application to a cloud server. Editor hello?

twic 4y ago

I feel like i'm taking crazy pills (at a low dose) when i read this stuff. I deploy Java applications. In a runnable condition, they aren't a single file, but they aren't many - maybe a dozen jars plus some scripts. Our build process puts all that in a tarball. Deployment comprises copying the tarball to the server, then unpacking it [1]. That is one step more than deploying a single binary, but it's a trivial step, and both steps are done by a release script, so there is a single user-visible step. The additional pain associated with deploying a tarball rather than a single binary is negligible. It simply is not worth worrying about [2]. But Go enjoyers make such a big deal of this single binary! What am i missing? Now, this post does talk about Docker. If you use Docker to deploy, then yes, that is more of a headache. But Docker is not the only alternative to a single binary! You can just deploy a tarball! [1] We do deploy the JDK separately. We have a script which takes a local path to a JDK tarball and a hostname, and installs the JDK in the right place on the target machine. This is a bit caveman, and it might be better to use something like Ansible, or make custom OS packages for specific JDKs, or even use something like asdf. But we don't need to deploy JDKs very often, so the script works for us. [2] Although if you insist, it's pretty easy to make a self-expanding-and-running zip, so you could have a single file if you really want: https://github.com/vmware-archive/executable-dist-plugin https://github.com/vmware-archive/executable-dist-plugin

simiones 4y ago
Related to [1], I thought modern Java deployment style is to bundle the required modules of the JDK with your app, rather than any concept of a "deployed JDK". As it is, the difficulty of deploying a JDK + your app is much more than a single static binary, Go-style.
- 5e92cb50239222b 4y ago
  It depends on your requirements and environment. With most things I am working on the commands to deploy both look exactly the same: $ rsync -a foo server:/opt/foo $ rsync -a bar server:/opt/bar Can you guess which one is a static binary written in Go, and which one is a directory with a slimmed down JRE produced by jlink + an application jar? For those using containers, there's no practical difference between the two.

lenkite 4y ago

Is this article from 2016? You can do all this with Java nowadays. I have observed a lot of folks on HN whose last knowledge about Java was from a decade plus ago pontificating about Java deficiencies that no longer exist today. Use the GraalVM native build tools https://graalvm.github.io/native-build-tools/latest/index.html https://graalvm.github.io/native-build-tools/latest/index.ht.... "Use Maven to Build a Native Executable from a Java Application" https://www.graalvm.org/22.2/reference-manual/native-image/guides/use-native-image-maven-plugin/ https://www.graalvm.org/22.2/reference-manual/native-image/g...

Rapzid 4y ago
.Net too, also same prob with decade old worldview about .Net One thing that hasn't changed is that C# > Java ;P
cxr 4y ago
You, and any other non-Golang programmer, could visit the golang.org site, download the latest release, untar it, write a hello-world service, and run "go build". It will take about that many steps and about 5 minutes, and you'll have your single-file binary ready for deployment. Can you compare doing the same thing with Java? How many more steps does it take, assuming that you don't already have a standard Java dev setup that's configured and ready-to-go on your machine? Now, let's let you put your thumb on the scale and assume that you do already the standard setup, but you want to pursue what this posts lays out and that you insist can be done with Java. How much more effort does it take just to go from "typical Java setup" to "setup that actually lets you do what this article describes"? If the answer is not zero but your position is still that there's nothing special here because "You can do all this with Java nowadays", then it's because you're not understanding what "here" and "this" actually are.
- lenkite 4y ago
  "How much more effort does it take just to go from "typical Java setup" to "setup that actually lets you do what this article describes"? " 5-6 minutes extra. 3 more steps. You are making a mountain out of a mole-hill. Sorry to burst your bias-bubble, but Java deployment is damn easy nowadays. If you are leveraging a cloud build-pack, single file native exe deployments like Go's are un-optimal since they are slower - no intelligence to perform differential updates like the way you get for a Java fat-jar or tar-ball.

mirekrusin 4y ago

We're bundling b/e services from typescript monorepo in production as single bundle files - it works very well. Main reason was simply enforcing lockfile from monorepo.

neilv 4y ago

I'm often a fan of single source files, at the package level, including inline embedded API docs and unit tests.

tinglymintyfrsh 4y ago
That doesn't scale because it makes merging and file navigation nearly impossible, and the increased cognitive load is too much for anything real. Encapsulate functionality and break it into logical containers. That's what modules are. That's what files are. That's what functions are. That's what software engineering is.
- neilv 4y ago
  Sometimes it does scale.

nathants 4y ago

(swirls fancy wine) pairs well with single file frontends.

avg_dev 4y ago

I feel like I agree with the general ethos of the project. And I am also a fan of pragmatism; I think Fred Brooks referred to our trade as “toolsmiths” and I feel it is an apt word. Our work product exists solely to fill a need or to enable things that were not previously possible. I feel like I work hard not to be an idealist or to view well-written code as an end to itself. But I must confess, > Systemd also restarts the app daily to make sure it works properly long term leaves me with a viscerally negative feeling. I feel like daemons should be able to run for years unless there is some kind of leak. Maybe I am wrong.

layer8 4y ago
They should, but you don’t know if they will, and, when they suddenly crash after two years, whether they will be able to restart. Restarting daily ensures that any problems will be caught early, and that the last known-good configuration is only a day ago and not two years ago.
- whateveracct 4y ago
  Restarting daily also ensures you never find entire classes of lurking bugs involving memory leaks, in-memory caches, stuff like that.
  - layer8 4y ago
    On the other hand, they also never become relevant then.
speed_spread 4y ago
You can see the daily restart as a restartability test. Programs expecting to run forever may develop "bad habits"

galkk 4y ago

Just a note that you can do the same now with .net: https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview?tabs=cli https://learn.microsoft.com/en-us/dotnet/core/deploying/sing...

alexisread 4y ago

This can also be done with python and pyinstaller with the one file flag. It can bundle assets and C++ libs, as well as sign the executable (eg. signtool). Worked well for me for a zero-install bridge app. https://pyinstaller.org/en/stable/usage.html https://pyinstaller.org/en/stable/usage.html

osigurdson 4y ago

I’m jealous of this coming from C#. They kind of have it but not really.

ed25519FUUU 4y ago

Static binaries and automatic code formatting (no debates on code format whatsoever) are two incredible qualities of Go that should be copied to every new language but for whatever reason are mostly left out. And I’m not taking about “making a static binary in $LANG is easy, just follow these 7 steps…” trust me it’s nothing like Go then.