6 ms·
Building static binaries with Go on Linux
- SpecialistK 2y agoVery interesting and well described! If I were to have one nitpick, it would be the use of "Unix" when it's more specific to Linux. Ex. "The libc typically used on Unix systems is glibc" However I'm sure all of the concepts still apply on BSD, Solaris, etc.
- Onavo 2y agoGo is also famous for bypassing libc and issuing syscalls directly on quite a few platforms.
- galangalalgol 2y agoI thought I'd read they backed off of that and started using posix as an abstraction layer.
- bradfitz 2y agoGo used to do raw syscalls on macOS but changed. Same with some BSDs. Go always did ~libc on Windows (and Solaris) and still does. Go still does raw syscalls on Linux, as that's a stable ABI.
- pjmlp 2y agolibc is a UNIX concept, as the OS API surface, as described by POSIX. On any other no-UNIX derived OS, it is the C compiler standard library, covering only the ISO C specifies. All the remaining OS services are exposed by other libraries, which in Windows case, the bare minimum is user, kernel and gdi dlls for the Win32 personality. Systems like IBM i, IBM z, ClearPath MCP, .... also have similar set of libraries, as do non-POSIX RTOS for embedded.
- sbstp 2y agoI feel like there's a lot of potential between Go and Cosmopolitan libc. Go itself does not use libc much, as shown in the blog, but some great libraries like SQLite3 need it (unless you use https://pkg.go.dev/modernc.org/sqlite https://pkg.go.dev/modernc.org/sqlite). The ability to build a single static binary that works on Linux, Mac and Windows using Go would be life changing for the internal tools I develop at work.
- latchkey 2y ago> The ability to build a single static binary that works on Linux, Mac and Windows using Go would be life changing for the internal tools I develop at work. Just curious, life changing in what way? Obviously, 1 is better than 3, but I'm wondering if there is some other interesting reason.
- oguz-ismail 2y agoI recall reading about new Macs with ARM chips not supporting static binaries. Is it not true?
- deleted 2y ago[deleted]
- zamadatix 2y agoThat's been Apple's stance on full static linking MacOS in general, many years prior to the move to ARM e.g. https://developer.apple.com/library/archive/qa/qa1118/_index.html https://developer.apple.com/library/archive/qa/qa1118/_index... You're welcome to ignore it of course, it's just unofficial and a large pain.
- oguz-ismail 2y ago>You're welcome to ignore it of course How do you mean? Like, is it possible to run such binaries on M1? If so I'd really like to know how
- 2y ago
- wwarner 2y agoThere’s also Filippo Valsorda linking directly to Rust via .a files. https://words.filippo.io/rustgo/ https://words.filippo.io/rustgo/
- bradfitz 2y agoCareful. We (Tailscale) tried to use static Go binaries a year or two ago built with Zig (zig cc) and the SQLite performance was atrocious. It passed all our tests but it didn't survive deploying to prod. It was a very quick (and uneventful) rollback at least. (Needless to say, we have better load testing tooling now) I forget the details, but something about the libc allocator used by SQLite-with-Zig-libc being ... not good.
- eliben 2y agoI wonder if it's Zig or musl that's to blame; did you end up statically linking with musl using musl-gcc, or did you forego static linking entirely?
- jcelerier 2y agoIt's well-known that musl is in general much, much slower than glibc. People keep rediscovering that for some reason, likely because they hear of stuff like old echoes of Usenet posts ranting against glob "bloat", not being aware that a lot of what people call bloat is specialization of a lot of performance-sensitive algorithms to leverage SIMD, special-casing, math optimisations, etc. When you check the glibc mailing lists, it's obvious performance is a predominant concern.
- tuxxi 2y agoI’ve experienced the same performance issues with cgo + a library compiled with zig cc. IIRC it seemed like an issue with the zig tooling not plumbing the optimization flags through the ancient autotools build system for our required dependency. After a while fiddling, we just rolled it back too. I haven’t tried this in about a year, so maybe the tooling doesn’t have these issues now.
- JackYoustra 2y agoI've heard you can just swap out allocators pretty easily - did something prevent this? Or perhaps its not as straightforward as I've thought...
- dylanh 2y ago
- deleted 2y ago[deleted]
- daenney 2y agoIn the specific case of SQLite, you can use it through WASM now [1]. It uses the dependency and Cgo-free Wazero runtime. Performance so far has been better than the modernc transpile and it’s probably sufficient for a lot of use cases. [1] https://github.com/ncruces/go-sqlite3 https://github.com/ncruces/go-sqlite3
- _lvbh 2y agoWhy not https://pkg.go.dev/modernc.org/sqlite https://pkg.go.dev/modernc.org/sqlite
- kyrra 2y agoPerformance is likely part of it. The WASM solution looks faster than modernc: https://github.com/cvilsmeier/go-sqlite-bench https://github.com/cvilsmeier/go-sqlite-bench
- ncruces 2y agoAuthor here. Faster was a by product. Maintainability was the goal. API coverage, ergonomics, extensibility all rank higher in my book than performance. An example I'd like to cite is sqlite-vec. Alex was able to build a Cgo-free version of it, on his own, which works fine with my bindings. This would be much harder to do with modernc. https://github.com/asg017/sqlite-vec-go-bindings https://github.com/asg017/sqlite-vec-go-bindings I'm also adding support for building off the bedrock branch (begin concurrent, wal2). You just build the branch with wasi-sdk, then embed the resulting blob.
- 0xjnml 2y ago> This would be much harder to do with modernc. https://go.dev/play/p/IztwH9Jl68i https://go.dev/play/p/IztwH9Jl68i
- daenney 2y agoThe WASM solution doesn’t rely on a custom libc or transpiler to convert C code to Go. The transpile is an amazing feat of engineering, but it’s hard to debug. I can wrap my head around the small amount of wrapping the go-sqlite3 WASM library does. If I had to I can maintain that should the maintainer lose interest. I can’t say the same for the modernc transpile. You can also apply the WASM trick to other libraries with much less effort. And as noted, it seems to be performing better. As the wasm runtime improves it should pull further ahead.
- acatton 2y agoThe secret with sqlite is to use "-tags sqlite_omit_load_extension", if you don't use any extension. (which is 99% of the users) This is explained in https://www.arp242.net/static-go.html https://www.arp242.net/static-go.html
- neonsunset 2y ago[flagged]
- qaq 2y agoand yet a ton of fundamentally important projects are written in Go like k8s, Docker, Prometheus, Terraform ...
- papichulo2023 2y agoAnd barely any big open source is written in .NET, the only one I can recall without googling is Ryujinx
- tmjwid 2y agoJellyfin is one of the bigger ones for sure.
- neonsunset 2y agoYup, and Bitwarden is another one. Many larger game publishers run their infrastructure exclusively on .NET like Roblox or Ubisoft. At the end of the day, all the tools written in Go that are listed above are not exactly paragons of performance. Ryujinx is probably one of the best showcases of the kind of task that is impossible to solve in Go effectively, and yet can be solved very well with .NET. Or you could look at Garnet which is a pure C# Redis implementation which beats vanilla Redis, KeyDB and Dragonfly.
- qaq 2y agoMany large companies including Google, Uber and so on run a ton of Go services :)
- deleted 2y ago[deleted]
- 2y ago
- nrvn 2y agoProducing static PIE binaries is a bigger challenge still. $ go build -ldflags '-linkmode external -s -w -extldflags "--static-pie"' -buildmode=pie -tags 'osusergo,netgo,static_build' main.go For anyone curious to delve into what is this and why: https://www.leviathansecurity.com/blog/aslr-protection-for-statically-linked-executables https://www.leviathansecurity.com/blog/aslr-protection-for-s...
- moondev 2y agoWhat's the best way to include the go runtime itself, as in ability to invoke the "go" program from the program itself . I'm not talking about embedding it or downloading it. I want it included within the program.
- diggan 2y agoHow are you supposed to include it within the program without somehow "embedding" it? Or am I missing some vital understanding of what "include" vs "embedding" means here?
- moondev 2y agoBy embedding it, I mean using the embed feature to pack the golang binary into the program. What I am going after is similar to kubectl and kustomize. The kustomize source code included with kubectl, it's not a binary packed in and extracted