3 ms·

Basically some motivation and how to use go + semver. But is there a way to statically-compile dependencies? Is that even a thing? (I'm not a go user so forgi

Basically some motivation and how to use go + semver. But is there a way to statically-compile dependencies? Is that even a thing? (I'm not a go user so forgive me if I need a good RTFM session.) It seems like a lot of these problems come from two dependencies wanting different versions of a third dependency. Instead of just depending on a dependency as a semver string, I could (theoretically I think) depend on a statically-compiled version of the dependency so it's free to call whatever libs it wants - effectively eliminate the concept of transitive dependencies. For projects with large dependency graphs, you may end up with relatively large binaries since you end up with lots of duplicate object-code for common libraries, but I wonder how much of a problem this actually is (and if simple de-duping may solve a huge chunk of it). We spend lots of engineering effort to resolve dependencies as source just to end up compiling them into our executable anyway. I'm sure this isn't a new concept, but it struck me as odd that go is fighting with it so much recently considering statically-compiled, "library-free" executables is definitely in go's wheelhouse - why not extend it to libraries?

matt_m 9y ago
This could still become a problem (if each dependency statically bundles its own dependencies) if any of the types are exposed in an API. For example say there is a Twitch and Facebook SDK that both expose a SetUserProfile(Image) function, and Image is provided by a third library (that is used by both SDKs). If both SDKs use different versions of the Image library, the internals of the Image struct may have changed (maybe they added support for black and white images). Even though the change may be backwards compatible at a source code level, an Image struct v1.15 wouldn't work as an Image struct v1.16 unless it had the same internal private fields (and they were used in the same way). Now, your code that loads an Image from disk can't work. It will either use v1.15 and be compatible with Twitch, or use 1.16 and be compatible with Facebook, but not both. A shared dependency works around this issue as long as the changes are backwards compatible - in this case Go would see that the minimum required version is v1.16 and both SDKs (and your code) would use that (or a newer version if you list it as a requirement).
Groxx 9y ago
Just to briefly jump in: statically compiling dependencies: roughly equivalent to "I want to link against a Go lib"? AFAIK the standard answers are "use cgo" (which has problems[1]) and "maybe plugins?"[2] but I don't know how plugins are working out. static compilation solving transitive dependencies: only if you have no singletons. In that case, yes! But the outside world is a singleton - anything on disk, through network, etc, anything with external state now has to deal with all past and future versions of the code simultaneously. Possible, but difficult and/or wasteful. For pure code though, yeah, this is a great solution, and it's part of why npm's "use all the versions" works out for micro-libs so well - they tend to have their state injected, which allows you-the-user to control how versions interact (if at all). Unfortunately you're left with node_modules[3] (as you mentioned). [1]: https://dave.cheney.net/2016/01/18/cgo-is-not-go https://dave.cheney.net/2016/01/18/cgo-is-not-go + you just inherited a multi-threaded lib, which may cause forking problems later. [2]: https://golang.org/pkg/plugin/ https://golang.org/pkg/plugin/ [3]: https://pbs.twimg.com/media/C3SOI-_WAAAM4Js.jpg https://pbs.twimg.com/media/C3SOI-_WAAAM4Js.jpg
pjmlp 9y ago
The problem also happens in static linking, this is as old as the library concept. For example, the public symbols of the libraries might collide, they might have side-effects that misbehave because there are multiple versions, they can rely on yet on another library that can actually only be linked exactly once, ....
- ryanianian 9y ago
  Isn't that just a 'bug' in the algorithm for code-layout - that the symbol names weren't unique enough or something? This would require some breaking-changes in the way libraries are laid out and linked against, but basically you "should" be able to statically compile a dependency and then effectively hide all the symbols from its dependencies so nothing else knows how to link against them directly.

matt_m 9y ago

This could still become a problem (if each dependency statically bundles its own dependencies) if any of the types are exposed in an API. For example say there is a Twitch and Facebook SDK that both expose a SetUserProfile(Image) function, and Image is provided by a third library (that is used by both SDKs). If both SDKs use different versions of the Image library, the internals of the Image struct may have changed (maybe they added support for black and white images). Even though the change may be backwards compatible at a source code level, an Image struct v1.15 wouldn't work as an Image struct v1.16 unless it had the same internal private fields (and they were used in the same way). Now, your code that loads an Image from disk can't work. It will either use v1.15 and be compatible with Twitch, or use 1.16 and be compatible with Facebook, but not both. A shared dependency works around this issue as long as the changes are backwards compatible - in this case Go would see that the minimum required version is v1.16 and both SDKs (and your code) would use that (or a newer version if you list it as a requirement).

Groxx 9y ago

Just to briefly jump in: statically compiling dependencies: roughly equivalent to "I want to link against a Go lib"? AFAIK the standard answers are "use cgo" (which has problems[1]) and "maybe plugins?"[2] but I don't know how plugins are working out. static compilation solving transitive dependencies: only if you have no singletons. In that case, yes! But the outside world is a singleton - anything on disk, through network, etc, anything with external state now has to deal with all past and future versions of the code simultaneously. Possible, but difficult and/or wasteful. For pure code though, yeah, this is a great solution, and it's part of why npm's "use all the versions" works out for micro-libs so well - they tend to have their state injected, which allows you-the-user to control how versions interact (if at all). Unfortunately you're left with node_modules[3] (as you mentioned). [1]: https://dave.cheney.net/2016/01/18/cgo-is-not-go https://dave.cheney.net/2016/01/18/cgo-is-not-go + you just inherited a multi-threaded lib, which may cause forking problems later. [2]: https://golang.org/pkg/plugin/ https://golang.org/pkg/plugin/ [3]: https://pbs.twimg.com/media/C3SOI-_WAAAM4Js.jpg https://pbs.twimg.com/media/C3SOI-_WAAAM4Js.jpg

pjmlp 9y ago

The problem also happens in static linking, this is as old as the library concept. For example, the public symbols of the libraries might collide, they might have side-effects that misbehave because there are multiple versions, they can rely on yet on another library that can actually only be linked exactly once, ....

ryanianian 9y ago
Isn't that just a 'bug' in the algorithm for code-layout - that the symbol names weren't unique enough or something? This would require some breaking-changes in the way libraries are laid out and linked against, but basically you "should" be able to statically compile a dependency and then effectively hide all the symbols from its dependencies so nothing else knows how to link against them directly.