5 ms·

Reproducibility in Disguise

54 points by setheron 2y ago

skybrian 2y ago
I'm guessing this problem might be language-specific. Does Bazel do better at importing external dependencies for Go?
- setheron 2y ago
  You'd have the same problem with CGO or JNI for Java. C/C++ shared libraries are never part of the package versioning in language package managers (that's the part we stick our heads in the sand about)
- TheDong 2y ago
  If you look at how cgo is linked, you'll see it's a magic comment in go source code: https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80dca4f355b8fb884646370/jack.go#L4 https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80... If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. Admittedly, you'll usually get compilation errors for undefined references in compiled languages (modulo dlopen), so at least you'll get a build error instead of a runtime error like you do in python. The solution for both Go and Python is for bazel to also control the system dependencies, i.e. to force you to run all code in a container sandbox bazel controls, force you to specify all external dependencies from linux kernel version up to go compiler version, and build all of that itself. In other words, bazel is a lacking implementation of NixOS.
  - klodolph 2y ago
    > If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. Go has an internal build system as well as a compiler. The build system drives the compiler. When you use Bazel with Go, you’re bypassing the Go build system entirely. Bazel directly invokes the Go compiler. This means that any of those comments will also get ignored. (Generally, this is how Bazel works for other languages too. Bazel + Rust does not use Cargo. Bazel + Java does not use, like, Maven or Gradle.) You have to specify the dependencies in your build file. Let's say you have a Go library abc, with a C library xyz. go_library( name = "abc", srcs = [ "abc.go", ], cdeps = [ "//path/to/xyz", ], cgo = True, importpath = "path/to/abc", ) > In other words, bazel is a lacking implementation of NixOS. Bazel and NixOS are both good tools for making reproducible builds. They’re even a good match for each other—grab your compiler from NixPkgs, and build the rest of your project in Bazel. You get the benefit of Bazel’s fine-grained dependencies (Nix has very coarse dependency management), and you can get some specific version of GCC built for both Linux and macOS by tapping into NixPkgs. A marriage made in heaven. (Except for the fact that you are now using two tools which both have very steep learning curves.) There are a lot of interesting similarities between Bazel and Nix, which is not surprising, since they are solving similar problems.
    - TheDong 2y ago
      > That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. > This means that any of those comments will also get ignored I cloned this example: https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea61253a62b99a61edb5509f5de/examples/basic-gazelle https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea6... And then ran "go get github.com/xthexder/go-jack" and added an 'import _ "github.com/xthexder/go-jack"' to 'cmd/roll.go'. $ bazel run //:gazelle-update-repos $ bazel run //:basic-gazelle roll __main__/external/com_github_xthexder_go_jack/jack.go:11:10: fatal error: jack/jack.h: No such file or directory compilation terminated. compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2 $ apt-get install libjack-dev $ bazel run //:basic-gazelle roll Number rolled: 80 Sure looks like you're wrong. Bazel obviously didn't ignore those comments because it errored out on trying to find the include. (And of course it didn't ignore them, those comments are preprocessor instructions for the go compiler, which bazel runs under the hood). It obviously didn't need me to specify cdeps because it found the C dependency on my host when I installed it without changing a single bazel-related file. It looks like your understanding might be wrong, sorry.
      - klodolph 2y ago
        You’re making a complaint about Gazelle. If you import third-party packages with c dependencies with Gazelle, obviously, the two choices are it breaks hermeticity or you vendor the C code. Bazel does ignore those comments. Gazelle does not.
        TheDong 2y ago
        > Bazel does ignore those comments. Gazelle does not. I really don't understand what you're saying. gazelle does not seem to do anything related to the c dependencies there. The full diff gazelle generated is just changing "deps" in 'BUILD.bazel' files to include the library, and adding a "go_repository" to "deps.bzl". Not that it would matter anyway, gazelle is part of bazel. There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle. I'm only using gazelle here because that's the only go example that upstream has in the repo. The "bazel build" part is very clearly the part that is looking for these C files, and doing so in a non-hermetic way. It's possible to make it hermetic, sure, if you pay attention and vendor things or such, but it's obviously not required. That's all I was saying, just like you can have python break hermeticity and depend on external C libraries, you can have go break hermeticity and depend on external c libraries. It's possible to use bazel such that it's hermetic, but it doesn't stop you from doing it wrong. That's all this thread is about, is whether it's also possible to make these non-hermetic references in languages other than python. Like, if you go back and read my commend and your comment a few up, you'll see that my original claim was just that you could impurely reference the host "libjack-dev", which I have very clearly shown you can, and your claim was that it's literally impossible to do that, which it very obviously is possible.
- stitched2gethr 2y ago
  Agreed. It feels like shared dependencies are akin to globals in your language of choice. Sharing dependency instances among components creates problems. A uses b and c, both of which want to use different versions of d. The simple answer, which is understandably difficult for existing languages to solve, is to just keep 2 copies of d.
  - skybrian 2y ago
    Go does keep both versions like that for major versions (which are treated like independent libraries), but not for smaller upgrades that are supposed to be compatible.
spankalee 2y ago
edit: The previous title was "Reproducibility in Disguise: Bazel, Dependencies, and the Versioning Lie" Eh... the real lie is the signal version policy in the first place. It can be broken, at least in Google, and must be often otherwise there'd be massive gridlock. If you want to bring in a new third-party library that has a dependency that already exists in the monorepo, you'd better _pray_ that the version in the monorepo is already compatible. If not, you have to update that library and potentially every target that depends on it, maybe transitively. Trying to do that can sometimes take years - no joke. So the single-version policy flexes, and exceptions are supposed to be temporary, but they're not always. In the end there are two competing sometimes good, sometimes bad goals: - Libraries should be somewhat discouraged from making too many or too flippant breaking changes. By having to update clients, they feel the pain and take migration more into account. - Libraries should be able to make real improvements, even if they are breaking changes, and having clients shouldn't burden them with unbounded costs. They should be able to distribute reasonable costs of updating by letting clients update on their own time with versioning. Vendored third party packages only highlight this tension, because the external world has largely settled on versioning as the approach which clearly conflicts with signle-versions, but it really existed anyway.
- setheron 2y ago
  Single version policy doesn't solve version conflicts but it does put it right in your face which is as good as we can do. Without single versions, new packages to the SAT solver can introduce new duplicate shared objects secretly breaking your hermetic seal. It doesn't hide the complexity like package managers do by picking a language package by SemVer and crossing our fingers that the shared objects included will work too.
- deleted 2y ago
  [deleted]
nrclark 2y ago
One other hidden Bazel trap that I've seen is for companies to migrate a large codebase to Bazel, but then to rely on OS-provided tools and libraries. Commonly, this gets paired with a glib answer like "it's fine, we build from inside of a Docker container". But I've never seen that Docker image linked into Bazel's dependency resolver, or the compose scripts used to launch the container. This has the following effects: 1. There are unexpressed package/tool dependencies. 2. Across a large organization, Bazel's reproducibility guarantees go out the window. 3. Developers can't just clone the repo and start using Bazel. Instead, they have to pull down some pinned Docker image, or build it themselves and lose reproducibility. 4. This effectively poisons the cache whenever the Docker image is updated or rebuilt. If using a shared remote cache, it can be a major issue. If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem?
- setheron 2y ago
  Docker did us wonders but also set us back in many ways (same view point about the proliferation of development on MacOS). I've lately been thinking why isn't there a public vendored third_party set to use. If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.
  - klodolph 2y ago
    > If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win. “It has been 0 days since a minor version update in a third-party library has broken my code.” This is just a fiendishly difficult problem to solve.
- klodolph 2y ago
  > If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem? Here are some ways you can approach this problem: If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this. Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment. Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this. As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.
fragmede 2y ago
The gross hack is you import libbfoo1 and libfoo2 when breaking changes like that are involved
- setheron 2y ago
  Then the symbols interpose (shadow) and now you have divergent behavior depending on the order they are loaded (which can change in Python as they are dlopen)

54 points by setheron 2y ago

skybrian 2y ago

I'm guessing this problem might be language-specific. Does Bazel do better at importing external dependencies for Go?

setheron 2y ago
You'd have the same problem with CGO or JNI for Java. C/C++ shared libraries are never part of the package versioning in language package managers (that's the part we stick our heads in the sand about)
TheDong 2y ago
If you look at how cgo is linked, you'll see it's a magic comment in go source code: https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80dca4f355b8fb884646370/jack.go#L4 https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80... If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. Admittedly, you'll usually get compilation errors for undefined references in compiled languages (modulo dlopen), so at least you'll get a build error instead of a runtime error like you do in python. The solution for both Go and Python is for bazel to also control the system dependencies, i.e. to force you to run all code in a container sandbox bazel controls, force you to specify all external dependencies from linux kernel version up to go compiler version, and build all of that itself. In other words, bazel is a lacking implementation of NixOS.
- klodolph 2y ago
  > If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. Go has an internal build system as well as a compiler. The build system drives the compiler. When you use Bazel with Go, you’re bypassing the Go build system entirely. Bazel directly invokes the Go compiler. This means that any of those comments will also get ignored. (Generally, this is how Bazel works for other languages too. Bazel + Rust does not use Cargo. Bazel + Java does not use, like, Maven or Gradle.) You have to specify the dependencies in your build file. Let's say you have a Go library abc, with a C library xyz. go_library( name = "abc", srcs = [ "abc.go", ], cdeps = [ "//path/to/xyz", ], cgo = True, importpath = "path/to/abc", ) > In other words, bazel is a lacking implementation of NixOS. Bazel and NixOS are both good tools for making reproducible builds. They’re even a good match for each other—grab your compiler from NixPkgs, and build the rest of your project in Bazel. You get the benefit of Bazel’s fine-grained dependencies (Nix has very coarse dependency management), and you can get some specific version of GCC built for both Linux and macOS by tapping into NixPkgs. A marriage made in heaven. (Except for the fact that you are now using two tools which both have very steep learning curves.) There are a lot of interesting similarities between Bazel and Nix, which is not surprising, since they are solving similar problems.
  - TheDong 2y ago
    > That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. > This means that any of those comments will also get ignored I cloned this example: https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea61253a62b99a61edb5509f5de/examples/basic-gazelle https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea6... And then ran "go get github.com/xthexder/go-jack" and added an 'import _ "github.com/xthexder/go-jack"' to 'cmd/roll.go'. $ bazel run //:gazelle-update-repos $ bazel run //:basic-gazelle roll __main__/external/com_github_xthexder_go_jack/jack.go:11:10: fatal error: jack/jack.h: No such file or directory compilation terminated. compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2 $ apt-get install libjack-dev $ bazel run //:basic-gazelle roll Number rolled: 80 Sure looks like you're wrong. Bazel obviously didn't ignore those comments because it errored out on trying to find the include. (And of course it didn't ignore them, those comments are preprocessor instructions for the go compiler, which bazel runs under the hood). It obviously didn't need me to specify cdeps because it found the C dependency on my host when I installed it without changing a single bazel-related file. It looks like your understanding might be wrong, sorry.
    - klodolph 2y ago
      You’re making a complaint about Gazelle. If you import third-party packages with c dependencies with Gazelle, obviously, the two choices are it breaks hermeticity or you vendor the C code. Bazel does ignore those comments. Gazelle does not.
      - TheDong 2y ago
        > Bazel does ignore those comments. Gazelle does not. I really don't understand what you're saying. gazelle does not seem to do anything related to the c dependencies there. The full diff gazelle generated is just changing "deps" in 'BUILD.bazel' files to include the library, and adding a "go_repository" to "deps.bzl". Not that it would matter anyway, gazelle is part of bazel. There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle. I'm only using gazelle here because that's the only go example that upstream has in the repo. The "bazel build" part is very clearly the part that is looking for these C files, and doing so in a non-hermetic way. It's possible to make it hermetic, sure, if you pay attention and vendor things or such, but it's obviously not required. That's all I was saying, just like you can have python break hermeticity and depend on external C libraries, you can have go break hermeticity and depend on external c libraries. It's possible to use bazel such that it's hermetic, but it doesn't stop you from doing it wrong. That's all this thread is about, is whether it's also possible to make these non-hermetic references in languages other than python. Like, if you go back and read my commend and your comment a few up, you'll see that my original claim was just that you could impurely reference the host "libjack-dev", which I have very clearly shown you can, and your claim was that it's literally impossible to do that, which it very obviously is possible.
stitched2gethr 2y ago
Agreed. It feels like shared dependencies are akin to globals in your language of choice. Sharing dependency instances among components creates problems. A uses b and c, both of which want to use different versions of d. The simple answer, which is understandably difficult for existing languages to solve, is to just keep 2 copies of d.
- skybrian 2y ago
  Go does keep both versions like that for major versions (which are treated like independent libraries), but not for smaller upgrades that are supposed to be compatible.

spankalee 2y ago

edit: The previous title was "Reproducibility in Disguise: Bazel, Dependencies, and the Versioning Lie" Eh... the real lie is the signal version policy in the first place. It can be broken, at least in Google, and must be often otherwise there'd be massive gridlock. If you want to bring in a new third-party library that has a dependency that already exists in the monorepo, you'd better _pray_ that the version in the monorepo is already compatible. If not, you have to update that library and potentially every target that depends on it, maybe transitively. Trying to do that can sometimes take years - no joke. So the single-version policy flexes, and exceptions are supposed to be temporary, but they're not always. In the end there are two competing sometimes good, sometimes bad goals: - Libraries should be somewhat discouraged from making too many or too flippant breaking changes. By having to update clients, they feel the pain and take migration more into account. - Libraries should be able to make real improvements, even if they are breaking changes, and having clients shouldn't burden them with unbounded costs. They should be able to distribute reasonable costs of updating by letting clients update on their own time with versioning. Vendored third party packages only highlight this tension, because the external world has largely settled on versioning as the approach which clearly conflicts with signle-versions, but it really existed anyway.

setheron 2y ago
Single version policy doesn't solve version conflicts but it does put it right in your face which is as good as we can do. Without single versions, new packages to the SAT solver can introduce new duplicate shared objects secretly breaking your hermetic seal. It doesn't hide the complexity like package managers do by picking a language package by SemVer and crossing our fingers that the shared objects included will work too.
deleted 2y ago
[deleted]

nrclark 2y ago

One other hidden Bazel trap that I've seen is for companies to migrate a large codebase to Bazel, but then to rely on OS-provided tools and libraries. Commonly, this gets paired with a glib answer like "it's fine, we build from inside of a Docker container". But I've never seen that Docker image linked into Bazel's dependency resolver, or the compose scripts used to launch the container. This has the following effects: 1. There are unexpressed package/tool dependencies. 2. Across a large organization, Bazel's reproducibility guarantees go out the window. 3. Developers can't just clone the repo and start using Bazel. Instead, they have to pull down some pinned Docker image, or build it themselves and lose reproducibility. 4. This effectively poisons the cache whenever the Docker image is updated or rebuilt. If using a shared remote cache, it can be a major issue. If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem?

setheron 2y ago
Docker did us wonders but also set us back in many ways (same view point about the proliferation of development on MacOS). I've lately been thinking why isn't there a public vendored third_party set to use. If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win.
- klodolph 2y ago
  > If we can agree living at HEAD is preferred having a communal vendored third_party seems like a clear win. “It has been 0 days since a minor version update in a third-party library has broken my code.” This is just a fiendishly difficult problem to solve.
klodolph 2y ago
> If an organization isn't big enough to vendor every single tool dependency, shared library, etc (which basically requires building out an OS distribution in Bazel), what's the right way to approach this problem? Here are some ways you can approach this problem: If you have a bunch of shared tools & libraries and you don’t want to vendor the whole packages, you can vendor the artifacts. You can do this piecemeal or as a big chunk. Basically, instead of a Docker image containing your compiler or whatever, you have a tarball. You make Bazel responsible for downloading the tarball. Or maybe it’s several tarballs, it doesn’t matter. Bazel is actually pretty good at this. Another approach is to combine Bazel with something else reproducible, like Nix. For now, I’d suggest taking a very basic approach, the most simple approach, which is to create an environment using Nix (containing your toolchain, third-party libraries, and Bazel) and then building your code inside that environment using Bazel. You can use the local_repository() rule to access the Nix environment. Bazel + Nix is a really good idea, but maybe now is not the time to dive into that. When I’ve investigated the Bazel + Nix combination, it looks like neither system is exactly stable / mature enough yet. Bazel just recently launched bzlmod and the Bazel ecosystem is shifting to use bzlmod everywhere. Nix is shifting to flakes but there are some pains there too (especially around documentation, but there are also some things that don’t quite work with flakes yet). So in the future, you could do something kind of, well, cursed, where you have a Bazel and Nix lasagna. You use Nix to run Bazel, and then you use Nix packages as dependencies of your Bazel build. In theory, this could give you the best of both worlds—you get the wonderful fine-grained dependencies of Bazel, the rich build system, the super fast performance. And then you get access to reproducible builds of all sorts of third-party dependencies through Nix. In practice, you need to become something of an expert in both Bazel and Nix in order to do this. As a final option, depending on the languages you are using, you may be happy with just plain bzlmod. Like, Go integration with Bazel is damn solid. You don’t need to vendor anything, you can just keep using go.mod, and Bazel will deal with it (with some help from Gazelle). Bazel will even download the Go compiler for you, without any additional setup. A lot of other languages work the same way—it’s just that C and C++ are notable exceptions, where Bazel just grabs the system compiler by default, and package management for C and C++ is complete chaos.

fragmede 2y ago

The gross hack is you import libbfoo1 and libfoo2 when breaking changes like that are involved

setheron 2y ago
Then the symbols interpose (shadow) and now you have divergent behavior depending on the order they are loaded (which can change in Python as they are dlopen)