3 ms·

If you look at how cgo is linked, you'll see it's a magic comment in go source code: https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80dca4f355b8fb884

If you look at how cgo is linked, you'll see it's a magic comment in go source code: https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80dca4f355b8fb884646370/jack.go#L4 https://github.com/xthexder/go-jack/blob/bc8604043aba0b6af80... If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. Admittedly, you'll usually get compilation errors for undefined references in compiled languages (modulo dlopen), so at least you'll get a build error instead of a runtime error like you do in python. The solution for both Go and Python is for bazel to also control the system dependencies, i.e. to force you to run all code in a container sandbox bazel controls, force you to specify all external dependencies from linux kernel version up to go compiler version, and build all of that itself. In other words, bazel is a lacking implementation of NixOS.

klodolph 2y ago
> If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. Go has an internal build system as well as a compiler. The build system drives the compiler. When you use Bazel with Go, you’re bypassing the Go build system entirely. Bazel directly invokes the Go compiler. This means that any of those comments will also get ignored. (Generally, this is how Bazel works for other languages too. Bazel + Rust does not use Cargo. Bazel + Java does not use, like, Maven or Gradle.) You have to specify the dependencies in your build file. Let's say you have a Go library abc, with a C library xyz. go_library( name = "abc", srcs = [ "abc.go", ], cdeps = [ "//path/to/xyz", ], cgo = True, importpath = "path/to/abc", ) > In other words, bazel is a lacking implementation of NixOS. Bazel and NixOS are both good tools for making reproducible builds. They’re even a good match for each other—grab your compiler from NixPkgs, and build the rest of your project in Bazel. You get the benefit of Bazel’s fine-grained dependencies (Nix has very coarse dependency management), and you can get some specific version of GCC built for both Linux and macOS by tapping into NixPkgs. A marriage made in heaven. (Except for the fact that you are now using two tools which both have very steep learning curves.) There are a lot of interesting similarities between Bazel and Nix, which is not surprising, since they are solving similar problems.
- TheDong 2y ago
  > That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. > This means that any of those comments will also get ignored I cloned this example: https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea61253a62b99a61edb5509f5de/examples/basic-gazelle https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea6... And then ran "go get github.com/xthexder/go-jack" and added an 'import _ "github.com/xthexder/go-jack"' to 'cmd/roll.go'. $ bazel run //:gazelle-update-repos $ bazel run //:basic-gazelle roll __main__/external/com_github_xthexder_go_jack/jack.go:11:10: fatal error: jack/jack.h: No such file or directory compilation terminated. compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2 $ apt-get install libjack-dev $ bazel run //:basic-gazelle roll Number rolled: 80 Sure looks like you're wrong. Bazel obviously didn't ignore those comments because it errored out on trying to find the include. (And of course it didn't ignore them, those comments are preprocessor instructions for the go compiler, which bazel runs under the hood). It obviously didn't need me to specify cdeps because it found the C dependency on my host when I installed it without changing a single bazel-related file. It looks like your understanding might be wrong, sorry.
  - klodolph 2y ago
    You’re making a complaint about Gazelle. If you import third-party packages with c dependencies with Gazelle, obviously, the two choices are it breaks hermeticity or you vendor the C code. Bazel does ignore those comments. Gazelle does not.
    - TheDong 2y ago
      > Bazel does ignore those comments. Gazelle does not. I really don't understand what you're saying. gazelle does not seem to do anything related to the c dependencies there. The full diff gazelle generated is just changing "deps" in 'BUILD.bazel' files to include the library, and adding a "go_repository" to "deps.bzl". Not that it would matter anyway, gazelle is part of bazel. There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle. I'm only using gazelle here because that's the only go example that upstream has in the repo. The "bazel build" part is very clearly the part that is looking for these C files, and doing so in a non-hermetic way. It's possible to make it hermetic, sure, if you pay attention and vendor things or such, but it's obviously not required. That's all I was saying, just like you can have python break hermeticity and depend on external C libraries, you can have go break hermeticity and depend on external c libraries. It's possible to use bazel such that it's hermetic, but it doesn't stop you from doing it wrong. That's all this thread is about, is whether it's also possible to make these non-hermetic references in languages other than python. Like, if you go back and read my commend and your comment a few up, you'll see that my original claim was just that you could impurely reference the host "libjack-dev", which I have very clearly shown you can, and your claim was that it's literally impossible to do that, which it very obviously is possible.

klodolph 2y ago

> If bazel allows impurely using that dependency from the host, like bazel allows for python, then you'll be able to run into similar issues. That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. Go has an internal build system as well as a compiler. The build system drives the compiler. When you use Bazel with Go, you’re bypassing the Go build system entirely. Bazel directly invokes the Go compiler. This means that any of those comments will also get ignored. (Generally, this is how Bazel works for other languages too. Bazel + Rust does not use Cargo. Bazel + Java does not use, like, Maven or Gradle.) You have to specify the dependencies in your build file. Let's say you have a Go library abc, with a C library xyz. go_library( name = "abc", srcs = [ "abc.go", ], cdeps = [ "//path/to/xyz", ], cgo = True, importpath = "path/to/abc", ) > In other words, bazel is a lacking implementation of NixOS. Bazel and NixOS are both good tools for making reproducible builds. They’re even a good match for each other—grab your compiler from NixPkgs, and build the rest of your project in Bazel. You get the benefit of Bazel’s fine-grained dependencies (Nix has very coarse dependency management), and you can get some specific version of GCC built for both Linux and macOS by tapping into NixPkgs. A marriage made in heaven. (Except for the fact that you are now using two tools which both have very steep learning curves.) There are a lot of interesting similarities between Bazel and Nix, which is not surprising, since they are solving similar problems.

TheDong 2y ago
> That’s not how Bazel + CGO works at all. It sounds like you are making some guesses here but those guesses turned out to be wrong, sorry. > This means that any of those comments will also get ignored I cloned this example: https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea61253a62b99a61edb5509f5de/examples/basic-gazelle https://github.com/bazelbuild/rules_go/tree/634fc283f8d84ea6... And then ran "go get github.com/xthexder/go-jack" and added an 'import _ "github.com/xthexder/go-jack"' to 'cmd/roll.go'. $ bazel run //:gazelle-update-repos $ bazel run //:basic-gazelle roll __main__/external/com_github_xthexder_go_jack/jack.go:11:10: fatal error: jack/jack.h: No such file or directory compilation terminated. compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2 $ apt-get install libjack-dev $ bazel run //:basic-gazelle roll Number rolled: 80 Sure looks like you're wrong. Bazel obviously didn't ignore those comments because it errored out on trying to find the include. (And of course it didn't ignore them, those comments are preprocessor instructions for the go compiler, which bazel runs under the hood). It obviously didn't need me to specify cdeps because it found the C dependency on my host when I installed it without changing a single bazel-related file. It looks like your understanding might be wrong, sorry.
- klodolph 2y ago
  You’re making a complaint about Gazelle. If you import third-party packages with c dependencies with Gazelle, obviously, the two choices are it breaks hermeticity or you vendor the C code. Bazel does ignore those comments. Gazelle does not.
  - TheDong 2y ago
    > Bazel does ignore those comments. Gazelle does not. I really don't understand what you're saying. gazelle does not seem to do anything related to the c dependencies there. The full diff gazelle generated is just changing "deps" in 'BUILD.bazel' files to include the library, and adding a "go_repository" to "deps.bzl". Not that it would matter anyway, gazelle is part of bazel. There's no references to c dependencies anywhere in what gazelle produced. I could have hand-written those changes just as easily, without gazelle. I'm only using gazelle here because that's the only go example that upstream has in the repo. The "bazel build" part is very clearly the part that is looking for these C files, and doing so in a non-hermetic way. It's possible to make it hermetic, sure, if you pay attention and vendor things or such, but it's obviously not required. That's all I was saying, just like you can have python break hermeticity and depend on external C libraries, you can have go break hermeticity and depend on external c libraries. It's possible to use bazel such that it's hermetic, but it doesn't stop you from doing it wrong. That's all this thread is about, is whether it's also possible to make these non-hermetic references in languages other than python. Like, if you go back and read my commend and your comment a few up, you'll see that my original claim was just that you could impurely reference the host "libjack-dev", which I have very clearly shown you can, and your claim was that it's literally impossible to do that, which it very obviously is possible.