5 ms·

By the sake of god, please try to add also a more friendly build system too. Bazel is a disaster that would almost make maven looks lightweight, npm reliable an

by adev_ 8y ago

By the sake of god, please try to add also a more friendly build system too. Bazel is a disaster that would almost make maven looks lightweight, npm reliable and autotools userfriendly.

Judgmentality 8y ago
What do you dislike about bazel? Not championing it, just curious.
- adev_ 8y ago
  The list would be too long but out of order : - Using a JVM to compile mainly C++ and python, implying to deploy a JDK/JRE just for that. - Awful RAM consumption, compiling tensorflow takes 16GB of RAM to create a python wheel - Try to download the world and compile everything without allowing to specify external dependencies. - Compile time that are crazy long due to previous reason. - Invasive. Make very hard to integrate with anything else that does not build with Bazel - Make very hard, or sometimes, impossible to tune compiler flags - Just not reproducible. - Unstable, try to build a recent Google package with a 6 month old Bazel and good luck. I strongly advise you to watch this video of the last FOSDEM : https://archive.fosdem.org/2018/schedule/event/how_to_make_package_managers_cry/ https://archive.fosdem.org/2018/schedule/event/how_to_make_p...
  - haberman 8y ago
    I strongly agree with you on the first one. Many of your other ones are byproducts of the fact that Bazel is primarily a build-from-source system. This has some benefits, particularly in a C++ ecosystem where binary compatibility across versions basically doesn't exist. But it also has some big drawbacks when it comes to compile times. I do see Bazel seems to support depending on a prebuilt .so, though I have not tried this: https://docs.bazel.build/versions/master/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries https://docs.bazel.build/versions/master/cpp-use-cases.html#... > Invasive. Make very hard to integrate with anything else that does not build with Bazel I think your main options here are: 1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries https://docs.bazel.build/versions/master/cpp-use-cases.html#...) 2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/master/bazel/third_party https://github.com/googlecartographer/cartographer/tree/mast... > Make very hard, or sometimes, impossible to tune compiler flags "bazel --copt=<your options here> :target"? > Just not reproducible. What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there. Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need.
    - adev_ 8y ago
      > Many of your other ones are byproducts of the fact that Bazel is primarily a build-from-source system. This has some benefits, particularly in a C++ ecosystem where binary compatibility across versions basically doesn't exist. But it also has some big drawbacks when it comes to compile times. Nix, Guix and Spack packager managers solved the C++ ABI issue a long time ago already without the crazy needs of Bazel in term of resource consumption, integration and compile time. They even supports binary distributions for some of them. > I think your main options here are: > 1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#... https://docs.bazel.build/versions/master/cpp-use-cases.html#...) > 2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/mast.. https://github.com/googlecartographer/cartographer/tree/mast.... I know that. But all of them are terrible options. I do not want to depend of SQLite, OpenSSL, libxml or whatever other system library compiled by Bazel, nor I want Bazel to takes 45 minutes to recompile them. Additionally, this will cause diamond dependency problem with other softwares that use Bazel artefact without compiling with Bazel. > "bazel --copt=<your options here> :target"? Can I use that to specify a flag to some target and not some other ? Without having to build sequentially each of them ? Concrete example of the Maddness: SQLITE will not compile if you enable some options that would make tensorflow faster.... Bazel recursively compile both. > What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there. Bazel try to build in isolated environment but do half of the job. It still depends on system compiler, and do not chroot nor "compiler-wrap" ( c.f Spack ) making the build still very vulnerable of system side effect and update. > Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need. I can understand that Bazel is very convenient in Google environment with Google resources. But it's a nightmare for everyone I talked to outside of Google.
      - haberman 8y ago
- Q6T46nT668w6i3m 8y ago
  I’m not the OP (but I share their annoyance), I use a dozen of build systems but TensorFlow is the only project I use that uses Bazel.
  - solomatov 8y ago
    Having experience with bazel, I wouldn't say that tensorflow is the best possible use of bazel. You can use of it in other project, for example, in the bazel itself: https://github.com/bazelbuild/bazel https://github.com/bazelbuild/bazel
solomatov 8y ago
Bazel is the best system which I have ever used.
- adev_ 8y ago
  Lucky you, I never met a user outside of Google that like it.
  - solomatov 8y ago
    I am not a Google employee. There're several reasons why it worked so well for us: - It can take advantage of many cores which you have in your computers. - It's polyglot. We have several languages in our project, and mixing several build systems were a nightmare. - It's correct. Very rarely you really need to run clean builds. It has its learning curve, but once you got it, you start to love it.
    - curiousgal 8y ago
      >it can take advantage of many cores which you have in your computers. Which is a bad thing when you're on a shared server. I remember it wasn't straightforward to limit the number of cores it utilizes and it even ignored the flag I passed it. I eventually resorted to waiting until the weekend when no one in the lab was running jobs.
      - maccam94 8y ago
        Ouch. For a workaround in the future, maybe you can lower the priority of your bazel server via nice/renice?
      - grandmczeb 8y ago
        Does “--jobs N” not work?
        jmmv 8y ago
        Unfortunately not. "--jobs N" limits the parallelism during the execution phase (when Bazel executes compilers, etc.) but doesn't limit the parallelism of the analysis phase (when Bazel loads BUILD files and constructs the build graph). For the latter there is a "--loading_phase_threads N" flag. We are actively trying to make Bazel better use the local resources, and also clean up the current mess that the flags are in this area. See the discussion at https://groups.google.com/d/msg/bazel-dev/7pcrJY6Xo78/JDe256-tBQAJ https://groups.google.com/d/msg/bazel-dev/7pcrJY6Xo78/JDe256... for some context.

by adev_ 8y ago

By the sake of god, please try to add also a more friendly build system too. Bazel is a disaster that would almost make maven looks lightweight, npm reliable and autotools userfriendly.

Judgmentality 8y ago

What do you dislike about bazel? Not championing it, just curious.

adev_ 8y ago
The list would be too long but out of order : - Using a JVM to compile mainly C++ and python, implying to deploy a JDK/JRE just for that. - Awful RAM consumption, compiling tensorflow takes 16GB of RAM to create a python wheel - Try to download the world and compile everything without allowing to specify external dependencies. - Compile time that are crazy long due to previous reason. - Invasive. Make very hard to integrate with anything else that does not build with Bazel - Make very hard, or sometimes, impossible to tune compiler flags - Just not reproducible. - Unstable, try to build a recent Google package with a 6 month old Bazel and good luck. I strongly advise you to watch this video of the last FOSDEM : https://archive.fosdem.org/2018/schedule/event/how_to_make_package_managers_cry/ https://archive.fosdem.org/2018/schedule/event/how_to_make_p...
- haberman 8y ago
  I strongly agree with you on the first one. Many of your other ones are byproducts of the fact that Bazel is primarily a build-from-source system. This has some benefits, particularly in a C++ ecosystem where binary compatibility across versions basically doesn't exist. But it also has some big drawbacks when it comes to compile times. I do see Bazel seems to support depending on a prebuilt .so, though I have not tried this: https://docs.bazel.build/versions/master/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries https://docs.bazel.build/versions/master/cpp-use-cases.html#... > Invasive. Make very hard to integrate with anything else that does not build with Bazel I think your main options here are: 1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries https://docs.bazel.build/versions/master/cpp-use-cases.html#...) 2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/master/bazel/third_party https://github.com/googlecartographer/cartographer/tree/mast... > Make very hard, or sometimes, impossible to tune compiler flags "bazel --copt=<your options here> :target"? > Just not reproducible. What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there. Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need.
  - adev_ 8y ago
    > Many of your other ones are byproducts of the fact that Bazel is primarily a build-from-source system. This has some benefits, particularly in a C++ ecosystem where binary compatibility across versions basically doesn't exist. But it also has some big drawbacks when it comes to compile times. Nix, Guix and Spack packager managers solved the C++ ABI issue a long time ago already without the crazy needs of Bazel in term of resource consumption, integration and compile time. They even supports binary distributions for some of them. > I think your main options here are: > 1. prebuild the other projects, then depend on the .so from Bazel (https://docs.bazel.build/versions/master/cpp-use-cases.html#... https://docs.bazel.build/versions/master/cpp-use-cases.html#...) > 2. write a BUILD file for the external project. Here is an example of a project that builds a bunch of non-Bazel deps by writing Bazel BUILD files for each of them: https://github.com/googlecartographer/cartographer/tree/mast.. https://github.com/googlecartographer/cartographer/tree/mast.... I know that. But all of them are terrible options. I do not want to depend of SQLite, OpenSSL, libxml or whatever other system library compiled by Bazel, nor I want Bazel to takes 45 minutes to recompile them. Additionally, this will cause diamond dependency problem with other softwares that use Bazel artefact without compiling with Bazel. > "bazel --copt=<your options here> :target"? Can I use that to specify a flag to some target and not some other ? Without having to build sequentially each of them ? Concrete example of the Maddness: SQLITE will not compile if you enable some options that would make tensorflow faster.... Bazel recursively compile both. > What isn't reproducible? I tend to think of reproducibility as a strength of Bazel. Because all of your dependencies are explicit and Bazel fetches a known version, the build is less dependent on your system environment and whatever you happen to have installed there. Bazel try to build in isolated environment but do half of the job. It still depends on system compiler, and do not chroot nor "compiler-wrap" ( c.f Spack ) making the build still very vulnerable of system side effect and update. > Disclosure: I am a Googler. I have some gripes with Bazel, but overall I think it gets some important ideas right. You have a BUILD file that is declarative, then any imperative code you need goes into separate .bzl files to define the rules you need. I can understand that Bazel is very convenient in Google environment with Google resources. But it's a nightmare for everyone I talked to outside of Google.
    - haberman 8y ago
Q6T46nT668w6i3m 8y ago
I’m not the OP (but I share their annoyance), I use a dozen of build systems but TensorFlow is the only project I use that uses Bazel.
- solomatov 8y ago
  Having experience with bazel, I wouldn't say that tensorflow is the best possible use of bazel. You can use of it in other project, for example, in the bazel itself: https://github.com/bazelbuild/bazel https://github.com/bazelbuild/bazel

solomatov 8y ago

Bazel is the best system which I have ever used.

adev_ 8y ago
Lucky you, I never met a user outside of Google that like it.
- solomatov 8y ago
  I am not a Google employee. There're several reasons why it worked so well for us: - It can take advantage of many cores which you have in your computers. - It's polyglot. We have several languages in our project, and mixing several build systems were a nightmare. - It's correct. Very rarely you really need to run clean builds. It has its learning curve, but once you got it, you start to love it.
  - curiousgal 8y ago
    >it can take advantage of many cores which you have in your computers. Which is a bad thing when you're on a shared server. I remember it wasn't straightforward to limit the number of cores it utilizes and it even ignored the flag I passed it. I eventually resorted to waiting until the weekend when no one in the lab was running jobs.
    - maccam94 8y ago
      Ouch. For a workaround in the future, maybe you can lower the priority of your bazel server via nice/renice?
    - grandmczeb 8y ago
      Does “--jobs N” not work?
      - jmmv 8y ago
        Unfortunately not. "--jobs N" limits the parallelism during the execution phase (when Bazel executes compilers, etc.) but doesn't limit the parallelism of the analysis phase (when Bazel loads BUILD files and constructs the build graph). For the latter there is a "--loading_phase_threads N" flag. We are actively trying to make Bazel better use the local resources, and also clean up the current mess that the flags are in this area. See the discussion at https://groups.google.com/d/msg/bazel-dev/7pcrJY6Xo78/JDe256-tBQAJ https://groups.google.com/d/msg/bazel-dev/7pcrJY6Xo78/JDe256... for some context.