8 ms·

Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precise

Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precisely specified declaratively. The urge to hack in a little imperative code or jam in a build step that you just "know" needs to happen in a certain order is always there and it can be very difficult to satisfy the taskmaster that is the build system. When you have weird cases like a script that produces a variable set of output files based on the content of some input file, it's tough to get that working in the build system. But... if you climb that cliff, there are many rewards at the top. Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state), you can automatically parallelize (because the build system can tell declaratively which tasks do no depend on each other) and you can ask it interesting queries about the dependency graph. That sounds an awful lot like the runtime safety, performance, and static analysis static types give you. The challenge is that for small programs, dynamic types are just easier to get going with. And every build script starts pretty small. So you have a plethora of simple, hackable build systems like make that work fine at first but then get more and more painful as the program grows. With software, there is at least some customer-visible pain that can enable you to justify a rewrite in a statically-typed language. But build infrastructure is always a hidden cost so it's really hard to get the resources to migrate to a different build language. Maybe the lesson here is to bite the bullet and start with a hermetic build system at first.

adgasf 7y ago
Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just: cc_binary( name = "app", srcs = glob([ "*.hpp", "*.cpp" ]), ) That's more concise than most build systems AND it's cross-platform.
- joshuamorton 7y ago
  Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.
  - adgasf 7y ago
    > bazel works better if you have 1 library per header What do you mean?
    - klipt 7y ago
      You could have a cc_binary for your main foo_main.cc file that depends on cc_library targets separately specified for each bar_lib.h/bar_lib.cc. That makes the graph more granular, e.g. if you just update the foo_main.cc you don't need to recompile the libraries. Or you can reuse bar_lib in a different binary.
      - adgasf 7y ago
        Does Bazel break up these steps automatically under the hood? It could, in theory.
        SolarNet 7y ago
        It does, this is testable a number of ways. The funniest one is to do a `#include "foo.bar"` (and make sure the file exists and is in `srcs`!) and watch bazel freak out with "I have no clue what kind of file that is, how do I analyze that!" Something to be aware of though is that Bazel does do some stupid things around this space. Like adding and removing files causes the build rule to change (if using a glob), which will proc a full rebuild of the library.
      - grandmczeb 7y ago
        > if you just update the foo_main.cc you don't need to recompile the libraries. Are you sure this is required? I just tried with a test repo and Bazel only performed two actions after I updated foo_main.cpp (compile foo_main.cpp and link app.)
        SolarNet 7y ago
        The case in question is .h file include chains. And it does work correctly.
- SolarNet 7y ago
  Yes but there is a hundred megabytes of system behind that. Tens of thousands of lines of starlark code, and tens (maybe hundreds) of thousands of lines of java code besides. A lot of work goes into those 4 lines. This is evidenced by the simple question of, how do you set up a custom compiler for that code properly? The answer being a couple hundred lines of starlark code, for what in other build systems is effectively `CC=clang` (to be fair bazel allows this but does not support or encourage it, especially for anything besides toy builds). I like Bazel (a lot!) but you are oversimplifying it here.
  - HereBeBeasties 7y ago
    I have to ask, "so what?" Provided the abstraction doesn't leak I really don't care, in the same way that end users of Microsoft Word don't care about the algorithms that kern their text, about how DirectWrite does subpixel rendering, or anything else. Yes, it's all fiercely large and complex. The original user is pointing out that this is all hidden for simple projects, waiting in the wings for when you need it, and therefore doesn't matter.
    - SolarNet 7y ago
      Yes but it's minimizing the complexity of the cliff. The original author they were responding to was describing the static/dynamic discrepancy, and how it requires a lot of work. The author I was replying points out how simple the syntax is. But that's missing the point. I provided a direct counter example: Adding a new compiler to bazel is very complicated, and by new compiler I include something as simple as asserting "all of my C++ code is c++17" at a project level is (when done the correct way, and not one of the many hacky prone to failure ways) a 100+ line toolchain definition (because C++17 should be treated as an entirely new compiler from the perspective of bazel; treating it as a command line argument gets you in a bad place where you hit weird errors). To draw an analogy, it's like saying that C++ code like: some_matrix[i, j]; is great syntax and cross platform! When in reality it involves (in user space code) 3 complex classes (at a minimum!), overloading the comma operator(!), implicitly casting from builtin types, and probably hundreds of lines of template code (if you want any sort of extensibility). True it is cross platform and great syntax. But it obfuscates the amount of code and understanding required to do anything but what the most basic syntax allows, for example extending the system in any way.
- flohofwoe 7y ago
  Simple things like this are simple in all C/C++ build systems, the question is how the complex things are handled (multi-platform support, detecting and selecting different versions of the same compiler, cross-compiling, IDE support, multi-stage builds where source files are generated in the build which need to be inputs to later build stages, custom build steps, or bootstrapping tools which need to be used as compilers or code-generators for later build stages, etc etc etc...)
  - adgasf 7y ago
    OP was suggesting that systems like Bazel are harder to get started with but pay off with larger, more complex builds. However, I am claiming that Bazel is also good for the simple cases.
    - cma 7y ago
      It might be that stuff (that is still beginner stuff) a bit beyond the first step is harder. Hello world is simple in python, but it is still hard to get started writing real-time code in it even if hello world might be your first step.
- mzs 7y ago
  I guess but this is also portable and more people know how to extend it: $ cat Makefile SRCS = foo.cpp bar.cpp baz.cpp OBJS = $(SRCS:.cpp=.o) PROG = app EXT = .PHONEY = all clean all : $(PROG) LDLIBS = $(OBJS) $(PROG) : $(PROG:$(EXT)=.cc) $(OBJS) clean : rm -f $(PROG) $(OBJS) $ make SRCS="`echo *.cpp`" clean all rm -f app a.o b.o c.o c++ -O2 -pipe -c a.cpp -o a.o c++ -O2 -pipe -c b.cpp -o b.o c++ -O2 -pipe -c c.cpp -o c.o c++ -O2 -pipe app.cc a.o b.o c.o -o app $
  - pjmlp 7y ago
    If that was the case, autoconf and autotools would have never been a thing. The "beauty" of UNIX falls apart when trying to code across all of them. For example, I cannot even remember if that Makefile would work 100% with a POSIX make.
    - mzs 7y ago
      That Makefile works with BSD and GNU make, no extensions used. Bazel isn't really nicer than the autotools when you need something complicated, if it even supports your platform. Here's a fun bazel bug report: https://github.com/bazelbuild/bazel/issues/4350#issuecomment-353742540 https://github.com/bazelbuild/bazel/issues/4350#issuecomment... I've always managed to avoid autotools unless it was used before with things like LDFLAGS='-lsocket -lns' make ...
      - pjmlp 7y ago
        For me being POSIX compliant goes a little beyond BSD and GNU make.
        mzs 7y ago
        What does posix compliant* even mean for make? The posix standard doesn't even require support for c++! That's a VERY portable Makefile. It would work on any make from the late seventies (assuming you also had a c++ compiler and system Makefiles that knew how to use it). It only uses a small subset of the rules in the standard. * https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html https://pubs.opengroup.org/onlinepubs/009695399/utilities/ma...
- bsder 7y ago
  The syntax is almost never the issue. The issue is that declarative build systems ALWAYS need an escape hatch for the exceptions--and the exceptions grow over time. We've been here before. It was called Make. And then it was called Ant. And then Maven. And Google still built their own system. And they will build another again. Nobody ever learns the lesson that a build system is a program--period. Sorry to burst your bubble: there will be circular dependencies--you will have to deal with it. There will be two different C compilers on two different architectures. You will have to deal with it. Libraries will be in the wrong place--you will have to deal with it. These libraries come from the system but these libraries come from an artifact--you will have to deal with it. The only way to deal with it is to have a genuine programming language underneath.
  - EricBurnett 7y ago
    You say that like bazel hasn't already been proven out on a few billion LoC :). You're quite right that exceptions will exist - your job is to either fix them or squeeze them into the build system. Both are well trod paths. You're quite right that these edge cases aren't _easy_ - but in what system would they be? > there will be circular dependencies--you will have to deal with it. Add them both to the same compilation unit (cc_library, whatever). Or extract an ABI (e.g. Turbine) and compile against that. > There will be two different C compilers on two different architectures. You will have to deal with it. Poke around https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains https://github.com/tensorflow/tensorflow/tree/master/third_p... . I see arm and x86, Windows and many variants of Linux, multiple versions of GCC, multiple versions of clang, multiple versions of cuda, python 2 and 3, and names I don't even recognize. See also https://docs.bazel.build/versions/master/platforms-intro.html https://docs.bazel.build/versions/master/platforms-intro.htm.... > Libraries will be in the wrong place--you will have to deal with it. Just write the path to the file, or write a build rule that makes it visible in a standard location, or make it part of the platform definition, or use something like https://docs.bazel.build/versions/master/be/workspace.html#bind https://docs.bazel.build/versions/master/be/workspace.html#b... to alias it into your workspace. (Not recommended, but supported.) > Libraries will be in the wrong place--you will have to deal with it. Platform-specific library paths are very common. These days it's probably better specified as part of your "platform", but https://docs.bazel.build/versions/master/be/functions.html#select https://docs.bazel.build/versions/master/be/functions.html#s... is a convenient way to make in-place selectors. I won't pretend bazel can express _everything_, but there's little you can't hack in _somehow_ with sufficient motivation, and moving towards the bazel-recommended patterns brings growing peace-of-mind (and faster builds). (Disclaimer: Googler, work closely with the bazel team. I like bazel).
lima 7y ago
> Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state) Note that Bazel builds still depend on the ambient system environment, like the compiler toolchain and system libraries, and it's possible to poison a shared build cache. Shared caches are typically used with remote execution and CI servers, where the build environment is fully controlled.
- adgasf 7y ago
  Doesn't Bazel hash the compiler and env variables to prevent this? The only way to poison the cache then is to break things at an OS level, which is highly unlikely.
  - theamk 7y ago
    Oh, it is pretty simple to poison the cache. Bazel does not really hash any of the system stuff -- like system headers, and system-provided .a and .so files (unless they are explicitly declared in WORKSPACE, but this is optional, and I have never seen this done for libc stuff fot example). So all you need to do is to find one person who likes to mess with their system (for example, by installing latest versions of the libraries from random PPA's), and you'll end up with poisoned cache for everyone.
    - vlovich123 7y ago
      On any large team your compiler should be checked into revision control or have a way to fetch a specific artifact & have that pointer checked in. That way you're guaranteeing everyone is using the right thing.
      - theamk 7y ago
        Are you talking in general, or bazel specifically? Because in general yes, I agree, it would be nice. Probably not in source control though -- putting in compilers + libraries will make checkouts very slow, git is not really designed for tens of gigabytes of stuff. For bazel specifically, this is not really well supported. The default rules and all examples just trust system-installed compilers. There is ongoing work on building in docker (with all the corresponding security issues), and there are rumors that Google internal version has some sort of magic, but this is not a "mainstream" configuration.
jcranmer 7y ago
The problem with fully declarative systems is that, if you need something that is beyond the scope of the declarative semantics, you are screwed. Build systems have the annoying property that they are ultimately highly declarative, but they need some pretty extreme scope to handle edge cases. Any C/C++ project is going to boil down to "compile these N source files, and link them into a library/executable", but the necessary build flags is going to come from several locations, there are multiple variations of build flags and compilers that need to be used at the same time, and there is going to be project-custom code to generate source files that involves potentially building the tools to generate the files as part of the build system. And mixed-language projects add more layers of pain.
- theamk 7y ago
  bazel in particular is pretty nice with that. There are multiple extension points -- there is embedded Python-like language, you can wrap arbitrary commands, and if this is not enough, you can dynamically general build files as well.
bennofs 7y ago
Hermetic build systems make a lot of sense for companies, where you often want to control your whole stack and might even use a monorepo with all external dependencies vendored. However, there are at least two cases where you do not want real hermetic builds: - in the case of open source software that needs to be packaged for distributions, vendoring all the dependencies is bad. As the name already says, distributions remix software components and thus might want to compile your software against a different version of a library than the particular version you pick. - for libraries, consumers use your library just as a component and might want to combine it with a specific version of other libraries, so vendoring also sucks. Unfortunately, many declarative build systems make it hard to separate the "building a component" part from the "composing a product" part. I could find no way in Bazel to depend on system software (using for example pkgconfig). In my opinion, build systems for open source software (as I mentioned, for companies Bazel makes sense since you often want to vendor all your deps anyway) need to have three aspects: - build rules for just for this component - dependencies on other components/libraries - a method to lock down components/libraries to a specific version, creating an offical "product" with an official supported combination of all dependencies. This still makes it possible for distributions to create their own "product", without having to rewrite most parts of the build system. At the same time, developers can work with the official library versions (for compat, maybe you should even have multiple "sets" of dependencies), so getting started is easy and you don't need to install lots of dependencies first before you can hack on the project. Often, build systems do things that are nice for onboarding (like "build systems" that pull git repos of dependencies during build, in order to have a single command to build the product from scratch after checkout). But those things directly conflict with the needs of distributions. We need to separate the "instructions to build, supplying all deps externally explictly" (in the context of building a package for a distribution) from the "just build it from the command line on a dev machine" (here, dependencies should be implictly fetched automatically, to give a nice user experience).
- codesuki 7y ago
  You could probably write a rule similar to go_repository[1] that would invoke pkgconfig to find the location of local libs instead of downloading a lib. [1] https://github.com/bazelbuild/bazel-gazelle/blob/master/internal/go_repository.bzl https://github.com/bazelbuild/bazel-gazelle/blob/master/inte...

adgasf 7y ago

Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just: cc_binary( name = "app", srcs = glob([ "*.hpp", "*.cpp" ]), ) That's more concise than most build systems AND it's cross-platform.

joshuamorton 7y ago
Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.
- adgasf 7y ago
  > bazel works better if you have 1 library per header What do you mean?
  - klipt 7y ago
    You could have a cc_binary for your main foo_main.cc file that depends on cc_library targets separately specified for each bar_lib.h/bar_lib.cc. That makes the graph more granular, e.g. if you just update the foo_main.cc you don't need to recompile the libraries. Or you can reuse bar_lib in a different binary.
    - adgasf 7y ago
      Does Bazel break up these steps automatically under the hood? It could, in theory.
      - SolarNet 7y ago
        It does, this is testable a number of ways. The funniest one is to do a `#include "foo.bar"` (and make sure the file exists and is in `srcs`!) and watch bazel freak out with "I have no clue what kind of file that is, how do I analyze that!" Something to be aware of though is that Bazel does do some stupid things around this space. Like adding and removing files causes the build rule to change (if using a glob), which will proc a full rebuild of the library.
    - grandmczeb 7y ago
      > if you just update the foo_main.cc you don't need to recompile the libraries. Are you sure this is required? I just tried with a test repo and Bazel only performed two actions after I updated foo_main.cpp (compile foo_main.cpp and link app.)
      - SolarNet 7y ago
        The case in question is .h file include chains. And it does work correctly.
SolarNet 7y ago
Yes but there is a hundred megabytes of system behind that. Tens of thousands of lines of starlark code, and tens (maybe hundreds) of thousands of lines of java code besides. A lot of work goes into those 4 lines. This is evidenced by the simple question of, how do you set up a custom compiler for that code properly? The answer being a couple hundred lines of starlark code, for what in other build systems is effectively `CC=clang` (to be fair bazel allows this but does not support or encourage it, especially for anything besides toy builds). I like Bazel (a lot!) but you are oversimplifying it here.
- HereBeBeasties 7y ago
  I have to ask, "so what?" Provided the abstraction doesn't leak I really don't care, in the same way that end users of Microsoft Word don't care about the algorithms that kern their text, about how DirectWrite does subpixel rendering, or anything else. Yes, it's all fiercely large and complex. The original user is pointing out that this is all hidden for simple projects, waiting in the wings for when you need it, and therefore doesn't matter.
  - SolarNet 7y ago
    Yes but it's minimizing the complexity of the cliff. The original author they were responding to was describing the static/dynamic discrepancy, and how it requires a lot of work. The author I was replying points out how simple the syntax is. But that's missing the point. I provided a direct counter example: Adding a new compiler to bazel is very complicated, and by new compiler I include something as simple as asserting "all of my C++ code is c++17" at a project level is (when done the correct way, and not one of the many hacky prone to failure ways) a 100+ line toolchain definition (because C++17 should be treated as an entirely new compiler from the perspective of bazel; treating it as a command line argument gets you in a bad place where you hit weird errors). To draw an analogy, it's like saying that C++ code like: some_matrix[i, j]; is great syntax and cross platform! When in reality it involves (in user space code) 3 complex classes (at a minimum!), overloading the comma operator(!), implicitly casting from builtin types, and probably hundreds of lines of template code (if you want any sort of extensibility). True it is cross platform and great syntax. But it obfuscates the amount of code and understanding required to do anything but what the most basic syntax allows, for example extending the system in any way.
flohofwoe 7y ago
Simple things like this are simple in all C/C++ build systems, the question is how the complex things are handled (multi-platform support, detecting and selecting different versions of the same compiler, cross-compiling, IDE support, multi-stage builds where source files are generated in the build which need to be inputs to later build stages, custom build steps, or bootstrapping tools which need to be used as compilers or code-generators for later build stages, etc etc etc...)
- adgasf 7y ago
  OP was suggesting that systems like Bazel are harder to get started with but pay off with larger, more complex builds. However, I am claiming that Bazel is also good for the simple cases.
  - cma 7y ago
    It might be that stuff (that is still beginner stuff) a bit beyond the first step is harder. Hello world is simple in python, but it is still hard to get started writing real-time code in it even if hello world might be your first step.
mzs 7y ago
I guess but this is also portable and more people know how to extend it: $ cat Makefile SRCS = foo.cpp bar.cpp baz.cpp OBJS = $(SRCS:.cpp=.o) PROG = app EXT = .PHONEY = all clean all : $(PROG) LDLIBS = $(OBJS) $(PROG) : $(PROG:$(EXT)=.cc) $(OBJS) clean : rm -f $(PROG) $(OBJS) $ make SRCS="`echo *.cpp`" clean all rm -f app a.o b.o c.o c++ -O2 -pipe -c a.cpp -o a.o c++ -O2 -pipe -c b.cpp -o b.o c++ -O2 -pipe -c c.cpp -o c.o c++ -O2 -pipe app.cc a.o b.o c.o -o app $
- pjmlp 7y ago
  If that was the case, autoconf and autotools would have never been a thing. The "beauty" of UNIX falls apart when trying to code across all of them. For example, I cannot even remember if that Makefile would work 100% with a POSIX make.
  - mzs 7y ago
    That Makefile works with BSD and GNU make, no extensions used. Bazel isn't really nicer than the autotools when you need something complicated, if it even supports your platform. Here's a fun bazel bug report: https://github.com/bazelbuild/bazel/issues/4350#issuecomment-353742540 https://github.com/bazelbuild/bazel/issues/4350#issuecomment... I've always managed to avoid autotools unless it was used before with things like LDFLAGS='-lsocket -lns' make ...
    - pjmlp 7y ago
      For me being POSIX compliant goes a little beyond BSD and GNU make.
      - mzs 7y ago
        What does posix compliant* even mean for make? The posix standard doesn't even require support for c++! That's a VERY portable Makefile. It would work on any make from the late seventies (assuming you also had a c++ compiler and system Makefiles that knew how to use it). It only uses a small subset of the rules in the standard. * https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html https://pubs.opengroup.org/onlinepubs/009695399/utilities/ma...
bsder 7y ago
The syntax is almost never the issue. The issue is that declarative build systems ALWAYS need an escape hatch for the exceptions--and the exceptions grow over time. We've been here before. It was called Make. And then it was called Ant. And then Maven. And Google still built their own system. And they will build another again. Nobody ever learns the lesson that a build system is a program--period. Sorry to burst your bubble: there will be circular dependencies--you will have to deal with it. There will be two different C compilers on two different architectures. You will have to deal with it. Libraries will be in the wrong place--you will have to deal with it. These libraries come from the system but these libraries come from an artifact--you will have to deal with it. The only way to deal with it is to have a genuine programming language underneath.
- EricBurnett 7y ago
  You say that like bazel hasn't already been proven out on a few billion LoC :). You're quite right that exceptions will exist - your job is to either fix them or squeeze them into the build system. Both are well trod paths. You're quite right that these edge cases aren't _easy_ - but in what system would they be? > there will be circular dependencies--you will have to deal with it. Add them both to the same compilation unit (cc_library, whatever). Or extract an ABI (e.g. Turbine) and compile against that. > There will be two different C compilers on two different architectures. You will have to deal with it. Poke around https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains https://github.com/tensorflow/tensorflow/tree/master/third_p... . I see arm and x86, Windows and many variants of Linux, multiple versions of GCC, multiple versions of clang, multiple versions of cuda, python 2 and 3, and names I don't even recognize. See also https://docs.bazel.build/versions/master/platforms-intro.html https://docs.bazel.build/versions/master/platforms-intro.htm.... > Libraries will be in the wrong place--you will have to deal with it. Just write the path to the file, or write a build rule that makes it visible in a standard location, or make it part of the platform definition, or use something like https://docs.bazel.build/versions/master/be/workspace.html#bind https://docs.bazel.build/versions/master/be/workspace.html#b... to alias it into your workspace. (Not recommended, but supported.) > Libraries will be in the wrong place--you will have to deal with it. Platform-specific library paths are very common. These days it's probably better specified as part of your "platform", but https://docs.bazel.build/versions/master/be/functions.html#select https://docs.bazel.build/versions/master/be/functions.html#s... is a convenient way to make in-place selectors. I won't pretend bazel can express _everything_, but there's little you can't hack in _somehow_ with sufficient motivation, and moving towards the bazel-recommended patterns brings growing peace-of-mind (and faster builds). (Disclaimer: Googler, work closely with the bazel team. I like bazel).

lima 7y ago

> Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state) Note that Bazel builds still depend on the ambient system environment, like the compiler toolchain and system libraries, and it's possible to poison a shared build cache. Shared caches are typically used with remote execution and CI servers, where the build environment is fully controlled.

adgasf 7y ago
Doesn't Bazel hash the compiler and env variables to prevent this? The only way to poison the cache then is to break things at an OS level, which is highly unlikely.
- theamk 7y ago
  Oh, it is pretty simple to poison the cache. Bazel does not really hash any of the system stuff -- like system headers, and system-provided .a and .so files (unless they are explicitly declared in WORKSPACE, but this is optional, and I have never seen this done for libc stuff fot example). So all you need to do is to find one person who likes to mess with their system (for example, by installing latest versions of the libraries from random PPA's), and you'll end up with poisoned cache for everyone.
  - vlovich123 7y ago
    On any large team your compiler should be checked into revision control or have a way to fetch a specific artifact & have that pointer checked in. That way you're guaranteeing everyone is using the right thing.
    - theamk 7y ago
      Are you talking in general, or bazel specifically? Because in general yes, I agree, it would be nice. Probably not in source control though -- putting in compilers + libraries will make checkouts very slow, git is not really designed for tens of gigabytes of stuff. For bazel specifically, this is not really well supported. The default rules and all examples just trust system-installed compilers. There is ongoing work on building in docker (with all the corresponding security issues), and there are rumors that Google internal version has some sort of magic, but this is not a "mainstream" configuration.

jcranmer 7y ago

The problem with fully declarative systems is that, if you need something that is beyond the scope of the declarative semantics, you are screwed. Build systems have the annoying property that they are ultimately highly declarative, but they need some pretty extreme scope to handle edge cases. Any C/C++ project is going to boil down to "compile these N source files, and link them into a library/executable", but the necessary build flags is going to come from several locations, there are multiple variations of build flags and compilers that need to be used at the same time, and there is going to be project-custom code to generate source files that involves potentially building the tools to generate the files as part of the build system. And mixed-language projects add more layers of pain.

theamk 7y ago
bazel in particular is pretty nice with that. There are multiple extension points -- there is embedded Python-like language, you can wrap arbitrary commands, and if this is not enough, you can dynamically general build files as well.

bennofs 7y ago

Hermetic build systems make a lot of sense for companies, where you often want to control your whole stack and might even use a monorepo with all external dependencies vendored. However, there are at least two cases where you do not want real hermetic builds: - in the case of open source software that needs to be packaged for distributions, vendoring all the dependencies is bad. As the name already says, distributions remix software components and thus might want to compile your software against a different version of a library than the particular version you pick. - for libraries, consumers use your library just as a component and might want to combine it with a specific version of other libraries, so vendoring also sucks. Unfortunately, many declarative build systems make it hard to separate the "building a component" part from the "composing a product" part. I could find no way in Bazel to depend on system software (using for example pkgconfig). In my opinion, build systems for open source software (as I mentioned, for companies Bazel makes sense since you often want to vendor all your deps anyway) need to have three aspects: - build rules for just for this component - dependencies on other components/libraries - a method to lock down components/libraries to a specific version, creating an offical "product" with an official supported combination of all dependencies. This still makes it possible for distributions to create their own "product", without having to rewrite most parts of the build system. At the same time, developers can work with the official library versions (for compat, maybe you should even have multiple "sets" of dependencies), so getting started is easy and you don't need to install lots of dependencies first before you can hack on the project. Often, build systems do things that are nice for onboarding (like "build systems" that pull git repos of dependencies during build, in order to have a single command to build the product from scratch after checkout). But those things directly conflict with the needs of distributions. We need to separate the "instructions to build, supplying all deps externally explictly" (in the context of building a package for a distribution) from the "just build it from the command line on a dev machine" (here, dependencies should be implictly fetched automatically, to give a nice user experience).

codesuki 7y ago
You could probably write a rule similar to go_repository[1] that would invoke pkgconfig to find the location of local libs instead of downloading a lib. [1] https://github.com/bazelbuild/bazel-gazelle/blob/master/internal/go_repository.bzl https://github.com/bazelbuild/bazel-gazelle/blob/master/inte...