7 ms·

Bazel is not just a build system though. It’s also dependency management system, parallel test system, cross language build system, cross platform build system,

by 0xbear 9y ago

Bazel is not just a build system though. It’s also dependency management system, parallel test system, cross language build system, cross platform build system, and a bunch of other things, with build file syntax a 10 year old could learn in an hour. Some of those things you could plausibly do with CMake, but a lot more effort. Once you learn how to use it (which doesn’t take long) Bazel is pretty darn nice.

mikepurvis 9y ago
Nonetheless, Bazel is very much set up for Google's mentality of building giant static binaries from head and shipping them off to a server somewhere, where they'll be replaced in two weeks with the next release, and then never thought of again. There's some software out here in the world that works like that, but most of it doesn't. In any case, CMake is _also_ a dependency management system, but works with libraries in the way that they are shipped on a typical Linux distribution. It lets you find common stuff easily, and has multiple extension points in terms of adding support for new libraries. CMake itself can ship your find module, the library you're linking against can ship it, or you can ship it yourself.
- justicezyx 9y ago
  Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries There are other means to share build artifacts or code across language boundary, not necessarily through availability from OS distros. For example, Docker registry solves this issue with container images and overlay FS. I think one way or another, Google's model has been proven to work for large-scale (probably the largest C++ code base on earth) c++ code base. In that I have my full confidence of adopting Google's proven approach. disclaimer: Work at google, but not the c++ team
  - vvanders 9y ago
    > Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries Maybe in the cloud, however that's only a portion of where you see C++ deployed. For embedded systems(where C & Rust are the only other options) I don't see it going this way anytime soon.
    - jopsen 9y ago
      Repeatable builds are pretty universally desirable, why wouldn't you want it for embedded systems? It seems pretty important for security, etc.. Note: Rust and cargo.lock files another example of things going in this direction.
    - jsolson 9y ago
      Aren't embedded systems already there? What is a firmware blob if not a monolithic static build artifact? Bazel is perfect for this. I recently taught it how to build boot images for Xilinx FPGAs including a bitstream, first stage bootloader, kernel, and initial ramdisk, all hermetically built and bottled up into a boot.bin. I can reproduce my firmware from source control and get out the same bits. How neat is that?
      - 0xbear 9y ago
        Speaking of which, any possibility of open sourcing this? We are looking to get into FPGAs and would rather not reinvent a bicycle.
        jsolson 9y ago
        Yes. It's a little complicated as I did it for a personal project, but I'm also pulling it into Google for my 20% project. It is by FAR easier/faster for me to release under Google's copyright, but there's a bit of "paperwork" I need to go through to get it staged and approved for the Google GitHub. Since it seems there's interest I'll take a stab at that on Monday. It usually takes a week or two, but no promises. I don't anticipate any blocking issues, though, as the bazel bits don't expose anything about Google. One caveat -- as released it won't quite be bitwise hermetic, as it depends on an external Xilinx toolchain, and Xilinx does non-hermetic things to their bits that are awkward to work around (you can, but my code to do that is a much less mature shell script full of dd and cat commands to rip apart bitstreams and glue them back together -- I've since found open source tooling for accomplishing the same task, but haven't adopted it yet). Drop me an email (addr in profile) and I'll tell you when it's up.
  - flukus 9y ago
    > Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries Isn't this the reason behind the equifax breach? From what I understand (not a java dev) Apache struts is deployed along side the application, so the patch couldn't be applied until someone manually updated the dependencies and deployed a new version of the app. If this was a "traditional" shared library on a distro then it would have been patched by the system administrators. When you bundle dependencies via any method you are also bundling the exploits.
    - DannyBee 9y ago
      traditional shared libraries never have good enough interfaces to not break stuff. So tradtitionally, the shared library isn't updated anyway until the app is fixed and deployed for the new version of the shared lib. Bundled dependencies are easy to handle from a security perspective. Build stamping tells you what CLs are contained (or just go by monotonically incrementing number if it's bundled deps). Someone commits the CL that fixes the dependency. In any sane bundled system, this increments that number. You say that any binary in prod must be something past that CL (which can be verified using the build stamping). Done. If it's running, it's fixed. Modify to have whatever grace period you want. The same works for shared dependencies, of course. In practice, both shared dependencies and static dependencies have tradeoffs, but security is not one of them. Both can be and should be easy to make secure in the sense of "make sure it contains the fix". Both may require app redeployment in practice, and expecting them not to is not a good strategy right now (for most programming languages)
      - flukus 9y ago
        > traditional shared libraries never have good enough interfaces to not break stuff. So tradtitionally, the shared library isn't updated anyway until the app is fixed and deployed for the new version of the shared lib. I agree and disagree. Traditionally a lot of shared libraries are in the OS and quite good at not breaking things, you can run windows update or apt-get upgrade and apps shouldn't break. Breakages generally happen in higher level libraries, but that is a cultural issue more than a technical one. If more care is put into the interface and managing breaking changes then it is doable. But regardless of how libraries are being deployed "move fast and break things" is not an attitude I want from the developers of them. Most language specific package managers I've used do a terrible job of telling developers which packages are obsolete and no longer supported or of what the ramifications for updating a package is. Most libraries in these repositories don't even support previous versions. > Bundled dependencies are easy to handle from a security perspective. But not from a user or corporate perspective. I shudder to think at how many old and unpatched libraries are currently in use in the corporate world where the have an attitude of "deploy, then don't touch until we request a feature". It's an uphill battle to even upgrade when there are no compatibility issues, if they need development time to update then they're likely to stick to an old version forever. Users are unlikely to upgrade as well for various reasons, so if a shared library is bundled with an app it is unlikely to be patched.
  - jonstewart 9y ago
    However large and robust the deployment may be, it's still a narrow use case/environment compared. It may be very well outside the norm of C++ use cases, frankly. A lotta people make Windows apps still.
krona 9y ago
> Once you learn how to use it (which doesn’t take long) Bazel is pretty darn nice. I've spent a few hundred hours integrating bits of CMake build scripts in to various Bazel builds. If you're doing something which is explicitly catered for by Bazel itself then it works well, so long as you avoid the bugs. Otherwise, you'll have to confront the hermetic build abstractions which prevent otherwise simple operations. Bazel toolchain files are just as bad as CMake, if not worse. I'll give Bazel another try in a few years. For now, I still prefer CMake+Ninja, worts and all.
- 0xbear 9y ago
  Hermetic is good though. Hermetic means reproducible.
  - krona 9y ago
    Use Docker?
    - 0xbear 9y ago
      Bazel is on its way to be more like Google’s internal implementation: distributed builds, object file cache, distributed tests. That means your C++ build and test runs will take barely any time at all, same as they do at Google. That, in turn, enables the kind of velocity unachievable through other means.
      - dekhn 9y ago
        I'm sorry but, "bwahahahha". C++ build and test times at Google are astronomical. It's one of the reasons go exists: they built it to compile and link faster.
        0xbear 9y ago
        If a minute to build and link half a million lines of C++ is “astronomical”, then sure. Try to build that on your local box, then get back to me. Go exists solely because Rob Pike had nothing better to do, and a lot of googlers hate Java.

by 0xbear 9y ago

mikepurvis 9y ago

Nonetheless, Bazel is very much set up for Google's mentality of building giant static binaries from head and shipping them off to a server somewhere, where they'll be replaced in two weeks with the next release, and then never thought of again. There's some software out here in the world that works like that, but most of it doesn't. In any case, CMake is _also_ a dependency management system, but works with libraries in the way that they are shipped on a typical Linux distribution. It lets you find common stuff easily, and has multiple extension points in terms of adding support for new libraries. CMake itself can ship your find module, the library you're linking against can ship it, or you can ship it yourself.

justicezyx 9y ago
Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries There are other means to share build artifacts or code across language boundary, not necessarily through availability from OS distros. For example, Docker registry solves this issue with container images and overlay FS. I think one way or another, Google's model has been proven to work for large-scale (probably the largest C++ code base on earth) c++ code base. In that I have my full confidence of adopting Google's proven approach. disclaimer: Work at google, but not the c++ team
- vvanders 9y ago
  > Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries Maybe in the cloud, however that's only a portion of where you see C++ deployed. For embedded systems(where C & Rust are the only other options) I don't see it going this way anytime soon.
  - jopsen 9y ago
    Repeatable builds are pretty universally desirable, why wouldn't you want it for embedded systems? It seems pretty important for security, etc.. Note: Rust and cargo.lock files another example of things going in this direction.
  - jsolson 9y ago
    Aren't embedded systems already there? What is a firmware blob if not a monolithic static build artifact? Bazel is perfect for this. I recently taught it how to build boot images for Xilinx FPGAs including a bitstream, first stage bootloader, kernel, and initial ramdisk, all hermetically built and bottled up into a boot.bin. I can reproduce my firmware from source control and get out the same bits. How neat is that?
    - 0xbear 9y ago
      Speaking of which, any possibility of open sourcing this? We are looking to get into FPGAs and would rather not reinvent a bicycle.
      - jsolson 9y ago
        Yes. It's a little complicated as I did it for a personal project, but I'm also pulling it into Google for my 20% project. It is by FAR easier/faster for me to release under Google's copyright, but there's a bit of "paperwork" I need to go through to get it staged and approved for the Google GitHub. Since it seems there's interest I'll take a stab at that on Monday. It usually takes a week or two, but no promises. I don't anticipate any blocking issues, though, as the bazel bits don't expose anything about Google. One caveat -- as released it won't quite be bitwise hermetic, as it depends on an external Xilinx toolchain, and Xilinx does non-hermetic things to their bits that are awkward to work around (you can, but my code to do that is a much less mature shell script full of dd and cat commands to rip apart bitstreams and glue them back together -- I've since found open source tooling for accomplishing the same task, but haven't adopted it yet). Drop me an email (addr in profile) and I'll tell you when it's up.
- flukus 9y ago
  > Trend wise, applications are expanding more and more into controlling its whole life-cycle from binary to deployment. Repeatable build Containers VM Unikernels Standardized libraries Isn't this the reason behind the equifax breach? From what I understand (not a java dev) Apache struts is deployed along side the application, so the patch couldn't be applied until someone manually updated the dependencies and deployed a new version of the app. If this was a "traditional" shared library on a distro then it would have been patched by the system administrators. When you bundle dependencies via any method you are also bundling the exploits.
  - DannyBee 9y ago
    traditional shared libraries never have good enough interfaces to not break stuff. So tradtitionally, the shared library isn't updated anyway until the app is fixed and deployed for the new version of the shared lib. Bundled dependencies are easy to handle from a security perspective. Build stamping tells you what CLs are contained (or just go by monotonically incrementing number if it's bundled deps). Someone commits the CL that fixes the dependency. In any sane bundled system, this increments that number. You say that any binary in prod must be something past that CL (which can be verified using the build stamping). Done. If it's running, it's fixed. Modify to have whatever grace period you want. The same works for shared dependencies, of course. In practice, both shared dependencies and static dependencies have tradeoffs, but security is not one of them. Both can be and should be easy to make secure in the sense of "make sure it contains the fix". Both may require app redeployment in practice, and expecting them not to is not a good strategy right now (for most programming languages)
    - flukus 9y ago
      > traditional shared libraries never have good enough interfaces to not break stuff. So tradtitionally, the shared library isn't updated anyway until the app is fixed and deployed for the new version of the shared lib. I agree and disagree. Traditionally a lot of shared libraries are in the OS and quite good at not breaking things, you can run windows update or apt-get upgrade and apps shouldn't break. Breakages generally happen in higher level libraries, but that is a cultural issue more than a technical one. If more care is put into the interface and managing breaking changes then it is doable. But regardless of how libraries are being deployed "move fast and break things" is not an attitude I want from the developers of them. Most language specific package managers I've used do a terrible job of telling developers which packages are obsolete and no longer supported or of what the ramifications for updating a package is. Most libraries in these repositories don't even support previous versions. > Bundled dependencies are easy to handle from a security perspective. But not from a user or corporate perspective. I shudder to think at how many old and unpatched libraries are currently in use in the corporate world where the have an attitude of "deploy, then don't touch until we request a feature". It's an uphill battle to even upgrade when there are no compatibility issues, if they need development time to update then they're likely to stick to an old version forever. Users are unlikely to upgrade as well for various reasons, so if a shared library is bundled with an app it is unlikely to be patched.
- jonstewart 9y ago
  However large and robust the deployment may be, it's still a narrow use case/environment compared. It may be very well outside the norm of C++ use cases, frankly. A lotta people make Windows apps still.

krona 9y ago

> Once you learn how to use it (which doesn’t take long) Bazel is pretty darn nice. I've spent a few hundred hours integrating bits of CMake build scripts in to various Bazel builds. If you're doing something which is explicitly catered for by Bazel itself then it works well, so long as you avoid the bugs. Otherwise, you'll have to confront the hermetic build abstractions which prevent otherwise simple operations. Bazel toolchain files are just as bad as CMake, if not worse. I'll give Bazel another try in a few years. For now, I still prefer CMake+Ninja, worts and all.

0xbear 9y ago
Hermetic is good though. Hermetic means reproducible.
- krona 9y ago
  Use Docker?
  - 0xbear 9y ago
    Bazel is on its way to be more like Google’s internal implementation: distributed builds, object file cache, distributed tests. That means your C++ build and test runs will take barely any time at all, same as they do at Google. That, in turn, enables the kind of velocity unachievable through other means.
    - dekhn 9y ago
      I'm sorry but, "bwahahahha". C++ build and test times at Google are astronomical. It's one of the reasons go exists: they built it to compile and link faster.
      - 0xbear 9y ago
        If a minute to build and link half a million lines of C++ is “astronomical”, then sure. Try to build that on your local box, then get back to me. Go exists solely because Rob Pike had nothing better to do, and a lot of googlers hate Java.