9 ms·

Working at Google, Blaze is one of the technologies that amazes me most. Any engineer can build any Google product from source on any machine just by invoking

by habosa 12y ago

Working at Google, Blaze is one of the technologies that amazes me most. Any engineer can build any Google product from source on any machine just by invoking a Blaze command. I may not want to build GMail from source (could take a while) but it's awesome to know that I can. I think this could be hugely useful to very large open source projects (like databases or operating systems) that may be intimidating for contributors to build and test.

middleclick 12y ago
So the builds are reproducible automatically?
- hanwenn 12y ago
  See http://bazel.io/docs/FAQ.html http://bazel.io/docs/FAQ.html, "Will Bazel make my builds reproducible automatically? For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (eg. executing binaries through a shell script inside a rule), you will need to take some extra care: Do not use dependencies that were not declared. Sandboxed execution (–spawn_strategy=sandboxed, only on Linux) can help find undeclared dependencies. Avoid storing timestamps in generated files. ZIP files and other archives are especially prone to this. Avoid connecting to the network. Sandboxed execution can help here too. Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages."
  - hansjorg 12y ago
    When you say Java, does that include Android? I see that Android is supported, but couldn't find anything about reproducibility. Reproducible Android builds would be very interesting.
    - esrauch 12y ago
      http://bazel.io/docs/roadmap.html http://bazel.io/docs/roadmap.html It looks like they only have iOS and not Android in this first release, but they are planning on adding Android support ~June of this year.
  - fizbin 12y ago
    Specifically, people should note that many code generators are not carefully designed for strict reproducibility, and will stick time stamps in generated output. Even if you undo that, code generation tools are liable to at some point traverse a dictionary without caring about whether the result is deterministic. I spent some time at Google fighting with antlr to try to get it to have deterministic output and I still think that I left some corner case uncovered.
- deleted 12y ago
  [deleted]
nevir 12y ago
Well, that's also largely due to all the source (transitive dependencies) being present in one monolithic repo.
- woah 12y ago
  Yea what's up with that? Sounds like a pretty terrible practice to me. Is that something that c++ forces you to do?
  - dpark 12y ago
    No, it's something that Perforce allows (because it scales sufficiently). It has nothing to do with C++. You can impose order within the monolithic repo by partitioning projects into their own branches or directories and only pulling down the necessary pieces. Whether this is better than a bunch of small repos is debateable.
    - nevir 12y ago
      Perforce (the product) doesn't really scale sufficiently for the Googles/Amazons/Microsofts of the world, sadly. I think they've all moved to custom forks/implementations due to the insane SPOF that Perforce servers are (and their hardware requirements). But up til that point, heck yeah!
      - deleted 12y ago
        [deleted]
      - dpark 12y ago
        Well, Perforce in its default state may not scale sufficiently, but at least two of the companies on that list have managed to make it work (presumably with a lot of investment, though). :) I didn't know Amazon was using Perforce. I interviewed someone from Amazon recently and he indicated they were on Git for most things now.
        gagomes 12y ago
thrownaway2424 12y ago
Yes, Blaze and a hojillion computers will give you a spiffy build system. The public now has the former, but not the latter :)
- skj 12y ago
  One piece at a time! Also, who is to say that Google's way of orchestrating those hojillion computers is best? Separating the two pieces, as has been done here, makes it possible for others to create different (and maybe better) orchestrations.
malkia 12y ago
I've started at Google 4 months ago, and it's one of the best things to discover. Now open-sourced :)
- philippnagel 12y ago
  Though not really open source ;)
  - nevir 12y ago
    https://news.ycombinator.com/item?id=9257571 https://news.ycombinator.com/item?id=9257571
    - philippnagel 12y ago
      Whoops, I am an idiot.
      - urda 12y ago
        But you're our idiot!
solve 12y ago
Wow, sounds like enterprise Gentoo, in a good way.
emu 12y ago
Standard caveat: I don't speak for my employer. Using Bazel (aka. Blaze) every day is one of the things that has made me dread ever leaving Google. Fast, reproducible builds are amazing. Once you have used this tool, it is very hard to go back. Personally, I'm thrilled that it has been open sourced.
- GauntletWizard 12y ago
  Having recently left Google, GRPC (stubby) was my biggest concern; I spent about two weeks hacking together a good code generator for GoRPC before GRPC came out and obviated the time. Now, I'm glad I haven't bothered with a build system, which was going to be next. Nice to see a bunch of projects that've been generalizable and heavily used internally finally see the light of the outside world. Now, to start evangelizing them.
  - oppositelock 12y ago
    A huge +1 on this as well. I left Google a couple of years ago and we ended up building our own rpc around protos (Thrift just doesn't cut it), and our Make/maven based build has the standard problems with such things, so I'm really looking forward to using grpc and bazel in the near future. A huge thumbs up to Google!
    - beagle3 12y ago
      Can you give a (very quick) pointer/explanation to what about Thrift didn't cut it for you?
      - hueving 12y ago
        It wasn't invented in Google and is therefore inferior. Google has a massive incentive to develop projects specific to their requirements and then open source and evangelize them to stomp out approaches not optimized for them.
        12y ago
deleted 12y ago
[deleted]
OneMoreGoogler 12y ago
> Any engineer can build any Google product from source on any machine A little too optimistic :) You can't build Android, Chrome, ChromeOS, iOS apps, etc. via blaze.
- ovidiup 12y ago
  When I worked at Google I built a Blaze extension to be able to build Android apps. It worked really well, though I'm not sure how well it was maintained after I left in 2010. Internally at Google, Blaze was extremely customizable, and I hope Bazel too, so one can easily add support for building iOS apps etc. EDIT #1: I see support for building Objective-C apps is already present in Bazel. EDIT #2: Bazel uses Skylark, a Python-like language, which could be used to implement all sorts of extensions, including the one I was referring to.
  - rictic 12y ago
    There's an extension language in bazel named Skylark, which will be familiar to you if you wrote build_defs internally: http://bazel.io/docs/skylark/concepts.html http://bazel.io/docs/skylark/concepts.html
  - zak_mc_kracken 12y ago
    The Chromium tool chain is pretty insane. ninja, fetch, etc... There's really no excuse for this since Google has such a strong (and now open source) build system.
- rpereira 12y ago
  http://i.imgur.com/zQ1m75U.gif http://i.imgur.com/zQ1m75U.gif
ambrop7 12y ago
Look at Nix & NixOS: http://nixos.org/ http://nixos.org/ It would be interesting to see a comparison of Bazel/Blaze to Nix.

by habosa 12y ago

middleclick 12y ago

So the builds are reproducible automatically?

hanwenn 12y ago
See http://bazel.io/docs/FAQ.html http://bazel.io/docs/FAQ.html, "Will Bazel make my builds reproducible automatically? For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (eg. executing binaries through a shell script inside a rule), you will need to take some extra care: Do not use dependencies that were not declared. Sandboxed execution (–spawn_strategy=sandboxed, only on Linux) can help find undeclared dependencies. Avoid storing timestamps in generated files. ZIP files and other archives are especially prone to this. Avoid connecting to the network. Sandboxed execution can help here too. Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages."
- hansjorg 12y ago
  When you say Java, does that include Android? I see that Android is supported, but couldn't find anything about reproducibility. Reproducible Android builds would be very interesting.
  - esrauch 12y ago
    http://bazel.io/docs/roadmap.html http://bazel.io/docs/roadmap.html It looks like they only have iOS and not Android in this first release, but they are planning on adding Android support ~June of this year.
- fizbin 12y ago
  Specifically, people should note that many code generators are not carefully designed for strict reproducibility, and will stick time stamps in generated output. Even if you undo that, code generation tools are liable to at some point traverse a dictionary without caring about whether the result is deterministic. I spent some time at Google fighting with antlr to try to get it to have deterministic output and I still think that I left some corner case uncovered.
deleted 12y ago
[deleted]

nevir 12y ago

Well, that's also largely due to all the source (transitive dependencies) being present in one monolithic repo.

woah 12y ago
Yea what's up with that? Sounds like a pretty terrible practice to me. Is that something that c++ forces you to do?
- dpark 12y ago
  No, it's something that Perforce allows (because it scales sufficiently). It has nothing to do with C++. You can impose order within the monolithic repo by partitioning projects into their own branches or directories and only pulling down the necessary pieces. Whether this is better than a bunch of small repos is debateable.
  - nevir 12y ago
    Perforce (the product) doesn't really scale sufficiently for the Googles/Amazons/Microsofts of the world, sadly. I think they've all moved to custom forks/implementations due to the insane SPOF that Perforce servers are (and their hardware requirements). But up til that point, heck yeah!
    - deleted 12y ago
      [deleted]
    - dpark 12y ago
      Well, Perforce in its default state may not scale sufficiently, but at least two of the companies on that list have managed to make it work (presumably with a lot of investment, though). :) I didn't know Amazon was using Perforce. I interviewed someone from Amazon recently and he indicated they were on Git for most things now.
      - gagomes 12y ago

thrownaway2424 12y ago

Yes, Blaze and a hojillion computers will give you a spiffy build system. The public now has the former, but not the latter :)

skj 12y ago
One piece at a time! Also, who is to say that Google's way of orchestrating those hojillion computers is best? Separating the two pieces, as has been done here, makes it possible for others to create different (and maybe better) orchestrations.

malkia 12y ago

I've started at Google 4 months ago, and it's one of the best things to discover. Now open-sourced :)

philippnagel 12y ago
Though not really open source ;)
- nevir 12y ago
  https://news.ycombinator.com/item?id=9257571 https://news.ycombinator.com/item?id=9257571
  - philippnagel 12y ago
    Whoops, I am an idiot.
    - urda 12y ago
      But you're our idiot!

solve 12y ago

Wow, sounds like enterprise Gentoo, in a good way.

emu 12y ago

Standard caveat: I don't speak for my employer. Using Bazel (aka. Blaze) every day is one of the things that has made me dread ever leaving Google. Fast, reproducible builds are amazing. Once you have used this tool, it is very hard to go back. Personally, I'm thrilled that it has been open sourced.

GauntletWizard 12y ago
Having recently left Google, GRPC (stubby) was my biggest concern; I spent about two weeks hacking together a good code generator for GoRPC before GRPC came out and obviated the time. Now, I'm glad I haven't bothered with a build system, which was going to be next. Nice to see a bunch of projects that've been generalizable and heavily used internally finally see the light of the outside world. Now, to start evangelizing them.
- oppositelock 12y ago
  A huge +1 on this as well. I left Google a couple of years ago and we ended up building our own rpc around protos (Thrift just doesn't cut it), and our Make/maven based build has the standard problems with such things, so I'm really looking forward to using grpc and bazel in the near future. A huge thumbs up to Google!
  - beagle3 12y ago
    Can you give a (very quick) pointer/explanation to what about Thrift didn't cut it for you?
    - hueving 12y ago
      It wasn't invented in Google and is therefore inferior. Google has a massive incentive to develop projects specific to their requirements and then open source and evangelize them to stomp out approaches not optimized for them.
      - 12y ago

deleted 12y ago

[deleted]

OneMoreGoogler 12y ago

> Any engineer can build any Google product from source on any machine A little too optimistic :) You can't build Android, Chrome, ChromeOS, iOS apps, etc. via blaze.

ovidiup 12y ago
When I worked at Google I built a Blaze extension to be able to build Android apps. It worked really well, though I'm not sure how well it was maintained after I left in 2010. Internally at Google, Blaze was extremely customizable, and I hope Bazel too, so one can easily add support for building iOS apps etc. EDIT #1: I see support for building Objective-C apps is already present in Bazel. EDIT #2: Bazel uses Skylark, a Python-like language, which could be used to implement all sorts of extensions, including the one I was referring to.
- rictic 12y ago
  There's an extension language in bazel named Skylark, which will be familiar to you if you wrote build_defs internally: http://bazel.io/docs/skylark/concepts.html http://bazel.io/docs/skylark/concepts.html
- zak_mc_kracken 12y ago
  The Chromium tool chain is pretty insane. ninja, fetch, etc... There's really no excuse for this since Google has such a strong (and now open source) build system.
rpereira 12y ago
http://i.imgur.com/zQ1m75U.gif http://i.imgur.com/zQ1m75U.gif

ambrop7 12y ago

Look at Nix & NixOS: http://nixos.org/ http://nixos.org/ It would be interesting to see a comparison of Bazel/Blaze to Nix.