5 ms·
The next generation of Bazel builds
- LegNeato 2y agoAs the manager of the original Buck team, rules should not be starlark. They should be `wasm`.
- esafak 2y agoWhy?
- matthewbauer 2y agoWhat? Isn't wasm just a bytecode? How could you write rules in wasm?
- kridsdale1 2y agoI don’t understand
- kerbs 2y agoAlso dying for an explanation here
- bloppe 2y agoWasm, specifically using the component model [1], is really good for plugin use-cases. - Polyglot; you can write plugins (in this case Bazel rules) in "any" language, and the component model is easily the best system ever created for cross-language interoperability. - Ecosystem; tooling is still generally immature, but you'd be surprised how much buy-in there is. Web apps are a huge incentive for language developers to support it, and Wasm now runs in a lot of places besides the browser too. Envoy and Istio support Wasm plugins. So does Postgres. So does neovim. - Component interfaces are richly typed. You get things like lists, structs, "resources" (aka objects with instance methods), etc. all supported idiomatically in whichever language you're using and these can be used across language boundaries. Even between GC and non-GC languages. Even with Rust's ownership model. - Plugins are fully sandboxed, so a host can allow plugins to have things like `while` loops while enforcing runtime limits on a per-instruction basis. It also means Bazel doesn't have to invent it's own bespoke sandboxing solutions for every supported platform, as it does today. And Wasm is designed to handle untrusted Web code, so there are tons of security guarantees. - Performance is already really good, and there's every reason to believe it will get better. Again, Browser support is big here. The things the Chrome team did to optimize V8 are pretty incredible, and they're now turning their attention to Wasm in a big way. [1]: https://component-model.bytecodealliance.org/design/why-component-model.html https://component-model.bytecodealliance.org/design/why-comp...
- bloppe 2y agoI'm super into the component model but honestly the tooling is still not ready. Basically only LLVM (and really just C) has good enough tooling. The rest of the wasm tool chains are still heavily focused on core modules and if they support components it's only `wasi:cli/run`
- numbsafari 2y agoMentions Buck, but no mention of Pants or Please.
- simfoo 2y agoThe author clarifies that he wrote the section about Buck2 to demonstrate the need to be Bazel compatible (as opposed to Buck2), because the friction to try it out in a real code base is essentially insurmountable.
- simfoo 2y ago>On the other hand, we need a tiny build system that does all of the work locally and that can be used by the myriad of open-source projects that the industry relies on. This system has to be written in Rust (oops, I said it) with minimal dependencies and be kept lean and fast so that IDEs can communicate with it quickly. This is a niche that is not fulfilled by anyone right now and that my mind keeps coming to; Yes please!
- deleted 2y ago[deleted]
- greener_grass 2y agoEveryone says they want a "tiny" "minimal" "lean" build-system, but there is lots of real complexity in these areas: - Cross-compilation and target platform information - Fetching dependencies - Toolchains I'm not sure a system that solves these would still be considered minimal by most, but those are table-stakes features in my view. If you don't need these things, maybe stick with Make?
- foota 2y agoI think the idea for these words here is more about preferring speed over remote execution and large build caching type of features, but not limiting the subset of toolchain functionality etc.,. In theory if you scoped your build tool to only support builds of sufficiently small size you can probably remove a lot of complexity you have to deal with otherwise.
- mikepurvis 2y agoIntelligent caching is also table-stakes though. It requires a detailed dependency graph and change tracking, and that's not something that can simply be relegated to a plugin— it's fundamental.
- greener_grass 2y agoI agree, yet Make, CMake or even Node package scripts are more popular than Bazel.
- zokier 2y agoI find the dismissal of buck2 pretty shallow. Most of the world is not already heavily invested to bazel, so compatibility is imho overstated; I don't see it being that far-fetched for something like buck2 to leapfrog bazel. That being said, buck2 definitely would need some love from outside meta to really be viable competitor, right now it still feels like half-complete code drop
- IshKebab 2y agoI agree. The biggest issue with Buck 2 for me (apart from documentation) is the lack of something like bzlmod. There's actually a decent number of modules available for Bazel now: https://registry.bazel.build/all-modules https://registry.bazel.build/all-modules But with Buck2 you're stuck with `http_archive` and vendoring.
- setheron 2y agobzlmod feels better at first but you're left navigating version conflicts anyways. Vendoring deps is the only true real sane way for large repos
- keddad 2y agoSeems like a non-issue to me. I work with a relatively large Bazel monorepo, and we have to vendor pretty much anything anyways. Many 3rd-party rules might need patches to work properly with our custom toolchains/rules. Bzlmod sounds nice for small projects, but for big monorepos within organizations with established processes it is more of a hassle, and I imagine that most Bazel users are not using it for small projects.
- IshKebab 2y agoYeah it's more of an issue for small projects. But I don't think Buck/Bazel should be reserved for megarepos with thousands of contributors. Why can't small projects use it?
- aseipp 2y agoTo be honest, I'd rather not see Buck2 repeat the mistakes of Bazel so early on, especially when it took a lot of time before settling on bzlmod. Frankly, I'd rather it go the other way: just have a gigantic 'buck2pkgs' repo that has everything inside of it -- literally fucking everything -- just like Nix and Nixpkgs have done. I've watched and committed to Nixpkgs for over 10 years now and I think being a monorepo has massively contributed to its success, ease of contribution, and general usability. And in practice it means 99% of projects only need one "dependency" and they are done and have everything they could want. In theory a buck2pkgs could even be completely divorced from Meta's existing Prelude, because you can just reimplement it all yourself, without being constrained by their backwards compatibility with Buck1. That would reduce the surface area between the two projects substantially and let you do a lot of huge cleanups in the APIs. I actually started doing this a while back but it is of course a massive amount of work. I do think it's a much better way to go, though...
- throwaway127482 2y agoOne thing not mentioned is the massive amount of complexity and difficulty involved in Starlark rules. In the next generation of build tools, I really wish Starlark could be replaced with some subset of TypeScript, which would have drastically better IDE support. And I wish the typing would be done in such a way that it's harder to do things that are bad for performance. Starlark is super difficult to read, navigate, and write, and there are a lot of performance gotchas. I know there are efforts like starpls (language server for Starlark) but in my experience it really falls short. I think buck2 has type annotations but I kind of hate Python-style types, so I wish we could just use TypeScript :P TypeScript is even better positioned to fill this role now that tsc is being re-written in Go.
- foota 2y agoI'm curious, how did you run into performance issues with starlark?
- gaogao 2y ago> Starlark is intended to be simple. There are no user-defined types, no inheritance, no reflection, no exceptions, no explicit memory management. Execution is finite. The language does not allow recursion or unbounded loops. That has some pretty useful properties for a config language that I'm not sure TypeScript helps much on. That said, Lean also has these properties too as an alternate language with good IDE support, and it would be pretty fun to write proofs of your build scripts.
- tetromino_ 2y agoYou'll be happy to know that the Starlark team has recently started working on adding static typing! See https://github.com/orgs/bazelbuild/projects/21 https://github.com/orgs/bazelbuild/projects/21 and https://bazel.build/rules/language#StarlarkTypes https://bazel.build/rules/language#StarlarkTypes > a lot of performance gotchas Can you list some examples?
- mook 2y agoI worked on a project that used Bazel for a bit (but ended up not being a great fit). I think the part I hated most about Bazel was the fact that there were in fact two dialects; one for BUILD files and one for .bzl files. The things you do in them are different but the documentation is always vague on what goes where.
- AtlasBarfed 2y agoBuild systems are workflow engines. Workflow engines fundamentally separate and structure code and execution in discrete steps with a graph visualizer (hopefully the graph is acyclical). Maybe it's just a bad example, but the code I saw looked very very unseparated. I understand most people who do builds want start out with the one build file to rule them all model. But ultimately you are dealing with a workflow execution. If you were doing anything aside the most trivial single library build. Next generation of whatever build needs to have the visualization and visualized execution assets. There's a great many things that workflow engines fundamentally applied to. Unfortunately they have been politically associated with management trying to replace developer labor, usually because of the snake oil salesman, the workflow engine people are selling to the the c-suite people. A good common visualization and execution set that's cross language would have been really helpful in so many computing domains. It's almost like we need the SQL of workflow engines to be invented
- sunshowers 2y agoYou're not wrong about build systems being workflow engines, but in my experience the risk of over-generalizing is a worse user experience. One of the most valuable lessons I've learned in my career is that a clean UX often requires messy abstractions underneath.
- mdaniel 2y ago> (I can’t name names because they were never public, but if you probe ChatGPT to see if it knows about these efforts, it somehow knows specific details.) I'm going to laugh if Google joins the NYT in suing OpenAI for copyright infringement from having trained on proprietary data. That said: I also wonder how that actually made it in
- jmmv 2y agoThese were not well-guarded secrets (and the ideas were pretty "obvious" if you had faced the problems before in dealing with Bazel and large codebases). In talking to people at the conference, several knew about these efforts (and some even knew the codenames), so I assume these had been mentioned here-and-there in discussion threads.
- kridsdale1 2y agoI can say with full certainty that thousands of engineers at Google were feeding their day to day LLM assisted coding queries in to the ChatGPT web ui for about 2 years before a custom Chrome extension was pushed to employee laptops to stop it. So GPT has likely post-trained on a lot of Google code knowledge.
- londons_explore 2y agoI just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.
- tetromino_ 2y ago50 lines in a single .cc source file which is only compiled once to produce the final artifact - sure, easy to handle. Now consider that you are editing 50 lines of source for a tool which will then need to be executed on some particular platform to generate some parts of your project. Now consider that you are editing 50 lines defining the structure and dependency graph of your project itself.
- rfoo 2y agoIt is well possible by changing 50 lines of code in a 10GB project you have to rebuild the entire project, if everything (indirectly) depends on what you just changed.
- Nzen 2y agoIf your system is C based, tup [0] fulfills your request by watching the outputs of the specified commands. It isn't, however, appropriate for systems like java that create intermediate files that the developer didn't write [1]. [0] https://gittup.org/tup/ex_a_first_tupfile.html https://gittup.org/tup/ex_a_first_tupfile.html [1] https://github.com/gittup/tup/issues/113 https://github.com/gittup/tup/issues/113 Back to bazel, I am of the impression that some of its complexity comes from a requirement to handle heterogeneous build systems. For example, some python development requires resolving both python dependencies and C. Being good at either is a bunch of work; but, handling both means rolling your own polyglot system or coordinating both as a second class citizen.
- taeric 2y agoI mean, easy things are easy to build? It is not at all uncommon to have changes percolate out into larger impacts than you'd expect, though. Especially in projects that attempt whole program optimizations as part of the build. Consider anything that basically builds a program that is used at build time. Which is not that uncommon when you consider that ML models have grown significantly. Change that tool, and suddenly you have to rebuild the entire project if you didn't split it out into a separate graph. (I say ML, but really any simple linter/whatever is the same here.)
- jeffbee 2y agoI find anyone this dismissive of Java hard to take seriously. Java is a cutting-edge language platform where the achievable performance at the limit is on-par with C++. Also, the article's hook didn't grab me. My laptop is way, way more powerful than any "beefy workstation" they ever assigned me at Google, starting with a dual-core 2400MHz Opteron. Bazel's server performance on my laptop or workstation are total non-issues. There are issues with Bazel of course, that's just not one that smells right to me.
- hn_go_brrrrr 2y agoWhy do you think this author (a long-time senior Blaze maintainer) believes this? Do you think the Blaze team just not have the necessary expertise?
- jeffbee 2y agoI have no idea but dismissing Java as not "a real systems language" just isn't a convincing critique.
- lostdog 2y agoFor one, Java programs always take too long to start, and that's not something you want in a program that's supposed to be interactive.
- jeffbee 2y agoSee, that's exactly the type of uncritical statement that will force me to discount the rest of your statements on a whole range of topics. bazelisk help 0.01s user 0.00s system 26% cpu 0.053 total
- lostdog 2y agoTakes 1.97s on my machine, which is a lot of time for just showing a help message. Maybe you have the setup where Bazel is staying alive in the background or something?
- lihaoyi 2y agoA big problem with Bazel not mentioned here is the complexity. It's just really hard for many people to grasp, and adopting Bazel at the two places I worked was a ~10 person-year effort for the rollout with ongoing maintenance after. That's a lot of effort! IMO Bazel has a lot of good ideas to it: hierarchical graph-based builds, pure hermetic build steps, and so on. Especially at the time, these were novel ideas. But in Bazel they are buried behind a sea of other concepts that may not be so critical: `query` vs `aquery` vs `cquery`, action-graph vs configured-action-graph vs target-graph, providers vs outputs, macro vs rule-impl, etc. Some of these are necessary for ultra-large-scale builds, some are compromises due to legacy, but for the vast majority of non-Google-scale companies there may be a better way. I'm hoping the next generation of build tools can simplify things enough that you don't need a person-decade of engineering work to adopt it. My own OSS project Mill (https://mill-build.org/ https://mill-build.org/) is one attempt in that direction, by re-using ideas from functional and object-oriented programming that people are already familiar with to make build graphs easier to describe and work with. Maybe a new tool won't be able to support million-file monorepos with 10,000 active contributors, but I think for the vast majority of developers and teams that's not a problem at all.
- eej71 2y agoI'm glad to see someone else describe their experience this way too. bazel has arrived at $WORK and it has been a non-trivial amount of work for even the passionate advocates of bazel. I know it was written by the Very Smart People at google. They are clearly smarter than me so I must be the dummy. Especially since I never passed their interview tests. :-) Of course given all things google, by the time I'm fully onboard the train, the cool kids will be making a new train and then I'll have to hop onto that way to enjoy the rewards of the promised land that never quite seem to arrive.
- sepositus 2y ago> know it was written by the Very Smart People at google For Google. That's the key. I have the privilege of experiencing both sides, having been at Google for nine years. I never had a problem with Blaze, but using Bazel in a smaller company has been extremely painful. I think there are just very few places that have the exact problems as Google where something like Bazel would be a great fit.
- lostdog 2y agoGreat article, that gets the critique exactly right. The most frustrating part of Bazel is how shoddy the workmanship is. For example, Bazel throws away your analysis cache when you change flags that have nothing to do with what's being built or how, like flags that change what tests are run. If course the biggest issue is that tiny operations take more time than necessary. For example, at $PreviousJob we wrote custom CLI tools and put them in our monorepo, but since you need Bazel to run them, running a basic tool took 9-12 seconds. So this revelation from the article was totally unsurprising: > This rewrite was carefully crafted to optimize memory layouts, avoiding unnecessary pointer chasing (which was impossible to avoid in Java). The results of this experiment proved that Blaze could be made to analyze large portions of Google’s build graph in just a fraction of the time, without any sort of analysis caching or significant startup penalties. Yeah, with attentive engineering it's not surprising that you can get massive speedups. Finally, Bazel is ok at building multiple languages, but awful at getting them to depend on each other. I don't know what's going on here, but whatever magic makes it possible for any language to depend on C and C++ was not extended to other possible dependencies. So now you get to fight or rewrite all the rules, which is another bag of worms.
- mgce 2y ago> The most frustrating part of Bazel is how shoddy the workmanship is. Without commenting on the above statement: > For example, Bazel throws away your analysis cache when you change flags that have nothing to do with what's being built or how, like flags that change what tests are run. I don't think this is a good example. Bazel's analysis cache clearing is to preserve correctness in the context of a legitimately algorithmically difficult problem. The fact Bazel has this limitation is a testament toward its correctness strengths. I'm not aware of systems that have solved that algorithmic problem but curious if anyone knows any. Also Bazel avoids that problem for most "flags that change which tests are run", since that subset of the problem is more solvable. --test_env was a notable exception, which was fixed (https://github.com/fmeum/bazel/commit/eb494194c1c466f7fd735572fd9466729ea08efc https://github.com/fmeum/bazel/commit/eb494194c1c466f7fd7355...) but not sure if it's in the latest Bazel release yet? But generally changing --test* has much smaller performance impact.
- kridsdale1 2y agoIn the 2020s I have used BUCK, BUCK2, and Blaze every working day. I don’t feel like getting in to details but I’ll just say that BUCK2 is the best of the 3 and I’m delighted it’s open source. I don’t have experience with Bazel as it exists externally but I expect the internal Blaze has advanced far beyond it by now as the entire Google engineering world depends on it.
- anon6362 2y agohttps://github.com/facebook/sapling/tree/main/eden https://github.com/facebook/sapling/tree/main/eden I'm still unsure if a complete, viable eden solution was released with all of its necessary components.