14 ms·
Buck2: Our open source build system
- rektide 4y agoI was reading on & on, going, yeah sounds great, but when are you going to mention that it runs on Mercurial, which just puts such massive distance between FB & the rest of the planet? They do mention that it supports virtual file systems through Sapling, which now encompasses their long known EdenFS. I'd like to know more but right off the bat Sapling says it is: > Sapling SCM is a cross-platform, highly scalable, Git-compatible source control system. Git compatible eh? Thats a lot less foreboding. (It is however a total abstraction over git with it's own distinct cli tools.) I hope there are some good up to date resources on Sapling/EdenFS. I have heard some really interesting things about code kind of auto getting shipped up to the mothership & built/linted proactively, it just at commit points, which always sounded neat, and there seem to be some neat transparent thin checkout capabilities here. https://github.com/facebook/sapling https://github.com/facebook/sapling
- autarch 4y agoI'm not sure how Mercurial is relevant. From reading the Buck2 getting started docs, it looks like it works just fine with Git repos.
- rektide 4y agoIt indeed is not the primary concern of build systems. For many folks, there's some ci/CD systems checking stuff out & feeding their build tools. Buck2 notably though tries to be a good persistent incremental build system, and it needs much more awareness of things changing. It does integrate with Sapling for these kind of reasons. So the boundaries are a bit blurrier than they used to be between scm/build/ci/CD tools.
- fanzeyi 4y agoMy understanding is that Buck2 uses Watchman (disclaimer: I'm one of the maintainers) so it can work with both Git and Mercurial repositories efficiently, without compromising performance.
- ndmitchell 4y agoIt can use watchman, but for open source we default to inotify (and platform equivalent versions on Mac/Windows) to avoid you having to install anything.
- krallin 4y ago(engineer working on Buck2 here) Buck2 is actually used internally with Git repositories, so using Sapling / hg is definitely not a requirement
- RcouF1uZ4gsC 4y ago> Buck2 is an extensible and performant build system written in Rust I really appreciate tooling that is written in Rust or Go that produce single binaries with minimal runtime dependencies. Getting tooling written in for example Python to run reliably can be an exercise in frustration due to runtime environmental dependencies.
- TikolaNesla 4y agoYes, just what I thought when I installed the Shopify CLI (https://github.com/Shopify/cli https://github.com/Shopify/cli) a few days ago because they force you to install Ruby and Node
- rektide 4y agoPersonally it seems like a huge waste of memory to me. It's the electron of the backend. It's absolutely done for convenience & simplicity, with good cause after the pain we have endured. But every single binary bringing the whole universe of libraries with it offends. Why have an OS at all if every program is just going to package everything it needs? It feels like we cheapened out. Rather than get good & figure out how to manage things well, rather than driver harder, we're bunting the problem. It sucks & it's lo-fi & a huge waste of resources.
- thangngoc89 4y ago> with minimal runtime dependencies You’re probably thinking of static binary. I believe that OP is comparing a single binary vs installing the whole toolchain of Python/Ruby/Node and fetching the dependencies over the wire.
- crabbone 4y agoIf it's not a statically linked binary, then the problem is just as bad as it is with Python dependencies: instead, now you need to find the shared libraries that it linked with.
- crabbone 4y ago
- jeffbee 4y agoHrmm, it makes performance claims with regard to Buck1 but not to Bazel, the obvious alternative. Hardly anyone uses Buck1 so you'd think it would be relevant.
- dtolnay 4y agoI have a non-toy multi-language project in https://github.com/dtolnay/cxx https://github.com/dtolnay/cxx for which I have both Buck2 and Bazel build rules. On my machine `buck2 clean && time buck2 build :cxx` takes 6.2 seconds. `bazel clean && time bazel build :cxx` takes 19.9 seconds.
- jeffbee 4y agoThat's cool. I was not able to repro due to the buck2 instructions not working for me, two different ways Compiling gazebo v0.8.1 (/home/jwb/buck2/gazebo/gazebo) error[E0554]: `#![feature]` may not be used on the stable release channel --> gazebo/gazebo/src/lib.rs:10:49 Then with the alternate instructions: error: no such command: `+nightly-2023-01-24` Cargo does not handle `+toolchain` directives. Did you mean to invoke `cargo` through `rustup` instead?
- fanzeyi 4y agoIt looks like you don't have rustup.rs. You will need to install that since Buck2 is depending on nightly Rust features.
- yencabulator 4y agoFWIW buck2 doesn't seem to build with nightly-2023-01-24 at this time, nightly-2023-03-15 worked for me. (Nightlies from April cause an internal compiler error.)
- jeffbee 4y agoAnyway regardless of the fact that my local Rust environment isn't cut out to repro your result, how much of that speedup is due to parallelism that Buck2 offers and Bazel does not? When I build your cxx repo with bazel and load the invocation trace, the build was fully serialized.
- ethicalsmacker 4y agoKind of upset they didn't leverage a corny phrase like "the buck stops here".
- thefilmore 4y ago> In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Interesting, so twice the bang for your buck.
- faitswulff 4y agoBut if you need Buck2 then you’re back to one bang per buck
- bogwog 4y agoI feel so lucky that I found waf[1] a few years ago. It just... solves everything. Build systems are notoriously difficult to get right, but waf is about as close to perfect as you can get. Even when it doesn't do something you need, or it does things in a way that doesn't work for you, the amount of work needed to extend/modify/optimize it to your project's needs is tiny (minus the learning curve ofc, but the core is <10k lines of Python with zero dependencies), and doesn't require you to maintain a fork or anything like that. The fact that the Buck team felt they had to do a from scratch rewrite to build the features they needed just goes to show how hard it is to design something robust in this area. If there are any people in the Buck team here, I would be curious to hear if you all happened to evaluate waf before choosing to build Buck? I know FB's scale makes their needs unique, but at least at a surface level, it doesn't seem like Buck offers anything that couldn't have been implemented easily in waf. Adding Starlark, optimizing performance, implementing remote task execution, adding fancy console output, implementing hermetic builds, supporting any language, etc... [1]: https://waf.io/ https://waf.io/
- xxpor 4y agoI truly believe any build system that uses a general-purpose language by default is too powerful. It lets people do silly stuff too easily. Build systems (for projects with a lot of different contributors) should be easy to understand, with few, if any, project specific concepts to learn. There can always be an escape hatch to python (see GN, for example), but 99% of the code should just be boring lists of files to build.
- bogwog 4y agoI agree, that’s also pretty much why Starlark exists. However, there are many cases where you do need complex build logic. Personally, I always go for declarative CMake first, then waf as soon as I find my CMakeLists looking like something other than just a list of files. I’ve considered before creating a simple declarative language to build simple projects like that with waf, but I don’t like the idea of maintaining my own DSL for such little benefit, when CMake works just fine, and everyone knows how to use it. I feel like I’d end up with my own little TempleOS if I decided to go down that rabbit hole.
- evmar 4y agoHow do the "transitive-sets (tsets)" mentioned here compare to Bazel depsets[1]? Is it the same thing with a different name, or different in some important way? [1] https://bazel.build/rules/lib/depset https://bazel.build/rules/lib/depset
- cjhopman 4y agotsets are described in more detail here: https://buck2.build/docs/rule_authors/transitive_sets/ https://buck2.build/docs/rule_authors/transitive_sets/. Bazel's depsets were one of the influences on their design. To users, they will seem fairly similar and would be used for solving similar problems, there's some differences in the details of the APIs. I'm not well-versed on the internal implementation details of bazel's depsets, but one interesting thing about tsets that may further differentiate them is how they are integrated into the core, specifically that we try hard to never flatten them there. The main two places this comes up are: (1) when an action consumes a tset projection, the edges on the DICE graph (our incremental computation edges) only represent the direct tset roots that the action depends on, not the flattened full list of artifacts it represents and (2) when we compute the input digest merkle tree for uploading an actions inputs to RE, that generally doesn't require flattening the tset as we cache the merkle trees for each tset projection node and can efficiently merge them.
- candiddevmike 4y agoFor folks that are using these kinds of tools, any regrets? How much more complexity do they add vs make or shell scripts?
- numbsafari 4y agoSpeaking of the “tools” genrically, they are totally worth it because of their ability to aggressively cache and parallelize, but also because you end up with more declarative definitions of common build targets in a way that is more or less type safe. I personally think that makes these kinds of tools a win over make. Beyond that, they make it trivial to implement repeatable builds that run build steps in “hermetic” sandboxes. You can do all that with make, but you are abusing the hell out of the tool to get there, and it will look foreign to anyone familiar with using make “the traditional way”. That said, bazel’s set of prepublished rules, reliance on the jdk, etc, make it not worth the burden, imo/e. I think less ambitious, but similar tools are where it’s at. We use please for this reason, and are generally quite happy with how it balances between pragmatism and theory. In any event, having your build tool be a single binary is a major win. I’d rather use make than anything written in python or Java just because I don’t have to worry about the overhead that comes with those other tools and their “ecosystems”.
- optymizer 4y agoI've used Buck at Meta for years and while it is technically impressive and does an excellent job with speed, remote compilation and caching, I am not a fan of all of the undocumented, magic rules used all over the place in BUCK and .bzl files. I've yet to try BUCK on small projects though - I personally default to Makefiles in that case. On thing I definitely wouldn't use it is for Android development. The Android Studio integration is much worse than gradle's and adding external dependencies means you have to make BUCK modules for each one. I would however use it for large-scale projects or projects with more than a dozen developers.
- krschultz 4y agoIMO one of the nice things about Buck or Bazel is that once you learn it, switching languages doesn't require you to learn a completely new tool. Obviously the cost of learning it the first time is high and if you are used to one ecosystem may not be worth it. But I'm now on my 3rd different ecosystem that uses Buck/Bazel (Android, iOS, C++) and it's nice to not worry at all about the underlying tools.
- mgaunard 4y ago> Written in Rust stopped reading there.
- echelon 4y agoRust is everywhere. It's a great language.
- mgaunard 4y agoIt failed at its own premise (statically checked object lifetime). The only reason people use it is to have an excuse to rewrite things in something new, free from "legacy".
- redleader55 4y agoIt might be worth to continue reading at least up to the point where they explain why they chose rust, what benefits does it bring compared to the previous tool(based on JVM), etc. In a company like Meta, while inefficiencies are bound to exist, when it come to projects like this, where it's very easy to measure before and after, decisions to rewrite a huge piece of infra are not taken on a whim or because "rewrite it in Rust".
- 0cf8612b2e1e 4y agoI like to use Makefiles for project automation. Does Buck make it straightforward to run phony target tasks? I have been considering transitioning to Justfiles, but Buck will obviously be getting significantly more expose and mindshare.
- ndmitchell 4y agoThere's no such thing as a phony target, but there is both `buck2 build` and `buck2 run` - each target can say how to run and how to build it separately. So you can have a shell script in the repo, write an export_file rule for it, then do `buck2 run :my_script` and it will run.
- 0cf8612b2e1e 4y agoNuts. Possible, but I would be fighting the platform a bit. Especially if I might want something like phony1 to depends on phony2.
- mkl 4y agoWhat kinds of things are you using phony targets for?
- 0cf8612b2e1e 4y agoEverything I can. Building docs, checking health of prod, deploying to dev/prod (which has real+phony dependencies), starting the development web server with watchexec, running lint, tests, etc. Some of these might have real artifacts, but I typically have more phony than not. Make has quite a few flaws, but it is near universal and a good location to centralize all required project work. Even if the entire task is defined in a shell script elsewhere. That being said, I have being looking longingly at just[0] which is not just concerned with compiling C, but has task automation builtin from the beginning. [0] https://github.com/casey/just https://github.com/casey/just
- 4y ago
- mgaunard 4y agoNo one seems to know how to do practical and useful build systems, so I write my own. In particular the idea of writing something entirely generic that works for everything is a waste of time. The build system should be tailored to building your application in the way that matters to you and making the best of the resources that you have.
- Shish2k 4y agoThis sounds like a nightmare if you’re dealing with even single-digit numbers of projects - even just in my personal spare-time hobby projects, I have ~5 nodejs projects, ~5 php projects, ~5 rust projects, ~5 python projects - and I find myself wishing for a common build tool because right now, even if I only have one build system _per language_, that still means that eg migrating from travis to github actions meant that I needed to rewrite four separate build/test/deploy workflows...
- mgaunard 4y agoI have 500 projects. They're all made to follow the same conventions so neither of them specifies anything redundant.
- ihnorton 4y agoThe fact that Buck2 is written in a statically-compilable language is compelling, compared to Bazel and others. It's also great that Windows appears to be supported out of the box [1,1a] -- and even tested in CI. I'm curious how much "real world" usage it's gotten on Windows, if any. I don't see many details about the sandboxing/hermetic build story in the docs, and in particular whether it is supported at all on Linux or Windows (the only mention in the docs is Darwin). It's a good sign that the Conan integration PR [2] was warmly received (if not merged, yet). I would hope that the system is extensible enough to allow hooking in other dependency managers like vcpkg. Using an external PM loses some of the benefits, but it also dramatically reduces the level of effort for initial adoption. I think bazel suffered from the early difficulties integrating with other systems, although IIUC rules_foreign_cc is much better now. If I'm following the code/examples correctly, Buck2 supports C++ out of the box, but I can't quite tell if/how it would integrate with CMake or others in the way that rules_foreign_cc does. (one of the major drawbacks of vcpkg is that it can't do parallel dependency builds [3]. If Buck2 was able to consume a vcpkg dependency tree and build it in parallel, that would be a very attractive prospect -- wishcasting here) [1] https://buck2.build/docs/developers/windows_cheat_sheet/ https://buck2.build/docs/developers/windows_cheat_sheet/ [1a] https://github.com/facebook/buck2/blob/738cc398ccb976856728818685c7b3bcd339f9a4/prelude/cxx/cxx_toolchain.bzl#L18 https://github.com/facebook/buck2/blob/738cc398ccb9768567288... [2] https://github.com/facebook/buck2/pull/58 https://github.com/facebook/buck2/pull/58 [3] https://github.com/microsoft/vcpkg/discussions/19129 https://github.com/microsoft/vcpkg/discussions/19129
- fanzeyi 4y agoOne side effect of all the Metaverse investment is that Meta now has a lot more engineers working on Windows. You bet there will be real world usage. ;)
- e4m2 4y ago> There are also some things that aren't quite yet finished: > There are not yet mechanisms to build in release mode (that should be achieved by modifying the toolchain). > Windows/Mac builds are still in progress; open-source code is mostly tested on Linux. Source: https://buck2.build/docs/why https://buck2.build/docs/why.
- thomasahle 4y agoIt's probably more about better caching, but using buck2 internally at Meta reduced me buildtimes from minutes to seconds. A very welcome upgrade.
- Dowwie 4y agoFor what language?
- thomasahle 4y agoPython mainly.
- 1MachineElf 4y agoThere are a few references to NixOS on the code/issues.[0] I wonder what Meta's use case is for NixOS. [0] https://github.com/facebook/buck2/search?q=nixos&type=issues https://github.com/facebook/buck2/search?q=nixos&type=issues
- ndmitchell 4y agoThese were from an open source contributor - out the box Buck2 doesn't really have support for Nix. But because the rules are flexible, you can write your own version of Nix-aware Buck2.
- aseipp 4y agoThat person creating the issues was me. I simply know of Neil's work, was a user of his previous build system Shake, and we ran in some similar circles in the past -- so when I saw buck2's source initially get released a few months ago on GitHub, I just started using it immediately and giving feedback long before the initial release. Nobody at Meta uses NixOS for production work, from my understanding. Actually, the current tip of trunk for buck2 can't build on NixOS right now with buildRustPackage due to a problem with prost. I should probably file a root cause issue about that soon...
- yencabulator 4y agoEDIT: nevermind, I couldn't get it to build due to a prost issue, gave up and did `cargo build` instead. I misremembered. crane.dev with nightly-2023-03-15 from oxalica/rust-overlay built here okay (after I figured out all the BUCK2_BUILD_PROTOC, missing Cargo.lock, etc tricks). I did have some weird trouble with trying to import buck2 into my flake as a non-flake input, with complaints about "failed to resolve patches" for prost, but putting the flake.nix into the buck2 source tree worked.
- aseipp 4y agoJust using `cargo build` will work as you've seen, but building a Nix package won't work right now. If you're still keeping up with this thread, though, I have code in this repository to build a copy of buck2 with Nix -- it's several weeks out of date at this point though: - https://github.com/thoughtpolice/buck2-nix/tree/main/buck/nix/buck2 https://github.com/thoughtpolice/buck2-nix/tree/main/buck/ni... You should mostly be able to just 'callPackage' that and have it work if you have rust-overlay applied. Check the corresponding 'flake.nix' if you need it. The prost issue with buildRustPackage is preventing an upgrade, so ideally the minor patches to prost can get upstreamed, but I need to (again) file a ticket about moving this along.
- born-jre 4y agowow nobody has dropped obligatory xkcd 927 https://xkcd.com/927/ https://xkcd.com/927/
- PaulDavisThe1st 4y agoBuild systems are not standards.
- oggy 4y agoGreat to see this. I hope it takes off - Bazel is useful but I really like the principled approach behind it (see the Build Systems a la Carte paper), and Neil is scarily good from my experience of working with him so I'd expect that they've come up with something awesome. One thing I find annoying with all of these general, language-agnostic build systems though is that they break the "citizenship" in the corresponding language. So while you can usually relatively easily build a Rust project that uses crates.io dependencies, or a Python project with PyPi dependencies, it seems hard to make a library built using Bazel/Buck available to non-Bazel/Buck users (i.e., build something available on crates.io or PyPi). Does anyone know of any tools or approaches that can help with that?
- marcyb5st 4y agoRegarding bazel, the rules_python has a py_wheel rule that helps you creating wheels that you can upload to pypi (https://github.com/bazelbuild/rules_python/blob/52e14b78307a62aedb69694a222decf48d54a09b/python/packaging.bzl#L72 https://github.com/bazelbuild/rules_python/blob/52e14b78307a...). If you want to see an approach of bazel to pypi taken a bit to the extreme you can have a look at tensorflow on GitHub to see how they do it. They don't use the above-mentioned building rule because I think their build step is quite complicated (C/C++ stuff, Vida/ROCm support, python bindings, and multiOS support all in one before you can publish to pypi).
- dfinninger 4y agoI use py_wheel to build packages to be consumed by data scientists in my company. It works well and is reasonably straightforward. Although the packages are pure Python so I haven’t had to deal with native builds.
- dnsco 4y agoIf I'm understanding, for the rust specific case, this generates your BUCK files from your Cargo.toml: https://github.com/facebookincubator/reindeer https://github.com/facebookincubator/reindeer
- kccqzy 4y ago
- phendrenad2 4y ago> Build systems stand between a programmer and running their code, so anything we can do to make the experience quicker or more productive directly impacts how effective a developer can be. How about doing away with the build system entirely? Build systems seem like something that shouldn't exist. When I create a new C# .NET app in Visual Studio 2019, what "build system" does it use? You might have an academic answer, but that's beside the point. It doesn't matter. It just builds. Optimizing a build system feels like a lack of vision, and getting stuck in a local maxima where you think you're being more productive, but you're not seeing the bigger picture of what could be possible.
- palata 4y agoHow do you think VS builds your code?
- phendrenad2 4y agoThe point is it doesn't matter. What it's doing internally isn't interesting to me, a programmer. I'd rather not think about it. Unfortunately, people have decided that microoptimizations like writing declarative build definitions by hand is beneficial, so I've had to learn many build systems.
- __float 4y agoThe way it builds your software is also a declarative build system: MSBuild. It happens to be configured in XML instead of a Python-like language, and the IDE integration is probably better than what you get from Buck or similar. Building software can be complex sometimes, and _pretending_ it's a "microoptimization" to care about that is unnecessarily diminutive.
- palata 4y agoWell first, there is a build system in your VS. The fact that you don't know about it does not magically make it disappear. Some people have to understand some of it. For instance the person who maintains your project for you, apparently (because you'd rather not think about it). It's like if I said "I don't really know to understand the details of how a CPU works in my daily job. What about we just removed CPUs entirely?".
- LegNeato 4y agoCongrats to the team! Very excited to finally get to use this.
- orthoxerox 4y agoI really hope the team responsible for it is called Timbuktu.
- lopkeny12ko 4y agoI'm missing some historical context here. This article goes out of its way to compare and contrast with Bazel. Even the usage conventions, build syntax (Starlark), and RBE API are the same as in Bazel. Did FB fork Bazel in the early days but retain basically everything about it except the name? Why didn't they just...adopt Bazel, and contribute to it like any other open source project?
- 0xcafefood 4y agoOne thing you might be missing is that this is Buck2. Buck (https://github.com/facebook/buck https://github.com/facebook/buck) has been open sourced for nearly 10 years now. The lore I've heard is that former Googlers went to Facebook, built Buck based on Blaze, and Facebook open sourced that before Google open sourced Blaze (as Bazel). The first pull to the Buck github repo was on May 8, 2013 (https://github.com/facebook/buck/pulls?q=is%3Apr+sort%3Acreated-asc+ https://github.com/facebook/buck/pulls?q=is%3Apr+sort%3Acrea...). The first to Bazel was Sep 30, 2014 (https://github.com/bazelbuild/bazel/pulls?q=is%3Apr+sort%3Acreated-asc+ https://github.com/bazelbuild/bazel/pulls?q=is%3Apr+sort%3Ac...).
- dheera 4y agoSmells like what FB did with Caffe vs. Caffe2, the two of which have nothing to do with each other.
- bradfitz 4y agoI was visiting the Google Munich office on the day Google open sourced Blaze/Bazel. The Facebook Buck team sent a congratulations cake: https://photos.app.goo.gl/6KwE6qeD3i72kSo38 https://photos.app.goo.gl/6KwE6qeD3i72kSo38 Amusingly, the cake was in German but most of the Bazel team didn't really speak German. But it was yummy.
- ynx 4y agoBuck far predates Bazel, and was built by ex-googlers replicating Blaze. Skylark was a later evolution, after the python scripts grew out of control, and a cue that fb took from Google long after Buck had been near-universally deployed for several years.
- rockwotj 4y agoDoes anyone know how IDE support for Buck2 is? I couldn't find anything except some xcode config rules. Half the battle with Bazel/Buck/etc is that getting and IDE or LSP to work for C++/Java/Kotlin/Swift/etc is always a pain because those tools don't really work out of the box.
- habitue 4y agoprobably use a starlark plugin?
- piperswe 4y agoHow does that help with developing C in a Buck project?
- sebastos 4y agoThe vscode bazel plugin is basically completely abandoned. There's an issue that has been open for 3 years asking to add intellisense support for C++. Seems completely ludicrous to put in the massive effort it took to build Bazel and then fumble at the goal line by not supporting vscode.
- rockwotj 4y agoI think the recommendation for c/c++ in Bazel is to use this: https://github.com/hedronvision/bazel-compile-commands-extractor https://github.com/hedronvision/bazel-compile-commands-extra... And use the compile command json file to power clangd. I'm not a vscode person but I would hope the vscode c++ plugin would support that
- yurodivuie 4y agoDo smaller companies (smaller than Meta and Google) use these kinds of build tools much? It seems like a system that rebuilds everything whenever a dependency changes is more suited an environment that has very few, if any, external dependencies. Is anyone using Buck/Bazel and also using frameworks like Spring, or React, for example?
- chucknthem 4y agoUber adopted Bazel a few years ago for their Go and Java monorepos, which is the majority of their code at the time. I doin't know the state of their UI repos.
- krschultz 4y agoI would heavily consider this type of system once build times become a major pain point. That often happens somewhere around 20-50 people working in one codebase. So I think this is a problem space for medium sized companies. Truly small companies probably don't need this and should use the standard ecosystem tools, BUT if your team knows how to use it there's little downside in started from a Buck / Bazel. Especially since you get most of the benefit if you have a nice clean DAG of your modules, and that's easy to build at the beginning and hard to refactor into later.
- surrealize 4y agoI worked at a company that was about 150 people when I joined. It's not primarily a software company but the early team had a bunch of ex-google folks, and they chose Bazel. I encountered it for the first time there. We did use React, yes. I really liked the cross-language aspect of Bazel. Having one command that could compile everything and produce a deployable container in a highly reproducible way is great. It really cut down on "what's your compiler/tool version etc."-type back-and-forth during debugging with other engineers. The bazel JS/TS rules were tough to work with when we first started using it for JS (2018 I think), especially since we were using create-react-app at the time, and that didn't mesh well with the way bazel wants to work. It's gotten a lot better though. If I was making the choice from scratch in a new company/codebase, I think it'd really depend on the team. You kind of need broad-based buy-in to get the full benefits IMO.
- shaman1 4y agoExcuse my ignorance but what are the advantages of using such a system over the standard build systems of various languages (vite, gradle, maven, pip, etc)?
- gabrielrc 4y agoIn my experience there are two main advantages: reproducibility and cross-language support. Build systems like maven can read from the entire filesystem (e.g. it reads from ./~m2) and you might end up with artifacts that depend on the state of the machine. This makes debugging production issues harder. You can of course be careful with other build systems to stay reproducible but it's easy to make mistakes and buck will enforce that you don't. Companies like Meta and Google have huge monorepos and are using multiple languages. It's common for developers to deal with multiple languages and for projects to depend on other languages. Buck can deal with that very naturally and avoids the engineers to deal with multiple build tools. There are other upsides and downsides listed on their website.
- i-use-nixos-btw 4y agoIf you only use one language, and that language has a reliable, reproducible build system that gives you the guarantees and functionality that you require, then not much. Here’s how I used Bazel (and how I now use Nix). I am provided a configuration file that specifies what a given instance of my program must do. I use language A that can natively understand this configuration to code-generate a file in language B (which is significantly more suited to the performance requirements of the program) This file is then built along with generic program code. It is used to process a lot of data. As an interface to this program, I have a HTTP interface that can communicate with it. It needs to understand the kinds of outputs the program will produce, so some of this HTTP interface is also code generated in language A. The interface is interactive so typescript is generated and then compiled too. In order to process the output from the program, I need to produce extensions for languages that users of the output use: Python and R. These need to understand the kinds of data being used, so are also code generated - then built. They’re then tested against requirements defined by the config (so the tests are also code generated). Each of these stages have dependencies required when building and dependencies required when running, and there are several languages involved. I also need to be able to express the entire process as a function, because it’s something that needs to be run very frequently on different configs - sometimes in collections of configs that need everything built at once. It needs to be build on several different machines, sometimes desktops and sometimes remote servers, sometimes on clients’ hardware (depending on the needs). I need confidence that something that works on my development machine will work, in entirety, on a completely different machine with minimal prior setup. And I need it to be easy to do, easy to maintain, and I don’t want to mess around with many different build systems that have entirely different use cases, entirely different ideas of how build/runtime environments should be handled, entirely different languages to configure them - and many of them are rigid and don’t have a concept of functions, they’re just “state your dependencies, kthxbye”. All of the above is absolutely trivial with bazel if you know where to tread lightly (e.g. surrounding Python environments). It’s also very easy with Nix once you get used to it, and you don’t need to tread lightly there - it has stronger guarantees.
- rtpg 4y agoThe dynamic dependency stuff looks very nice! It feels like a good entrypoint for systems that are "merely" wanting good build caching, and not being "so huge git falls apart" big. My biggest gripe with Bazel is how when you're off the beaten path suddenly it feels like the ecosystem really doesn't want you to just solve problems yourself. Meanwhile in this Buck2 documentation, directly talking about adding good support for tools outsides of community-provided things. I still am not a superfan of the awkward way that custom implementations get declared (which I think comes from needing to support super-giant projects? But it's jsut awkward) and all the naming suffers from Google-like "we cannot call them functions but must call them factories" NIH things... but at least there are clear docs.
- chubot 4y agoThe linked paper is pretty interesting, and short at 4 pages. In plainer language, I'd say the observation/motivation is that not only do compiling and linking benefit from incrementality/caching/parallelism, but so does the build system itself. That is, the parsing of the build config, and the transformation of the high level target graph to the low level action graph. So you can implement the build system itself on top of an incremental computation engine. Also the way I think about the additional dependencies for monadic build systems is basically #include scanning. It's common to complain that Bazel forces you to duplicate dependency info in BUILD files. This info is already present (in some possibly sloppy form) in header files. So maybe they can allow execution of the preprocessor to feed back into the shape of the target graph or action graph. Although I wonder effect that has on performance. --- The point about Java vs. Rust is interesting too -- Java doesn't have async/await, or coroutines. I would have thought you give up some control over when things run with with async/await, but maybe not... I'd like to see how they schedule the tasks. Implementing Applicative Build Systems Monadically https://ndmitchell.com/downloads/paper-implementing_applicative_build_systems_monadically-01_jan_2022.pdf https://ndmitchell.com/downloads/paper-implementing_applicat...
- dbaupp 4y agoPants v2 supports dependency inference in a manner similar to what you’re hypothesising. It indeed can benefit from the general caching mechanism. https://blog.pantsbuild.org/why-dependency-inference/ https://blog.pantsbuild.org/why-dependency-inference/ It has been remarkably convenient for our Python monorepo.
- chubot 4y agoAh very interesting, I had heard of Pants, but I didn't know about Pants 2. Having less BUILD metadata definitely seems like a big win, and I imagine with some effort you can keep the parallelism/caching/distribution too (I'd be interested in details on that). Since Pants 2 has a core in Rust, I wonder if you considered Starlark vs. Python? I can see advantages to Python, but it seems like there are good open source implementations of Starlark now. With Pants 2 and Buck 2, now we're on the 3rd or 4th generation of the Bazel-like systems :)
- School-Cotton 4y ago> In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. In my experience Buck was spending a huge amount of time in GC, so this doesn’t surprise me. It must have been (ab)using Java in such a way that massive amounts of stuff were sprayed across the heap.
- dboreham 4y agoAh. The "you're holding it wrong" defense of GC :)
- kylecordes 4y agoThe essential characteristics of Buck2 look very appealing - but it's hard to see this catching up with the substantial ecosystem of language support rules for Bazel.
- hn_go_brrrrr 4y agoIf the language rules are all Starlark, shouldn't they be compatible?
- dikei 4y agoSame language does not necessarily mean Buck2 will provide the same API as Bazel to write language rules.
- EntrePrescott 4y agoHaven't looked at the Bazel codebase, but it would strongly surprise me if the language support rules were implemented in Starlark. More likely, I'd suppose them to be almost exclusively written in Java (and highly based on quite Bazel-specific classes), with Starlark only coming into play in the form of bindings for being used by the users for their BUILD/bzl definitions. Edit/Add: cf the part about language rules in this comment from a person who says they're a former bazel developer: https://news.ycombinator.com/item?id=35477309 https://news.ycombinator.com/item?id=35477309
- hn_go_brrrrr 4y agoOnly the C++ and Java rules are native, the rest are Starlark. I don't know Bazel internals so I can't really comment on your point about APIs.
- jmmv 4y agoAs a former Bazel developer and current Bazel user, I very much like the design principles that they outline for Buck2. In particular: * The fact that it is written in a compiled safe language is a breath of fresh air. I personally like Java the language and understand why Bazel was originally written in Java and how it has done a great job at "hiding" it, but it's still there. In particular, Java's memory and threading models has been problematic for certain scenarios. (I haven't kept up with the language advances and I believe there are new ways to fix this, but adopting them would require a major overhaul of Bazel's internals.) Plus Bazel being written in Java prevents it from being adopted in smaller projects that are /not/ written in Java--a bummer for the whole open source ecosystem. * The complete separation of language rules from the core is great. This is something that Bazel has wanted to achieve for a long time, but they are still stuck with native C++ and Java rules (it's really hard to rewrite them apparently). Not a huge deal, but in Buck2's case, their design highlights that it's clean enough to support this from day one. * The "single" phase execution is also nice to see. Bazel used to have three phases (loading, analysis, and execution) and later managed to interleave the first two. However, the separation is still annoying from a performance perspective, and also introduces some artifacts in the memory model. * It's good that various remote execution facilities as well as virtual file systems have been considered from day one. These do not matter much... until they do, at which point you want the flexibility to incorporate them. Bazel used to have this in the Google-internal version (there is that ACM paper that explains this), but the external version doesn't. For example, there is a patch to support an output tree on a lazy file system courtesy of the bb-clientd project, but after years it hasn't been upstreamed yet. * And lastly, it's also great to see that what they open sourced is what they use internally. Bazel isn't like that: Google tried to open source a "cleaner version" by removing certain warts that were considered dead ends... and that has been both good and bad. On the one hand, this has been key to developing Starlark to where it is today, but on the other, this has made it hard for certain communities to adopt Bazel (e.g. the Python rules were mostly unusable for a really long time). Now, a question: Buck2 uses the Starlark language, but that does not imply that they implement the same Build APIs to support the rules that Bazel has. Does anyone know to what extent the rules are compatible between the two? If Buck2 supported the Bazel rules ecosystem or with minor changes, that'd be huge!
- 4y ago
- noisy_boy 4y agoWonder if they have examples for Java where maven and groovy are the main two tools. Also, in case of our builds, we can benefit only so much from being faster during build phase because it is all the other bits like SonarQube scans, pushing artifacts to Artifactory, misc housekeeping bits, annoyingly slow Octopus deployments etc, that add most of the time to the long deployment cycle. Sometimes I think if a dedicated Go utility that takes care everything build related (parallelizing when possible) would make things faster; it will have the full picture after all. But then we will be reimplementing all the features of these various tools which is maybe ok at FB scale but would be too much for a smaller shop.
- prpl 4y agoYou can push those down too so they only operate on applicable code (and if caching is working, only the changed code) per-build, with a tool like bazel and probably buck2 - in addition to getting parallel execution per-target.
- djha-skin 4y agoEveryone says buck and bazel are so amazing but honestly, mono repos are unicorns. No one does this. It's useful to no one I know. I keep hearing it's useful to somebody, so it must be really useful when it is, but I've never ever seen buck, bazel or monorepos in real life. And it's been my career to build stuff.
- __float 4y agoThere's quite a few well-known places listed on https://bazel.build/community/users https://bazel.build/community/users across many industries. I think Buck and Pants and Please (and ...) are not as widely used, but if they had a list to add we'd have even more examples.
- 5Qn8mNbc2FNCiVV 4y agoI'm sorry to break it to you, but monorepos are extremely common. Doesn't mean they have to be as large but every company I've been at had a monorepo. And as soon as you have to manage PRs for multiple repos with a new cross-cutting feature or scheduling changes in the correct order you understand why they are so appealing.
- jhoechtl 4y agoThere was some initial fanfare about Sapling but it has pretty much waned. Is git simply good enough?
- Shish2k 4y agoI’ve switched over to using Sapling as a client for my GitHub-based projects - since it’s conceptually similar, with all-round-better UX, and no changes on the server side, it has made my life better and yet I don’t really have anything to say about it ¯\_(ツ)_/¯
- crakhamster01 4y agoJust curious, but are you ex-Meta? Sapling looks pretty neat, but I was hesitant to introduce it at the company I work at since I wasn't sure if it's UX would be well-received. It feels great to me since I got used to it at Meta, but also acknowledge that others have built up the muscle for git-centric workflows.
- smcleod 4y agoWhen it breaks down do they say the buck stops here?