3 ms·
I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them
by throwaway894345 6y ago
I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them to support any interesting case, and Nix has too many usability issues to use productively (and I’ve been in an organization that gave it an earnest shot).
- krasin 6y agoCan you please give an example of such an interesting case? I am genuinely curious. And I agree with the general point that monorepos require a great build tooling as a match.
- throwaway894345 6y agoAn interesting case of what? A monorepo? Issues with Bazel? With Nix?
- krasin 6y agoAn interesting case that's hard to support with Bazel. Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.
- throwaway894345 6y agoIt’s been a while since I tried it (maybe 2018?), but Python 3 support straight up didn’t work despite the documentation. It also seemed like adding support for your own toolchain required you to write Java plugins in a very confusing, inheritance-for-it’s-own-sake, raptly-2000s-OOP way.
- krasin 6y agoGot it. I can relate: while Python 3 support is probably fixed now, I agree that Bazel tooling for adding toolchains has been always abysmal. Things somewhat get better (https://docs.bazel.build/versions/master/toolchains.html https://docs.bazel.build/versions/master/toolchains.html) since it's now in Skylark, not in Java, but I still don't like it (aka not productive) Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.
- csnweb 6y agoWith GitHub actions you can quite easily specify a workflow for parts of your repo (simple file path filter https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths https://docs.github.com/en/actions/reference/workflow-syntax...). So you can basically just write one workflow for each project in the monorepo and have only those running where changes occured.
- throwaway894345 6y agoThat’s what I’m doing now, but you still have to push artifacts up to a repository and downstream jobs have to pull them down. Further still, managing the boilerplate for GitHub actions is no small task (e.g., you have a dozen Python projects and you want the actions for each to look similar), so to solve for that you have to build some hack that involves generating the Actions YAML before you commit or similar. Not the end of the world, but still far from my ideal state.
- CameronNemo 6y ago>have to build some hack that involves generating the Actions YAML before you commit or similar You could probably use jsonnet expressions and invoke them from a pre-commit hook. That seems fairly elegant to me.
- throwaway894345 6y agoI’m not doing json net, but I am generating them effectively from a pre-commit hook. One of the jobs that I generate validates that the actions that were committed are up-to-date. This works well because I don’t have too many dependencies, but if you have deep dependency trees then the builds will end up taking a long time and you end up reinventing Nix or Bazel poorly.
- CameronNemo 6y agoAre nix and Bazel in the same category? I have used nix very little and never used Bazel. But I have used Make and jsonnet pretty often. Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
- gen220 6y agoYeah, in my experience it takes X years of experience at a company that is already using a monorepo + blaze-derivative tooling, for the costs of setup and maintenance to make it worth adoption. That X is mostly a function of how open you are to the idea that maintaining + using a centralized, language-agnostic build graph is a Useful Thing. But once that X is crossed, I don't think it's possible to happily go back to a non-monorepo environment. :) FWIW, we've used Pants successfully for years, which has excellent Python support. However, we're in a long-haul migration to Bazel, so you might be better off revisiting Bazel if you want to invest in something that's extremely likely to issue dividends over the next 10 years.
- throwaway894345 6y agoWe used Pants as well, and it worked alright for Python. But it’s codebase was a mess and it was also difficult to extend. I get the strong feeling that there is a better way to make a build system—perhaps something like Nix but without so many usability problems—but no one is investing in that right now.
- deleted 6y ago[deleted]