5 ms·
I agree that the blog point didn't make a very strong argument, but there are some inaccuracies in your comment as well. With regard to the "only run X tests i
by eklitzke 4y ago
I agree that the blog point didn't make a very strong argument, but there are some inaccuracies in your comment as well.
With regard to the "only run X tests if X changed" problem, Bazel, Buck, and all the other monorepo build tools do this. I mean sure, if you're using some build system not meant for a monorepo you're going to have a terrible time, but who is really going to spend weeks or months converting to a monorepo and not also switch to Bazel (or something akin to it?) at the same time. In fact, I would say switching to Bazel (or Buck, etc.) for builds is a prerequisite to even starting on the path to switch to a monorepo.
This is just a really useful feature even if you're not in a monorepo. Sometimes you're changing some core header file or whatever and you really do need to run nearly all the tests in your test suite. Sometimes you're just changing some fairly self-contained file and only a few tests need to run. Sometimes you change some docs in the repo and you don't need to run any tests at all. Bazel will just do this automatically for local builds (it knows what tests have transitive dependencies that have changed since the last time those tests were run), and setting it up in CI is a few lines of bash or Python. To set this up in CI you basically just check which files have changed since the last time CI ran (e.g. using git diff), then you use bazel query to find all test targets that have transitive dependencies on those files, then you feed that list of test targets to bazel test. You can set this up per-developer branch for instance, so that if you have a bunch of developers all running tests on the same set of CI machines you get good caching.
With regard to colocating schemas in APIs, yes you can do that but it's really annoying to do with Protobuf/Thrift. First of all protobuf and thrift require that IDLs exist locally so they can do code generation, so if you have protobuf files split into multiple services you need a way to distribute them all which is super annoying. Additionally, in some cases there isn't a clear single owner of a particular IDL struct, for example let's say you have some date or time struct that many protobuf messages want to use in their fields. Which service do you define it in? Ignoring that, it is REALLY USEFUL to be able to modify the code for the producer of a message and the consumer of a message all at once, without having to make multiple commits in multiple repositories. This is especially true when it comes to testing. I have thing A producing a new field X, I want B to use the new field X, and I want to test that B uses it correctly. When everything is in one repo this just works, with multiple repos I need to first add the code to thing A, do a release of A (even if that's just making and pushing a git commit), then update B to consume the new thing and add the test, then if I realize I messed something up I need to go update repo A again to test it, and so on. Obviously this works and tons of people do it, but it sucks. I had to do this at my last job (which wasn't using a monorepo), and it worked but it was cumbersome and I hated it.
With regard to scaling, a lot has changed in git in the last two years to make it possible to run huge git monorepos without any weird hacks. The most notable such feature is sparse indexes, which let you clone a subset of a git repo locally and have it work normally. Here's a GitHub blog post about sparse indexes: https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/ https://github.blog/2021-11-10-make-your-monorepo-feel-small... . They also have a monorepo tag which you can use to look at other blog posts about monorepos (and as you'll note, most of these are pretty recent): https://github.blog/tag/monorepo/ https://github.blog/tag/monorepo/
The biggest downside of a monorepo in my opinion is that there are a lot of things that Bazel makes way harder than the default package manager for language X. Practically speaking it's probably going to be hard to use Bazel if you don't have a dedicated build team with experts who can spend all the time to figure out how to make Bazel work well in your organization. This is pretty different from just using pip or npm or yarn or whatever, where you can get things just working in a couple of minutes and the work to maintain the build system is probably just collectively a few hours of work a week from people spread throughout the organization who can be on any team. For a small organization I can't see it being worth the effort unless you already have a lot of engineers who have a background in Bazel, for example. But there's definitely a point where the high entry-level cost to Bazel and a monorepo makes sense.
- lmm 4y ago> you need a way to distribute them all which is super annoying Only if your tools are bad. In the stack I'm used to they're just another artifact that gets published as part of a module build. > Additionally, in some cases there isn't a clear single owner of a particular IDL struct, for example let's say you have some date or time struct that many protobuf messages want to use in their fields. Which service do you define it in? A common library, just like any other kind of code. > Ignoring that, it is REALLY USEFUL to be able to modify the code for the producer of a message and the consumer of a message all at once, without having to make multiple commits in multiple repositories. This is especially true when it comes to testing. I have thing A producing a new field X, I want B to use the new field X, and I want to test that B uses it correctly. When everything is in one repo this just works, with multiple repos I need to first add the code to thing A, do a release of A (even if that's just making and pushing a git commit), then update B to consume the new thing and add the test, then if I realize I messed something up I the new version ofneed to go update repo A again to test it, and so on. That's actually really important because it forces you to test all the intermediate states. If you can just change everything at once then you will, and you probably won't bother testing everything that actually gets deployed, and so you get into a situation where if you could deploy the new version of A and B at exactly the same time then it would work, but when there's any overlap in the rollout everything breaks horribly.
- sanderjd 4y agoIt really sucks to manage shared libraries, across many clients in a few languages. Any update requires updating the version everywhere, and you have to independently debug which version of which library was being used by which application at the time a bug occurred. It works, it isn't impossible, obviously lots of people manage it successfully (polyrepos are more common than monorepos in my experience), but it's a giant pain and it sucks.
- lmm 4y ago> Any update requires updating the version everywhere Much of the benefit of using an IDL like this is to be (mostly) forward compatible, so you don't have to upgrade everywhere immediately. > you have to independently debug which version of which library was being used by which application at the time a bug occurred You have to do that anyway; it's easier if your repository history reflects the reality of which applications were upgraded and deployed at which times. There's nothing worse than having to debug a deployed system that was deployed from parts of a single repo but at several different points in that repo's history.