5 ms·

There is available tooling to help you with that, not only has to be discipline. A monorepo oriented build tool such as Bazel will make sure that you don't com

There is available tooling to help you with that, not only has to be discipline. A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. This is extremely efficient. In fact, in my company with a few hundred projects we tried it and it was 10 times faster than our previous Maven based workflow. These tools also help you with dependency management so you keep your code organized. The killer feature is that you finally have true Continuous Integration because the build tool will figure out any dependencies of the code that you change, rebuilding it and testing to make sure that nothing is broken.

notacoward 7y ago
> A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use). The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.
- rkangel 7y ago
  It sounds like it might be worth trying Bazel (I haven't used either). The parent made a suggestion of a specific tool with one anecdatum about performance, and your argument is that a different tool doesn't do the same thing.
  - notacoward 7y ago
    My real argument, as represented by the majority of the text I wrote, was that tooling only solves one part of the problem. The lesser part, at that. But I guess that went over the downvote brigade's heads. Ditto for the fact that I did mention conversations with peers who used other tools (including Bazel BTW) and had similar experiences. Seriously, folks, read before you nitpick.
- yegle 7y ago
  You are describing modifying someone else's unrelated code will cause the rebuild of your code right? This looks like your targets unintentionally depend on the other code. To find out how/where the dependency was introduced, you can use bazel query 'somepath(your_target, some_other_file)'. Not sure if Buck have an equivalent, but bazel should work with Buck repo.
  - notacoward 7y ago
    > This looks like your targets unintentionally depend on the other code. Sometimes it's unintentional, for example an include left in after it's no longer needed. Other times it's intentional but lazy. For example, I often see commits that pull in one dependency just to use one tiny little utility function, but that dependency pulls in ten others, which each pull in ten others, and before you know it any change in a hundred files forces yours to be recompiled even when that tiny utility function wasn't touched. A responsible developer would refactor the header file to keep the number of spurious rebuilds down, but the world is full of irresponsible developers. With multiple repos, the public immutable part of an API and the private fluid part tend to find their way into separate files pretty quickly. With a monorepo they don't, which is kind of ironic because one of the purported benefits of a monorepo is that it's easy to reach into "someone else's code" and refactor it. Unfortunately, people just don't seem to use that ability for good often enough unless review culture forces them to.
- joshuamorton 7y ago
  > Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy. In practice this is almost never because of "a monorepo" and more because people don't like being good about including what they use (see: any Titus Winters talk ever). If, for example you have :test_foo and :foo_lib, things are fine and dandy. When you modify foo_lib you'll run :test_foo and nothing else. But often people are lazy and you have :foobarbaz_lib which is built from three source files, foo, bar, and baz. Each of those has a test, but all the tests depend on :foobarbaz_lib, because that's all that exists. So when you change foo, you run :baz_test, even though in practice, baz might depend on foo, and not the reverse. You fix this by making :foo_lib, :bar_lib, and :baz_lib each with their associated :target_test and explicit dependencies, and then :foobarbaz_lib depends on all three libs (or doesn't exist at all, and everyone who needs :foo_lib uses :foo_lib directly. So yes, absolutely, when/if you start (essentially) globbing your imports, bazel/buck/pants stop being awesome. So don't do that.
  - notacoward 7y ago
    > In practice this is almost never because of "a monorepo" I think you mean in the practice you've seen. I've seen some practice too, and it seems much more prevalent in monorepos. Yeah, still a small sample, correlation does not equal causation, yadda yadda, but I still think it's because monorepos make it so damn easy to reach out and add a huge tree of dependencies without having to think about it. My whole point is that it's possible to do the right thing, but requires more discipline.
    - joshuamorton 7y ago
      > add a huge tree of dependencies without having to think about it. Keep in mind that the alternative is "I have to build (or pull a prebuilt version of) the entirety of every repo that I depend on". An entire repo is always going to be larger than a single target. Yes you're right that pulling in the entirely of {{some large tool}} because you want `tool.strings.util.mangle_name` is bad. The alternatives are 1. (multi-repo) You stick mangle name in its own repo (think `left-pad`) 2. (monorepo) Depend on `tool` in its entirety 3. (either, though more common in multi-repo) Everyone re-implements mangle_name. If the way the name is mangled is important, and it changes in the future, you have major issues and incompatibilities. (think IPV4 -> IPV6 for an example where a "global" `parse_addr` function would have made everyone's lives easier). You can bet on the assumption that mangle_name will never change, but then you aren't doing software engineering[1] All of these are the wrong solution 4. Core utilities like this get factored out into their own files. In a multi-repo environment, everyone depends on the entire core/util repo. In a monorepo environment, you depend on the granular files you need. 4 is the only correct solution, and note that in a monorepo environment it results in you depending on fewer things than in a multi-repo environment. Yes there are a lot of ways to do things wrong, But that's true no matter what tooling you use. When used "correctly", monorepos have the best outcome, and the bad outcomes aren't actually that bad (sometimes longer builds and binary bloat if you don't tree-shake). You always need discipline. Monorepos don't need more. [1]: https://www.youtube.com/watch?v=tISy7EJQPzI https://www.youtube.com/watch?v=tISy7EJQPzI
lmm 7y ago
I don't trust incremental builds (whether via Bazel or any other tool) for an actual deployable build - I've seen them go wrong too many times. For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.
- yegle 7y ago
  Wait what? As long as the build process is reproducible, why do you care if it's incremental or not? If you don't trust bazel, people have been using distcc+ccache since forever.
  - twic 7y ago
    The concern is that an incremental build is not reproducible. In theory, it is, but in practice, sometimes it is not. Doing a clean build for the final artifact that is going to be released is pretty standard practice, IME.
    - joshuamorton 7y ago
      Have you used bazel? The number of times I've even considered using `clean` (the drop some caches make everything mostly clean though still use the global cache command) in my 2+ years with it I can count one one hand, and most of those were unnecessary.
  - Silhouette 7y ago
    Just as an anecdote, in all my time as a professional software developer, the number of projects I have worked on that have enjoyed reliable, reproducible, incremental builds is zero. The effects of global optimisations change. The output code winds up varying based on the order of incremental changes. Updated dependencies get overlooked because of race conditions or networking glitches or filesystem bugs or a hundred other unlikely but not impossible failures. Incremental builds are great for quick day-to-day development, but in my experience they are invariably unsuitable for robust production builds. For me, those should use a git clone or equivalent into an empty directory and then a completely clean build of everything, always.

notacoward 7y ago

> A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use). The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.

rkangel 7y ago
It sounds like it might be worth trying Bazel (I haven't used either). The parent made a suggestion of a specific tool with one anecdatum about performance, and your argument is that a different tool doesn't do the same thing.
- notacoward 7y ago
  My real argument, as represented by the majority of the text I wrote, was that tooling only solves one part of the problem. The lesser part, at that. But I guess that went over the downvote brigade's heads. Ditto for the fact that I did mention conversations with peers who used other tools (including Bazel BTW) and had similar experiences. Seriously, folks, read before you nitpick.
yegle 7y ago
You are describing modifying someone else's unrelated code will cause the rebuild of your code right? This looks like your targets unintentionally depend on the other code. To find out how/where the dependency was introduced, you can use bazel query 'somepath(your_target, some_other_file)'. Not sure if Buck have an equivalent, but bazel should work with Buck repo.
- notacoward 7y ago
  > This looks like your targets unintentionally depend on the other code. Sometimes it's unintentional, for example an include left in after it's no longer needed. Other times it's intentional but lazy. For example, I often see commits that pull in one dependency just to use one tiny little utility function, but that dependency pulls in ten others, which each pull in ten others, and before you know it any change in a hundred files forces yours to be recompiled even when that tiny utility function wasn't touched. A responsible developer would refactor the header file to keep the number of spurious rebuilds down, but the world is full of irresponsible developers. With multiple repos, the public immutable part of an API and the private fluid part tend to find their way into separate files pretty quickly. With a monorepo they don't, which is kind of ironic because one of the purported benefits of a monorepo is that it's easy to reach into "someone else's code" and refactor it. Unfortunately, people just don't seem to use that ability for good often enough unless review culture forces them to.
joshuamorton 7y ago
> Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy. In practice this is almost never because of "a monorepo" and more because people don't like being good about including what they use (see: any Titus Winters talk ever). If, for example you have :test_foo and :foo_lib, things are fine and dandy. When you modify foo_lib you'll run :test_foo and nothing else. But often people are lazy and you have :foobarbaz_lib which is built from three source files, foo, bar, and baz. Each of those has a test, but all the tests depend on :foobarbaz_lib, because that's all that exists. So when you change foo, you run :baz_test, even though in practice, baz might depend on foo, and not the reverse. You fix this by making :foo_lib, :bar_lib, and :baz_lib each with their associated :target_test and explicit dependencies, and then :foobarbaz_lib depends on all three libs (or doesn't exist at all, and everyone who needs :foo_lib uses :foo_lib directly. So yes, absolutely, when/if you start (essentially) globbing your imports, bazel/buck/pants stop being awesome. So don't do that.
- notacoward 7y ago
  > In practice this is almost never because of "a monorepo" I think you mean in the practice you've seen. I've seen some practice too, and it seems much more prevalent in monorepos. Yeah, still a small sample, correlation does not equal causation, yadda yadda, but I still think it's because monorepos make it so damn easy to reach out and add a huge tree of dependencies without having to think about it. My whole point is that it's possible to do the right thing, but requires more discipline.
  - joshuamorton 7y ago
    > add a huge tree of dependencies without having to think about it. Keep in mind that the alternative is "I have to build (or pull a prebuilt version of) the entirety of every repo that I depend on". An entire repo is always going to be larger than a single target. Yes you're right that pulling in the entirely of {{some large tool}} because you want `tool.strings.util.mangle_name` is bad. The alternatives are 1. (multi-repo) You stick mangle name in its own repo (think `left-pad`) 2. (monorepo) Depend on `tool` in its entirety 3. (either, though more common in multi-repo) Everyone re-implements mangle_name. If the way the name is mangled is important, and it changes in the future, you have major issues and incompatibilities. (think IPV4 -> IPV6 for an example where a "global" `parse_addr` function would have made everyone's lives easier). You can bet on the assumption that mangle_name will never change, but then you aren't doing software engineering[1] All of these are the wrong solution 4. Core utilities like this get factored out into their own files. In a multi-repo environment, everyone depends on the entire core/util repo. In a monorepo environment, you depend on the granular files you need. 4 is the only correct solution, and note that in a monorepo environment it results in you depending on fewer things than in a multi-repo environment. Yes there are a lot of ways to do things wrong, But that's true no matter what tooling you use. When used "correctly", monorepos have the best outcome, and the bad outcomes aren't actually that bad (sometimes longer builds and binary bloat if you don't tree-shake). You always need discipline. Monorepos don't need more. [1]: https://www.youtube.com/watch?v=tISy7EJQPzI https://www.youtube.com/watch?v=tISy7EJQPzI

lmm 7y ago

I don't trust incremental builds (whether via Bazel or any other tool) for an actual deployable build - I've seen them go wrong too many times. For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.

yegle 7y ago
Wait what? As long as the build process is reproducible, why do you care if it's incremental or not? If you don't trust bazel, people have been using distcc+ccache since forever.
- twic 7y ago
  The concern is that an incremental build is not reproducible. In theory, it is, but in practice, sometimes it is not. Doing a clean build for the final artifact that is going to be released is pretty standard practice, IME.
  - joshuamorton 7y ago
    Have you used bazel? The number of times I've even considered using `clean` (the drop some caches make everything mostly clean though still use the global cache command) in my 2+ years with it I can count one one hand, and most of those were unnecessary.
- Silhouette 7y ago
  Just as an anecdote, in all my time as a professional software developer, the number of projects I have worked on that have enjoyed reliable, reproducible, incremental builds is zero. The effects of global optimisations change. The output code winds up varying based on the order of incremental changes. Updated dependencies get overlooked because of race conditions or networking glitches or filesystem bugs or a hundred other unlikely but not impossible failures. Incremental builds are great for quick day-to-day development, but in my experience they are invariably unsuitable for robust production builds. For me, those should use a git clone or equivalent into an empty directory and then a completely clean build of everything, always.