4 ms·

Well, there aren't "subrepos" … it's all one giant monorepo. And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's

Well, there aren't "subrepos" … it's all one giant monorepo. And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's typically not recorded. (Excepting bazel, which set out to solve this problem; lo and behold it was designed by a company with a monorepo, too.) Some CI systems I've seen have half-assed attempts at it, such as "only run this CI step if the files given by this glob changed". But a.) it requires listing a transitive list of all globs that would apply to the current step, so it's not a good way to manage things and b.) every time I've seen this mis-feature, "change" is described as "in this commit"; that's incorrect. (I have base commit B, I push changes '1 and '2, for commit graph B - '1 - '2 ; CI detects for a step that the globbed files didn't change in '2, and ignores '1. The branch is green. I merge. The result merge commit 'M changes the union of files, so now the tests run, and the commit — now on HEAD — breaks the build. A subsequent unrelated commit M - '3 doesn't modify the relevant code; CI skips the tests and delivers a green result on a broken codebase. People erroneously think "problem fixed". I have seen this all play out in person, multiple times.)¹ (A "much easier" approach is to simply cache a single build step: you hash your inputs, and compute a cache keys; see if your output is cached. If yes use cache, if no build. Computing the cache key is the tricky part and risks that famous "top n problems in computer science … cache invalidation" quote.) ¹while I know how to compute better git diffs, the difference between the common ancestor, the result once the commit gets merged, etc. are subtle. Most devs are shockingly inexperience with git and don't even get this far into the problem, and CI system's insistence on only running on, e.g., "pushes" doesn't help.

vlovich123 4y ago
Sure. You can't go halfway on monorepos where you check in all the code into one spot but don't build any tooling to manage that. You need to use something like Bazel/Blaze, Buck, or other tools that meant to own responsibility for managing dependencies between projects. > Well, there aren't "subrepos" … it's all one giant monorepo. I think op meant organizationally you still have logical components. > And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's typically not recorded. (Excepting bazel, which set out to solve this problem; lo and behold it was designed by a company with a monorepo, too.) Everywhere I've worked that has had a monorepo (Google, Facebook), that's definitely the case. The CI automation would query Buck/Bazel to figure out the set of dependencies impacted by a PR. Of course, some PRs would have outsized impacts (e.g. changing libc) but at the same time, there's probably not much better than that. Apple was a bit different. While nominally each project had it's own git repository, you uploaded code to a central monorepo that organized everything by release. And it built every project incrementally and relinked/rebuilt whichever projects were impacted. They didn't at the time have a centralized CI system. But also Apple's system evolved from many decades ago and is a sane selection for building an OS from back then. Google's approach I think is generally accepted as a more effective strategy in some ways if you're going down that route. That being said, at Google scale you're shipping so much code there's still challenges. For example, there's so much code being changed at Google's scale, that they have to bundle things together into a single CI pass because there's just insufficient compute capacity available to do everything + avoid serialization of unrelated components. Of course, probabilistically there's a non zero chance that something is broken and they intelligently bisect and figure out what change needs to be omitted from the ship. Very complicated. But I think most people underappreciate that they'll never encounter these kinds of problems. If you just go all in on monorepo + Bazel + bazel-aware CI setup + build artifact caching, you're done and don't have to think about builds or code management very much after that. That's a really big superpower.
- tshaddox 4y ago
  > I think op meant organizationally you still have logical components. Yes, my impression was that all of these monorepo tools have a first-class notion of the subrepos/projects/workspaces/whatever that make up the monorepo. If you don’t have that, then I guess I don’t really know what you mean when you say you have a monorepo.

vlovich123 4y ago

Sure. You can't go halfway on monorepos where you check in all the code into one spot but don't build any tooling to manage that. You need to use something like Bazel/Blaze, Buck, or other tools that meant to own responsibility for managing dependencies between projects. > Well, there aren't "subrepos" … it's all one giant monorepo. I think op meant organizationally you still have logical components. > And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's typically not recorded. (Excepting bazel, which set out to solve this problem; lo and behold it was designed by a company with a monorepo, too.) Everywhere I've worked that has had a monorepo (Google, Facebook), that's definitely the case. The CI automation would query Buck/Bazel to figure out the set of dependencies impacted by a PR. Of course, some PRs would have outsized impacts (e.g. changing libc) but at the same time, there's probably not much better than that. Apple was a bit different. While nominally each project had it's own git repository, you uploaded code to a central monorepo that organized everything by release. And it built every project incrementally and relinked/rebuilt whichever projects were impacted. They didn't at the time have a centralized CI system. But also Apple's system evolved from many decades ago and is a sane selection for building an OS from back then. Google's approach I think is generally accepted as a more effective strategy in some ways if you're going down that route. That being said, at Google scale you're shipping so much code there's still challenges. For example, there's so much code being changed at Google's scale, that they have to bundle things together into a single CI pass because there's just insufficient compute capacity available to do everything + avoid serialization of unrelated components. Of course, probabilistically there's a non zero chance that something is broken and they intelligently bisect and figure out what change needs to be omitted from the ship. Very complicated. But I think most people underappreciate that they'll never encounter these kinds of problems. If you just go all in on monorepo + Bazel + bazel-aware CI setup + build artifact caching, you're done and don't have to think about builds or code management very much after that. That's a really big superpower.

tshaddox 4y ago
> I think op meant organizationally you still have logical components. Yes, my impression was that all of these monorepo tools have a first-class notion of the subrepos/projects/workspaces/whatever that make up the monorepo. If you don’t have that, then I guess I don’t really know what you mean when you say you have a monorepo.