3 ms·

I've worked in environments across the version-control gamut. The best-run places have had monorepos. But, for the life of me, I would not trust the companies t

by gen220 6y ago

I've worked in environments across the version-control gamut. The best-run places have had monorepos. But, for the life of me, I would not trust the companies that didn't have monorepos, to operate a monorepo. To go mono is to make an org-level engineering/cultural commitment, that you're going to invest in build tools, dependency graph management, third-party vendoring, trunk-driven development, and ci/cd infrastructure. Can you make a mono repo work without all of those things? Yes, but you are sacrificing most of its benefits. If your eng org cannot afford to make those investments (i.e. headcount is middling but there is zero business tolerance for investing in developer experience, or the company is old and the eng org is best described as a disconnected graph), forcing a monorepo is probably not the right idea for you. Monorepo vs microrepos is analogous in some ways to static vs dynamic typing debate. A well-managed monorepo prevents entire classes of problems, as does static typing. Dynamic typing has much lower table stakes for a "running program", as do microrepos. edit: It's worth noting that open source solutions to build tooling, dependency graph management, etc. have gotten extremely good in the last 10 years. At my present company (eng headcount ~200), we spend about 2 engineers-per-year on upgrades and maintenance of this infrastructure. These tools are still quite complex, but the table stakes for a monorepo are lower today than they were 10 years ago.

qznc 6y ago
What would be a scalable open-source monorepo stack? Git, Gitlab, Bazel?
- gen220 6y ago
  This is more or less what we use at my current place (100s of engineers). We haven't run into any dramatic issues at this scale. Our architecture is arcanist/phab over git --> stash (git) --> custom build server that's basically identical to gitlab --> artifactory. we use pants, but we made that choice before bazel was a thing. Bazel is probably the most "scalable" solution, but it isn't always the most intuitive/developer-friendly. It's worth exploring the alternatives (biggest ones imo are pants, buck) to see what you like the most. They're all pretty similar, but Bazel has some hard-to-replicate bells and whistles that mean it'll probably be the eventual winner.
- KerrickStaley 6y ago
  Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need to. You may also want a service like Artifactory for hosting binary blobs that feed into the build process.
  - qznc 6y ago
    I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significantly.
    - gen220 6y ago
      Interesting... when you say fetch, are you talking about a cold-fetch? `du -sh repo` for me is 15GB, and we don't experience the slowness you describe. If I'm checking out a branch that diverged from master 1 year ago, it takes maybe... 30 seconds? But I don't know, maybe we've cast different git-spells from you all? We don't use sub-modules, for example.
      - qznc 6y ago
        I assume to some degree our Bitbucket server is slow. Not sure if Bitbucket is the culprit or our IT. One the other hand, git-status is also slow like 30 seconds. That is something I rather attribute to Windows. Doing the same inside WSL is faster. Some people invested quite some time to speed it up. So apparently it isn't some low hanging fruit.
    - ptsneves 6y ago
      Have you considered making shallow clones? Also indeed you are doing many things that harm usability. I am surprised though you are saying you have issues with blobs even in LFS.
voxl 6y ago
It's strange that you make this incorrect comparison to static and dynamic typing. As with static typing, an automated tool prevents you from doing something. It comes for free at the cost of you needing to satisfy the tool. Where is this in a monorepo? You make it sound like the company itself must invest in assuring that the rules are not broken. That sounds like _dynamic_ typing to me.
- triceratops 6y ago
  > As with static typing, an automated tool prevents you from doing something. It comes for free at the cost of you needing to satisfy the tool. Where is this in a monorepo? In a monorepo if your change breaks someone else's code, you'll know immediately because you won't be able to check in the code with a breaking build. If you have the correct monorepo tooling set up it will help you figure out what you're breaking. With multi-repos others will have to upgrade to your new version and then see that their code is breaking. GP's analogy was imperfect because as you pointed out the company has to invest in the monorepo tooling. But it makes sense to me.
- gen220 6y ago
  > Where is this in a monorepo? You make it sound like the company itself must invest in assuring that the rules are not broken. My argument, as made in the original comment, is that a monorepo is not a git repository. It is a git repository, a build tool / dependency graph manager, and a continuous integration system. In the same way, a statically-typed project is not a collection of source files. It is a collection of source files, a linker, and a compiler. The linker and compiler do not come "for free" any more than a build tool / ci system do, although they have been around for a very long time so you might consider them part of the background. In a well-structured monorepo, the build tool and ci system are as much in the background as `gcc` or `go build` might be.

by gen220 6y ago

qznc 6y ago

What would be a scalable open-source monorepo stack? Git, Gitlab, Bazel?

gen220 6y ago
This is more or less what we use at my current place (100s of engineers). We haven't run into any dramatic issues at this scale. Our architecture is arcanist/phab over git --> stash (git) --> custom build server that's basically identical to gitlab --> artifactory. we use pants, but we made that choice before bazel was a thing. Bazel is probably the most "scalable" solution, but it isn't always the most intuitive/developer-friendly. It's worth exploring the alternatives (biggest ones imo are pants, buck) to see what you like the most. They're all pretty similar, but Bazel has some hard-to-replicate bells and whistles that mean it'll probably be the eventual winner.
KerrickStaley 6y ago
Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need to. You may also want a service like Artifactory for hosting binary blobs that feed into the build process.
- qznc 6y ago
  I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significantly.
  - gen220 6y ago
    Interesting... when you say fetch, are you talking about a cold-fetch? `du -sh repo` for me is 15GB, and we don't experience the slowness you describe. If I'm checking out a branch that diverged from master 1 year ago, it takes maybe... 30 seconds? But I don't know, maybe we've cast different git-spells from you all? We don't use sub-modules, for example.
    - qznc 6y ago
      I assume to some degree our Bitbucket server is slow. Not sure if Bitbucket is the culprit or our IT. One the other hand, git-status is also slow like 30 seconds. That is something I rather attribute to Windows. Doing the same inside WSL is faster. Some people invested quite some time to speed it up. So apparently it isn't some low hanging fruit.
  - ptsneves 6y ago
    Have you considered making shallow clones? Also indeed you are doing many things that harm usability. I am surprised though you are saying you have issues with blobs even in LFS.

voxl 6y ago

It's strange that you make this incorrect comparison to static and dynamic typing. As with static typing, an automated tool prevents you from doing something. It comes for free at the cost of you needing to satisfy the tool. Where is this in a monorepo? You make it sound like the company itself must invest in assuring that the rules are not broken. That sounds like _dynamic_ typing to me.

triceratops 6y ago
> As with static typing, an automated tool prevents you from doing something. It comes for free at the cost of you needing to satisfy the tool. Where is this in a monorepo? In a monorepo if your change breaks someone else's code, you'll know immediately because you won't be able to check in the code with a breaking build. If you have the correct monorepo tooling set up it will help you figure out what you're breaking. With multi-repos others will have to upgrade to your new version and then see that their code is breaking. GP's analogy was imperfect because as you pointed out the company has to invest in the monorepo tooling. But it makes sense to me.
gen220 6y ago
> Where is this in a monorepo? You make it sound like the company itself must invest in assuring that the rules are not broken. My argument, as made in the original comment, is that a monorepo is not a git repository. It is a git repository, a build tool / dependency graph manager, and a continuous integration system. In the same way, a statically-typed project is not a collection of source files. It is a collection of source files, a linker, and a compiler. The linker and compiler do not come "for free" any more than a build tool / ci system do, although they have been around for a very long time so you might consider them part of the background. In a well-structured monorepo, the build tool and ci system are as much in the background as `gcc` or `go build` might be.