5 ms·

“why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo?” Didn’t google have a monorepo before git was created? An

“why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo?” Didn’t google have a monorepo before git was created? And was created by academics? Legacy and momentum have a strong influence on the future. Hasn’t google also built a lot of tools for the monorepo and dedicates employees to it? That’s exactly the issue this article is about. From an external perspective, the speed and scale of product rollouts from the bigger tech companies is very slow. I don’t know if the tooling has much to do with it, but I suspect it might. I’ve heard some horror stories (some from here) about how it takes months to get small changes into production.

titanomachy 8y ago
Does Google require more engineers to support their build system than they would with a polyrepo? That question is not trivial to answer, IMO.
- ashelmire 8y ago
  Any is more than 0 though. In my experience (probably shared by many devs), polyrepos don’t require a team, or even a single person, dedicated to version control. It’s a minor part of the software management (usually: “mind if I create a new repo for this?” “Yes/no”). It does affect dependency management but no more than any external dependency.
  - thedufer 8y ago
    A polyrepo setup at Google's scale would pretty obviously require some dev work. For example, their CI/build story would be way more complex.
    - swish_bob 8y ago
      While that may be true, I'm not convinced it is a given. Any complicated enough monorepo requires complex CI/build tools, and Bazel/Blaze exist for a reason ...
      - zeveb 8y ago
        > Any complicated enough monorepo requires complex CI/build tools Any complicated multi-repo setup requires tooling, processes, procedures, cross-repo PRs & issue tracking, &c. &c. &c. The question is: which requires less cost in order to deliver business value? In my experience, on the teams I've been so far, the answer has been monorepos — but I don't know everything.
      - joshuamorton 8y ago
        At Google scale, you'd either need tooling for automated version bumps, or some other infra to manage versioning. You'd need cross repo bisection. You'd need a way to run all tests in all repos reflecting a new change. There's 10s or hundreds more frs I could list.
        ashelmire 8y ago
        “You'd need a way to run all tests in all repos reflecting a new change.” You really shouldn’t have to run every test on every product. Or really any other repos. Use semantic versioning, pin your dependencies, don’t make breaking changes on patch or minor versions.
        joshuamorton 8y ago
        Pinning your dependencies is an antipattern (or at least in the eyes of many people who support monorepos it is). It results in one of three things: 1. People never update their dependencies. This is bad (consider a security issue in a dependency) 2. Client teams are forced to take on the work of updating due to breaking changes in their dependencies. If they don't, we're back at 1. 3. Library teams are forced to backport security updates to N versions that are used across the company. But really, the question to ask is >don’t make breaking changes on patch or minor versions How can you be sure you aren't breaking anyone without running their code? You can be sure you aren't violating your defined APIs, but unless you're perfect, your API isn't, and there are undocumented invariants that you may change. Those break your users. Monorepo says that that was your responsibility, and therefore its your job to help them fix it. Polyrepo says that you don't need to care about that, you can just semver major version bump and be done with it, upgrading be damned. No semver means that you, not your users, feel the pain of making breaking changes. That's invariably a good thing.
    - ashelmire 8y ago
      The nice thing about polyrepo is that each repo doesn't care what the others are doing. They can use whatever tools they prefer (whatever is least work and most familiar for the team, perhaps). It might also encourage better documentation and adherence to good practices with regard to deprecation and maintenance.
bborud 8y ago
If you think companies like Google are slow, you should look at the enterprise world. I do some technical due diligence work from time to time, and although things have gotten better of the years it wasn't that long ago that there was a more than 50% chance of seeing companies that only have source code repositories at a team level and 20% chance of them not even using a VCS to manage code. I work for a company now where top management doesn't even understand what a repository is and what role it plays in software development. Yes it is that bad in much of the "entreprise" world. Yes, Google had a monorepo before git was in widespread use. They used Perforce while I was there, which was a miserable, miserable experience. It only worked because they poured engineering effort into making it somewhat tolerable. I think it would be wrong to say Google chose a monorepo because it was the best choice. To be honest, I don't think they really planned how to deal with many thousands of developers when they made the choice. They just did what seemed to make sense at the time and then had to make it work as the challenges started to mount.
marcosdumay 8y ago
> I don’t know if the tooling has much to do with it, but I suspect it might. As a development team grows, time to market also grows, in a superlinear fashion. This is known since people shared code on dead-tree pages, so the odds of tooling being the cause are low.

titanomachy 8y ago

Does Google require more engineers to support their build system than they would with a polyrepo? That question is not trivial to answer, IMO.

ashelmire 8y ago
Any is more than 0 though. In my experience (probably shared by many devs), polyrepos don’t require a team, or even a single person, dedicated to version control. It’s a minor part of the software management (usually: “mind if I create a new repo for this?” “Yes/no”). It does affect dependency management but no more than any external dependency.
- thedufer 8y ago
  A polyrepo setup at Google's scale would pretty obviously require some dev work. For example, their CI/build story would be way more complex.
  - swish_bob 8y ago
    While that may be true, I'm not convinced it is a given. Any complicated enough monorepo requires complex CI/build tools, and Bazel/Blaze exist for a reason ...
    - zeveb 8y ago
      > Any complicated enough monorepo requires complex CI/build tools Any complicated multi-repo setup requires tooling, processes, procedures, cross-repo PRs & issue tracking, &c. &c. &c. The question is: which requires less cost in order to deliver business value? In my experience, on the teams I've been so far, the answer has been monorepos — but I don't know everything.
    - joshuamorton 8y ago
      At Google scale, you'd either need tooling for automated version bumps, or some other infra to manage versioning. You'd need cross repo bisection. You'd need a way to run all tests in all repos reflecting a new change. There's 10s or hundreds more frs I could list.
      - ashelmire 8y ago
        “You'd need a way to run all tests in all repos reflecting a new change.” You really shouldn’t have to run every test on every product. Or really any other repos. Use semantic versioning, pin your dependencies, don’t make breaking changes on patch or minor versions.
        joshuamorton 8y ago
        Pinning your dependencies is an antipattern (or at least in the eyes of many people who support monorepos it is). It results in one of three things: 1. People never update their dependencies. This is bad (consider a security issue in a dependency) 2. Client teams are forced to take on the work of updating due to breaking changes in their dependencies. If they don't, we're back at 1. 3. Library teams are forced to backport security updates to N versions that are used across the company. But really, the question to ask is >don’t make breaking changes on patch or minor versions How can you be sure you aren't breaking anyone without running their code? You can be sure you aren't violating your defined APIs, but unless you're perfect, your API isn't, and there are undocumented invariants that you may change. Those break your users. Monorepo says that that was your responsibility, and therefore its your job to help them fix it. Polyrepo says that you don't need to care about that, you can just semver major version bump and be done with it, upgrading be damned. No semver means that you, not your users, feel the pain of making breaking changes. That's invariably a good thing.
  - ashelmire 8y ago
    The nice thing about polyrepo is that each repo doesn't care what the others are doing. They can use whatever tools they prefer (whatever is least work and most familiar for the team, perhaps). It might also encourage better documentation and adherence to good practices with regard to deprecation and maintenance.

bborud 8y ago

If you think companies like Google are slow, you should look at the enterprise world. I do some technical due diligence work from time to time, and although things have gotten better of the years it wasn't that long ago that there was a more than 50% chance of seeing companies that only have source code repositories at a team level and 20% chance of them not even using a VCS to manage code. I work for a company now where top management doesn't even understand what a repository is and what role it plays in software development. Yes it is that bad in much of the "entreprise" world. Yes, Google had a monorepo before git was in widespread use. They used Perforce while I was there, which was a miserable, miserable experience. It only worked because they poured engineering effort into making it somewhat tolerable. I think it would be wrong to say Google chose a monorepo because it was the best choice. To be honest, I don't think they really planned how to deal with many thousands of developers when they made the choice. They just did what seemed to make sense at the time and then had to make it work as the challenges started to mount.

marcosdumay 8y ago

> I don’t know if the tooling has much to do with it, but I suspect it might. As a development team grows, time to market also grows, in a superlinear fashion. This is known since people shared code on dead-tree pages, so the odds of tooling being the cause are low.