7 ms·

Hilariously misguided. Pretty funny to read that the things I do every day are impossible. Monorepo and tight coupling are orthogonal issues. Limits on coupli

Hilariously misguided. Pretty funny to read that the things I do every day are impossible. Monorepo and tight coupling are orthogonal issues. Limits on coupling come from the build system, not from the source repository. Yes, you should assume there is a sophisticated "VFS". What is this "checkout" you speak of? I have no time for that. I am too busy grepping the entire code base, which is apparently not possible. If the "the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo", then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical.

kiallmacinnes 8y ago
> then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical. I think, once you've chosen a path of mono or poly, you have quite a challenge ahead of you to migrate to the other. At that point, the tradeoffs arent based purely on the technical benefits - and "invest in monorepo tooling" may become a perfectly valid decision, as it's cheaper than "migrate to a polyrepo setup'. I'm not arguing either way for or against monorepo, just pointing out that "must be a good idea because Google does it" is invalid - technical merit is just one of the thousands of concerns to be balanced.
- nindalf 8y ago
  This makes sense to me. If you're considering a monorepo with millions of lines of code used concurrently by thousands of developers, it's absolutely possible. You just need a handful of developers working on the infra to make it happen. I do agree with GP though. I wish the author hadn't decided the things I do everyday are impossible.
  - oblio 8y ago
    > You just need a handful of developers working on the infra to make it happen. With thousands of developers banging on the code base, it's going to be more than "a handful of developers". It's going to be at least a few "handfuls" of developers full time plus probably many, many other full time equivalents spread out throughout the whole user base (testing, supporting other users, etc.).
    - swiftcoder 8y ago
      In reality, it's several hundred developers working full time on infra, and they are all overworked, and gradually falling behind. Monorepos at the scale of Google/Facebook are hard. We aren't talking about maintaining Mercurial here - we are talking about developing a brand new distributed VCS that happens to be 'Mercurial-compatible', and deploying/maintaining it for tens of thousands of developers working simultaneously.
      - bluGill 8y ago
        Development with thousands of developers is hard. The problems with monorepo and polyrepo are only subtly different. Either way you need a fairly large team just to handle the tools you need to solve your problems. Some of your problems will be because of repo organization (again, both choices have downsides that you need custom tooling to solve). Note that most of your problems will be related to having a thousands of developers and repo organization is irrelevant.
- rossjudson 8y ago
  I'd refine this to say "it's a good idea the way Google does it".
ashelmire 8y ago
“why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo?” Didn’t google have a monorepo before git was created? And was created by academics? Legacy and momentum have a strong influence on the future. Hasn’t google also built a lot of tools for the monorepo and dedicates employees to it? That’s exactly the issue this article is about. From an external perspective, the speed and scale of product rollouts from the bigger tech companies is very slow. I don’t know if the tooling has much to do with it, but I suspect it might. I’ve heard some horror stories (some from here) about how it takes months to get small changes into production.
- titanomachy 8y ago
  Does Google require more engineers to support their build system than they would with a polyrepo? That question is not trivial to answer, IMO.
  - ashelmire 8y ago
    Any is more than 0 though. In my experience (probably shared by many devs), polyrepos don’t require a team, or even a single person, dedicated to version control. It’s a minor part of the software management (usually: “mind if I create a new repo for this?” “Yes/no”). It does affect dependency management but no more than any external dependency.
    - thedufer 8y ago
      A polyrepo setup at Google's scale would pretty obviously require some dev work. For example, their CI/build story would be way more complex.
      - swish_bob 8y ago
        While that may be true, I'm not convinced it is a given. Any complicated enough monorepo requires complex CI/build tools, and Bazel/Blaze exist for a reason ...
LaGrange 8y ago
3) Google is committed to a monorepo to the point migrating away from it would be unpractical. Truth is, ending up with a monorepo is _really easy_. It usually starts with something that doesn't even _feel_ like more than one project: backend code, frontend templates and some celery/whatever tasks, maybe some minor utility CLI tools. And this happens at the stage nobody wants to even _think_ about more than one git repository. Once those are big enough, it's likely too late. But hey, you can always claim _you wanted it that way_. My cats always look good while pulling that one.
- banachtarski 8y ago
  I've worked with both monorepo orgs, and polyrepo orgs. I think if you have only used git as your VCS and not something like perforce, you're likely to get the wrong idea of how it works. Both CAN work, but for internal organizations with a reasonably sized team, I've come to realize that a mono repo is better. You attain "separation" by establishing different views of the code/data and at scale, the mental model of what's happening is much simpler.
  - LaGrange 8y ago
    > I think if you have only used git as your VCS and not something like perforce I think if you worked with Perforce, you're likely to get the wrong idea that people who dislike monorepos didn't work with Perforce. But the reality is that anyone who worked in this industry long enough did at some point end up traumatised by it, thanks. > the mental model of what's happening is much simpler. How does introducing the concept of "views" to the VCS model make anything simpler?
dtech 8y ago
3) A monorepo with significant investment in ecosystem and tooling is a better choise than a polyrepo For other (smaller) companies, polyrepo might be the better choice because [significant investment in ecosystem and tooling] is not appealing, and the investments of Google et al. have not leaked through sufficiently into general available tools. Some headway is being made in the latter [1], so monorepo might be the "obvious" best choice in 10 years or so. [1] For example, Git large file support is mostly from corporate contributors https://git-lfs.github.com/ https://git-lfs.github.com/ https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit
- eadmund 8y ago
  > For other (smaller) companies, polyrepo might be the better choice because [significant investment in ecosystem and tooling] is not appealing That's not the choice, though: significant investment in tooling is a function of codebase size. In my own experience, polyrepos require more tooling, because you're not just dealing with files & directories, you're also dealing with repos (& probably PRs & issues & other stuff in a forge).
  - briantakita 8y ago
    Another thing is that the OP categorized "medium sized menorepos" as being too large to fit on a laptop. Maybe the companies I've worked for is limited (mainly startups & consultancies), but the entire codebase, after several years of operation, still easily fits on a single laptop. Of course there is a threshold, however this is typically a concern of a large organization or an organization that has been producing software for a decade or more.
  - dtech 8y ago
    > In my own experience, polyrepos require more tooling That's not my experience. In my experience, polyrepo's significantly reduce complexity for a medium (30 developers) project. An example: the following things are good software development practices if you work with a master-PR branch model: 1. Tests must pass on CI before merging a branch to master 2. Before merging a branch into master, the latest master must be merged into the branch so that tests are still reliable This quickly becomes untenable if 30 people all commit to the same repo. By the time your PR is reviewed, it's outdated. So you merge master into your branch. By the time you come back to check your test results and merge, its outdated again. Repeat until 6 PM. So you need partial builds to keep build time low, and would probably like to amend 1 & 2 with "unless your code has zero overlap with the changes in master". These are not standard features of any CI system I know of, hence the need for tooling. Instead of tooling, polyrepo's provide the above benefits out-of-the-box. Just set your CI to build the repo, and it will do partial builds and PR-merging is uninfluenced by other repos. This is a huge advantage over monorepos. The downside is that if your repos have tight coupling, you'll need simultaneous PRs in more than 1 place or need to look up history/files in more than 1 repo. If this is more than a rare occurence, this downside is so large that polyrepo is not a suitable solution for your project. The projects of this size I've worked with did not have this problem, or the problem was solvable without much difficulty.
bborud 8y ago
It can work. That doesn't mean it is a universal solution. And it doesn't even mean it is a solution that is guaranteed to cover most projects. Whether or not a monorepo works depends on a lot of factors. In my experience the number of cases where it doesn't work appears to outnumber the cases where it works. It can work nicely when you have disciplined and demonstrably above average programmers that are good at structuring the internal architecture of systems and will know how to design for plasticity. It is also an advantage if all your code is written in the same style and doesn't come from a bunch of older codebases. But even then you can end up with messes that you will be likely to conveniently forget about. For instance while clear decoupling was a goal when I worked at Google, it wasn't always a reality. There were still lots of very deep and direct dependencies that should never have been there. It does not work well if you have "average" developers or if you have undisciplined developers or excessive bikeshedders (which kill productivity). Then there is the tooling. Most people do not work for Google and do not have the ability to spend as much money and time on tooling as Google does. What Google does largely works because of the tooling. It would suck balls without it. To be honest: some things sucked balls even with the tooling. Especially when working with people in different time zones. Google isn't really a valid example of why monorepo is a good idea because your average company isn't going to have a support structure even remotely as huge as Google. (If you disagre: hey, it's easy, go work for Google for a while and then tell me I'm wrong)
- Eridrus 8y ago
  At some point blaze added the concept of visibility to control dependencies, so teams can whitelist users of their code. Though you can always comment it out while developing locally.
hinkley 8y ago
> As described above, at scale, a developer will not be able to easily edit or search the entirety of the codebase on their local machine. Thus, the idea that one can clone all of the code and simply do a grep/replace is not trivial in practice. Yeah this is a pretty widespread and fundamental misunderstanding that leads to a lot of bad policy decisions. If 'grepping code' is your first resort then you're hitting things with a hammer. I'm writing code that a machine is supposed to understand. If the machine can't understand how the bits interact then I have much bigger problems than where my code is stored. Probably we're dealing with a lot of toxic machismo bullshit that is hurting our ability to deliver. If you want discipline, if you want cooperation, hell if you just want to be able to hire a bunch of new people when you land a big customer, you need some form of support for static analysis and the code navigation that it enables. Stop the propeller heads from using magic and runtime inference to wire up the parts of the system, or find a new gig. Even languages where static typing isn't a thing have documentation frameworks where you can provide hints that your IDE can understand (ex: jsdoc for Javascript). For a large team, working without any kind of static analysis is a recipe for a rigid oligarchy. Only people who have memorized the system can reason about it. Everybody else who tries to make ambitious changes ends up breaking something. See what happens when you trust new people with new ideas? New is bad. Be safe with us. And even if by some miracle you do make the change without blowing stuff up, you're still in the doghouse, because we have memorized the old way and you are disrupting things! Some crazy ideas work well. Some reasonable ideas fail horribly. To grow, people need the space to tinker and an opaque codebase ruins those opportunities. Transparency is also helpful when debugging a production issue, because people can work in parallel to the people most likely to solve the problem (even the person who is usually right is way off base occasionally). I should be able to learn and possibly contribute without jamming up the rest of the team by asking inane questions. You need pretty good but entirely achievable tooling and architecture to get that, but man when you do it's like getting over a cold and remembering what breathing feels like.

kiallmacinnes 8y ago

> then why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo? Choices: 1) Google is dumb. 2) Mono and poly are not identical. I think, once you've chosen a path of mono or poly, you have quite a challenge ahead of you to migrate to the other. At that point, the tradeoffs arent based purely on the technical benefits - and "invest in monorepo tooling" may become a perfectly valid decision, as it's cheaper than "migrate to a polyrepo setup'. I'm not arguing either way for or against monorepo, just pointing out that "must be a good idea because Google does it" is invalid - technical merit is just one of the thousands of concerns to be balanced.

nindalf 8y ago
This makes sense to me. If you're considering a monorepo with millions of lines of code used concurrently by thousands of developers, it's absolutely possible. You just need a handful of developers working on the infra to make it happen. I do agree with GP though. I wish the author hadn't decided the things I do everyday are impossible.
- oblio 8y ago
  > You just need a handful of developers working on the infra to make it happen. With thousands of developers banging on the code base, it's going to be more than "a handful of developers". It's going to be at least a few "handfuls" of developers full time plus probably many, many other full time equivalents spread out throughout the whole user base (testing, supporting other users, etc.).
  - swiftcoder 8y ago
    In reality, it's several hundred developers working full time on infra, and they are all overworked, and gradually falling behind. Monorepos at the scale of Google/Facebook are hard. We aren't talking about maintaining Mercurial here - we are talking about developing a brand new distributed VCS that happens to be 'Mercurial-compatible', and deploying/maintaining it for tens of thousands of developers working simultaneously.
    - bluGill 8y ago
      Development with thousands of developers is hard. The problems with monorepo and polyrepo are only subtly different. Either way you need a fairly large team just to handle the tools you need to solve your problems. Some of your problems will be because of repo organization (again, both choices have downsides that you need custom tooling to solve). Note that most of your problems will be related to having a thousands of developers and repo organization is irrelevant.
rossjudson 8y ago
I'd refine this to say "it's a good idea the way Google does it".

ashelmire 8y ago

“why on earth would google invest enormous effort constructing an entire ecosystem around a monorepo?” Didn’t google have a monorepo before git was created? And was created by academics? Legacy and momentum have a strong influence on the future. Hasn’t google also built a lot of tools for the monorepo and dedicates employees to it? That’s exactly the issue this article is about. From an external perspective, the speed and scale of product rollouts from the bigger tech companies is very slow. I don’t know if the tooling has much to do with it, but I suspect it might. I’ve heard some horror stories (some from here) about how it takes months to get small changes into production.

titanomachy 8y ago
Does Google require more engineers to support their build system than they would with a polyrepo? That question is not trivial to answer, IMO.
- ashelmire 8y ago
  Any is more than 0 though. In my experience (probably shared by many devs), polyrepos don’t require a team, or even a single person, dedicated to version control. It’s a minor part of the software management (usually: “mind if I create a new repo for this?” “Yes/no”). It does affect dependency management but no more than any external dependency.
  - thedufer 8y ago
    A polyrepo setup at Google's scale would pretty obviously require some dev work. For example, their CI/build story would be way more complex.
    - swish_bob 8y ago
      While that may be true, I'm not convinced it is a given. Any complicated enough monorepo requires complex CI/build tools, and Bazel/Blaze exist for a reason ...

LaGrange 8y ago

3) Google is committed to a monorepo to the point migrating away from it would be unpractical. Truth is, ending up with a monorepo is _really easy_. It usually starts with something that doesn't even _feel_ like more than one project: backend code, frontend templates and some celery/whatever tasks, maybe some minor utility CLI tools. And this happens at the stage nobody wants to even _think_ about more than one git repository. Once those are big enough, it's likely too late. But hey, you can always claim _you wanted it that way_. My cats always look good while pulling that one.

banachtarski 8y ago
I've worked with both monorepo orgs, and polyrepo orgs. I think if you have only used git as your VCS and not something like perforce, you're likely to get the wrong idea of how it works. Both CAN work, but for internal organizations with a reasonably sized team, I've come to realize that a mono repo is better. You attain "separation" by establishing different views of the code/data and at scale, the mental model of what's happening is much simpler.
- LaGrange 8y ago
  > I think if you have only used git as your VCS and not something like perforce I think if you worked with Perforce, you're likely to get the wrong idea that people who dislike monorepos didn't work with Perforce. But the reality is that anyone who worked in this industry long enough did at some point end up traumatised by it, thanks. > the mental model of what's happening is much simpler. How does introducing the concept of "views" to the VCS model make anything simpler?

dtech 8y ago

3) A monorepo with significant investment in ecosystem and tooling is a better choise than a polyrepo For other (smaller) companies, polyrepo might be the better choice because [significant investment in ecosystem and tooling] is not appealing, and the investments of Google et al. have not leaked through sufficiently into general available tools. Some headway is being made in the latter [1], so monorepo might be the "obvious" best choice in 10 years or so. [1] For example, Git large file support is mostly from corporate contributors https://git-lfs.github.com/ https://git-lfs.github.com/ https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit

eadmund 8y ago
> For other (smaller) companies, polyrepo might be the better choice because [significant investment in ecosystem and tooling] is not appealing That's not the choice, though: significant investment in tooling is a function of codebase size. In my own experience, polyrepos require more tooling, because you're not just dealing with files & directories, you're also dealing with repos (& probably PRs & issues & other stuff in a forge).
- briantakita 8y ago
  Another thing is that the OP categorized "medium sized menorepos" as being too large to fit on a laptop. Maybe the companies I've worked for is limited (mainly startups & consultancies), but the entire codebase, after several years of operation, still easily fits on a single laptop. Of course there is a threshold, however this is typically a concern of a large organization or an organization that has been producing software for a decade or more.
- dtech 8y ago
  > In my own experience, polyrepos require more tooling That's not my experience. In my experience, polyrepo's significantly reduce complexity for a medium (30 developers) project. An example: the following things are good software development practices if you work with a master-PR branch model: 1. Tests must pass on CI before merging a branch to master 2. Before merging a branch into master, the latest master must be merged into the branch so that tests are still reliable This quickly becomes untenable if 30 people all commit to the same repo. By the time your PR is reviewed, it's outdated. So you merge master into your branch. By the time you come back to check your test results and merge, its outdated again. Repeat until 6 PM. So you need partial builds to keep build time low, and would probably like to amend 1 & 2 with "unless your code has zero overlap with the changes in master". These are not standard features of any CI system I know of, hence the need for tooling. Instead of tooling, polyrepo's provide the above benefits out-of-the-box. Just set your CI to build the repo, and it will do partial builds and PR-merging is uninfluenced by other repos. This is a huge advantage over monorepos. The downside is that if your repos have tight coupling, you'll need simultaneous PRs in more than 1 place or need to look up history/files in more than 1 repo. If this is more than a rare occurence, this downside is so large that polyrepo is not a suitable solution for your project. The projects of this size I've worked with did not have this problem, or the problem was solvable without much difficulty.

bborud 8y ago

It can work. That doesn't mean it is a universal solution. And it doesn't even mean it is a solution that is guaranteed to cover most projects. Whether or not a monorepo works depends on a lot of factors. In my experience the number of cases where it doesn't work appears to outnumber the cases where it works. It can work nicely when you have disciplined and demonstrably above average programmers that are good at structuring the internal architecture of systems and will know how to design for plasticity. It is also an advantage if all your code is written in the same style and doesn't come from a bunch of older codebases. But even then you can end up with messes that you will be likely to conveniently forget about. For instance while clear decoupling was a goal when I worked at Google, it wasn't always a reality. There were still lots of very deep and direct dependencies that should never have been there. It does not work well if you have "average" developers or if you have undisciplined developers or excessive bikeshedders (which kill productivity). Then there is the tooling. Most people do not work for Google and do not have the ability to spend as much money and time on tooling as Google does. What Google does largely works because of the tooling. It would suck balls without it. To be honest: some things sucked balls even with the tooling. Especially when working with people in different time zones. Google isn't really a valid example of why monorepo is a good idea because your average company isn't going to have a support structure even remotely as huge as Google. (If you disagre: hey, it's easy, go work for Google for a while and then tell me I'm wrong)

Eridrus 8y ago
At some point blaze added the concept of visibility to control dependencies, so teams can whitelist users of their code. Though you can always comment it out while developing locally.

hinkley 8y ago

> As described above, at scale, a developer will not be able to easily edit or search the entirety of the codebase on their local machine. Thus, the idea that one can clone all of the code and simply do a grep/replace is not trivial in practice. Yeah this is a pretty widespread and fundamental misunderstanding that leads to a lot of bad policy decisions. If 'grepping code' is your first resort then you're hitting things with a hammer. I'm writing code that a machine is supposed to understand. If the machine can't understand how the bits interact then I have much bigger problems than where my code is stored. Probably we're dealing with a lot of toxic machismo bullshit that is hurting our ability to deliver. If you want discipline, if you want cooperation, hell if you just want to be able to hire a bunch of new people when you land a big customer, you need some form of support for static analysis and the code navigation that it enables. Stop the propeller heads from using magic and runtime inference to wire up the parts of the system, or find a new gig. Even languages where static typing isn't a thing have documentation frameworks where you can provide hints that your IDE can understand (ex: jsdoc for Javascript). For a large team, working without any kind of static analysis is a recipe for a rigid oligarchy. Only people who have memorized the system can reason about it. Everybody else who tries to make ambitious changes ends up breaking something. See what happens when you trust new people with new ideas? New is bad. Be safe with us. And even if by some miracle you do make the change without blowing stuff up, you're still in the doghouse, because we have memorized the old way and you are disrupting things! Some crazy ideas work well. Some reasonable ideas fail horribly. To grow, people need the space to tinker and an opaque codebase ruins those opportunities. Transparency is also helpful when debugging a production issue, because people can work in parallel to the people most likely to solve the problem (even the person who is usually right is way off base occasionally). I should be able to learn and possibly contribute without jamming up the rest of the team by asking inane questions. You need pretty good but entirely achievable tooling and architecture to get that, but man when you do it's like getting over a cold and remembering what breathing feels like.