5 ms·

Why not just have bunch of repos? You can use what google does with chrome and use depot tools to checkout from a bunch of repos. Edit: Why am I getting downvo

Why not just have bunch of repos? You can use what google does with chrome and use depot tools to checkout from a bunch of repos. Edit: Why am I getting downvoted for asking a question? We use multiple repos and we have a pretty large codebase consuming a TB a day.

q3k 6y ago
Because you lose the 'single head' concept of actually being able to always have a clear view of the current newest version and of linear version history. The ability to map a single revision number into the state of _all_ source code (including third party dependencies) is extremely powerful. You also lose the ability of performing sweeping changes across an entire codebase in lockstep (think: backwards compatibility breaking API changes). Not to mention that Google internally avoids multiple repositories (and instead runs of a single monorepo) - it's just the android/chrome/public stuff that's mostly split up.
- AlphaSite 6y ago
  Also there isnt really a lot of great tooling for making sweeping changes across repos.
- erik_seaberg 6y ago
  Don't make breaking API changes, you can't deploy them atomically. You have to support the old API until everyone has safely migrated and confirmed no rollback will be needed.
  - joshuamorton 6y ago
    There are more apis than just rpc ones. If I change my library's api, I can update the callers atomically.
  - fsociety 6y ago
    Then services would be stuck in a sea of technical debt. There are too many moving pieces to do that.
    - erik_seaberg 6y ago
      Technical debt is a tool to be used (with caution). Getting ludicrously far behind is a risk I'd avoid, but our stability is more important than staying on the bleeding edge and testing every single version of each of our dependencies.
  - djohnston 6y ago
    you can if it's a monorepo and all your callers are in the monorepo, which sort of answers OP's question
    - erik_seaberg 6y ago
      The commit can be atomic, but if prod is many machines, the deployment cannot.
- throwdbaaway 6y ago
  > performing sweeping changes across an entire codebase in lockstep This thinking right here, sounds like a reason why Google retires a lot more services at much higher frequency compared to other companies? In https://thehftguy.com/2019/12/10/why-products-are-shutdown-the-case-of-verizon-and-yahoo-groups/ https://thehftguy.com/2019/12/10/why-products-are-shutdown-t..., the HFT guy brought up the "5 years upgrade pain" as the reason why a service would be retired, at the time when the pain from upkeeping outweights the gain from revenue. By regularly making backward incompatible API change, as afforded by the monorepo, while also having engineers moving freely between projects, the upgrade pain cycle becomes way shorter for Google services. By the way, this line of thought also leaks into public facing source code, with guava being the poster child of breaking backward compatibility, althought it has learned its lesson starting with version 21.
ardit33 6y ago
We tried at Spotify the multi-repo thing and it was a total nightmare. Once you have lots of people working on a product, it becomes very hard to do changes on a multi repo setup. The alternatives: 1. Completely de-coupled code. I.e. teams ship libraries, like a 3rd part api. This slows down development considerably, and makes re-use of code harder. 2. Keep everyone on one repo, and allow to changes stream in. 1. Slows down dev. (it suddenly becomes more beourocratic), 2. requires scaling your code versioning tool (gir/mercurial, etc) and processes associated with it. Also, single repos, make sense in one domain (e.g. server side, ios, android, etc...). You can have different repos for different 'domains' where code doesn't intersect with each other that much.
- dehrmann 6y ago
  > makes re-use of code harder I've found this to be a feature of polyrepos because monorepos can easily become a rat's nest of dependencies. Polyrepos make you think harder about what should really be exposed and shared.
  - jonhohle 6y ago
    A phrase I like to use sums this up: “dependencies are easy to add, but hard to remove.”
  - joshuamorton 6y ago
    Build systems can enforce visibility, which solves this. Allowing fine grained visibility at the level of a file, package, or artifact is better than at the granularity of a repo.
    - dehrmann 6y ago
      Depends how easy it is to change visibility. My point is polyrepos make things you need to be careful about like adding internal dependencies and API changes hard. Monorepos make those deceptively easy.
      - 6y ago
klodolph 6y ago
Multiple repos... the most bureaucratic and painful way to scale your code base. Having seen "a bunch of repos" in action, I think people don't talk enough about just how awful the experience can be, and how much more work it is to manage multiple repos. As you increase the number of repos, the pain gets worse at a rate which is faster than linear. There are plenty of articles talking about how wonderful a monorepo is, just not many articles about how bad multirepo is. NPM hell is a close approximation of the multirepo experience. Try upgrading the dependencies in a large NPM project and you'll see all sorts of problems. You might find that upgrading X breaks Y, but you have to upgrade X in order to upgrade Z, and you need to upgrade Z for some reason. With monorepo, all the versions march forward in sync. If you fix trunk, you will probably ship it, eventually. With multirepo, you need to fight the tooling just to show (from the example above) that a patch to Y will let you upgrade your project to use a newer version of Z.
an_opabinia 6y ago
Giant companies consistently rather vendor the open source code they ingest rather than give back. Of course Google, Facebook etc have given back a lot to OSS. Just not extemporaneously to when they received the value. It may be years until the internal rebuild of something some guy copied from somewhere is rereleased. Git was designed for OSS. That includes its radically transparent form of development. Then again with submodules you can easily vendor your private stuff, rather than doing things the other way around.
- CJefferson 6y ago
  I have sympathy -- I "vendor" some libraries as I need to change something deep in the library. It's often not nice, and not something upstream would want.
- joshuamorton 6y ago
  This isn't really true. The Google oss policies (opensource.google) require that you use a recent version. As a result upatreaming functional changes is encouraged and easier than maintaining a local patch set. Sometimes there are local changes but they're usually non functional stuff to make the library integrate with google infra. They do vendor, but that's for security and efficiency reasons mostly, not to avoid giving back.
robocat 6y ago
Dan Luu gives a good summary of benefits of monorepo: http://danluu.com/monorepo/ http://danluu.com/monorepo/

q3k 6y ago

Because you lose the 'single head' concept of actually being able to always have a clear view of the current newest version and of linear version history. The ability to map a single revision number into the state of _all_ source code (including third party dependencies) is extremely powerful. You also lose the ability of performing sweeping changes across an entire codebase in lockstep (think: backwards compatibility breaking API changes). Not to mention that Google internally avoids multiple repositories (and instead runs of a single monorepo) - it's just the android/chrome/public stuff that's mostly split up.

AlphaSite 6y ago
Also there isnt really a lot of great tooling for making sweeping changes across repos.
erik_seaberg 6y ago
Don't make breaking API changes, you can't deploy them atomically. You have to support the old API until everyone has safely migrated and confirmed no rollback will be needed.
- joshuamorton 6y ago
  There are more apis than just rpc ones. If I change my library's api, I can update the callers atomically.
- fsociety 6y ago
  Then services would be stuck in a sea of technical debt. There are too many moving pieces to do that.
  - erik_seaberg 6y ago
    Technical debt is a tool to be used (with caution). Getting ludicrously far behind is a risk I'd avoid, but our stability is more important than staying on the bleeding edge and testing every single version of each of our dependencies.
- djohnston 6y ago
  you can if it's a monorepo and all your callers are in the monorepo, which sort of answers OP's question
  - erik_seaberg 6y ago
    The commit can be atomic, but if prod is many machines, the deployment cannot.
throwdbaaway 6y ago
> performing sweeping changes across an entire codebase in lockstep This thinking right here, sounds like a reason why Google retires a lot more services at much higher frequency compared to other companies? In https://thehftguy.com/2019/12/10/why-products-are-shutdown-the-case-of-verizon-and-yahoo-groups/ https://thehftguy.com/2019/12/10/why-products-are-shutdown-t..., the HFT guy brought up the "5 years upgrade pain" as the reason why a service would be retired, at the time when the pain from upkeeping outweights the gain from revenue. By regularly making backward incompatible API change, as afforded by the monorepo, while also having engineers moving freely between projects, the upgrade pain cycle becomes way shorter for Google services. By the way, this line of thought also leaks into public facing source code, with guava being the poster child of breaking backward compatibility, althought it has learned its lesson starting with version 21.

ardit33 6y ago

We tried at Spotify the multi-repo thing and it was a total nightmare. Once you have lots of people working on a product, it becomes very hard to do changes on a multi repo setup. The alternatives: 1. Completely de-coupled code. I.e. teams ship libraries, like a 3rd part api. This slows down development considerably, and makes re-use of code harder. 2. Keep everyone on one repo, and allow to changes stream in. 1. Slows down dev. (it suddenly becomes more beourocratic), 2. requires scaling your code versioning tool (gir/mercurial, etc) and processes associated with it. Also, single repos, make sense in one domain (e.g. server side, ios, android, etc...). You can have different repos for different 'domains' where code doesn't intersect with each other that much.

dehrmann 6y ago
> makes re-use of code harder I've found this to be a feature of polyrepos because monorepos can easily become a rat's nest of dependencies. Polyrepos make you think harder about what should really be exposed and shared.
- jonhohle 6y ago
  A phrase I like to use sums this up: “dependencies are easy to add, but hard to remove.”
- joshuamorton 6y ago
  Build systems can enforce visibility, which solves this. Allowing fine grained visibility at the level of a file, package, or artifact is better than at the granularity of a repo.
  - dehrmann 6y ago
    Depends how easy it is to change visibility. My point is polyrepos make things you need to be careful about like adding internal dependencies and API changes hard. Monorepos make those deceptively easy.
    - 6y ago

klodolph 6y ago

Multiple repos... the most bureaucratic and painful way to scale your code base. Having seen "a bunch of repos" in action, I think people don't talk enough about just how awful the experience can be, and how much more work it is to manage multiple repos. As you increase the number of repos, the pain gets worse at a rate which is faster than linear. There are plenty of articles talking about how wonderful a monorepo is, just not many articles about how bad multirepo is. NPM hell is a close approximation of the multirepo experience. Try upgrading the dependencies in a large NPM project and you'll see all sorts of problems. You might find that upgrading X breaks Y, but you have to upgrade X in order to upgrade Z, and you need to upgrade Z for some reason. With monorepo, all the versions march forward in sync. If you fix trunk, you will probably ship it, eventually. With multirepo, you need to fight the tooling just to show (from the example above) that a patch to Y will let you upgrade your project to use a newer version of Z.

an_opabinia 6y ago

Giant companies consistently rather vendor the open source code they ingest rather than give back. Of course Google, Facebook etc have given back a lot to OSS. Just not extemporaneously to when they received the value. It may be years until the internal rebuild of something some guy copied from somewhere is rereleased. Git was designed for OSS. That includes its radically transparent form of development. Then again with submodules you can easily vendor your private stuff, rather than doing things the other way around.

CJefferson 6y ago
I have sympathy -- I "vendor" some libraries as I need to change something deep in the library. It's often not nice, and not something upstream would want.
joshuamorton 6y ago
This isn't really true. The Google oss policies (opensource.google) require that you use a recent version. As a result upatreaming functional changes is encouraged and easier than maintaining a local patch set. Sometimes there are local changes but they're usually non functional stuff to make the library integrate with google infra. They do vendor, but that's for security and efficiency reasons mostly, not to avoid giving back.

robocat 6y ago

Dan Luu gives a good summary of benefits of monorepo: http://danluu.com/monorepo/ http://danluu.com/monorepo/