4 ms·

Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g.

by Nitramp 11y ago

Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g. OpenStack, and different products certainly have extremely different release cycles and even deployment platforms (hardware in cars, Android phones, web servers). The question really is whether you can scale your version control practices, build tools, and source organization habits across many diverse projects.

viraptor 11y ago
Do they really store youtube, self driving car and Android in the same repo? As far as I can tell, at least android bits live in many repositories: https://android.googlesource.com/ https://android.googlesource.com/ - I'd say over 300, with one for each utility.
- deleted 11y ago
  [deleted]
- Confusion 11y ago
  Exactly. I frankly have no clue what this article is about. I work at a company with < 10 developers. We started with one repo and now have over 20 repositories for various bits and pieces of our code. Each one maps to its own releasible component. I don't recognize the 'I don't want to waste time dealing with multiple commands to manage multiple repositories' at all. The only time there is a difference on the cli is when you clone a repo. If anything about this would hurt, we'd change it: optimizing our workflow is something we pay a lot of attention to. In fact, with hundreds of developers and everything in one repo, I don't see how you'd ever be able get a commit through : you'd be merging commits that others just did all the time and would have to get lucky?
  - teraflop 11y ago
    The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be automated. The problems arise when you have to combine code from different repositories into a single deployable product. Most of us don't take Amazon's hard-line stance of making absolutely everything a microservice, so we end up with libraries of reusable code that are referenced by multiple projects. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. That makes it easy for subtle inconsistencies and bugs to creep in, especially when the dependencies are multiple levels deep and are owned by different teams. If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. I don't really follow your comment about merging. Pretty much every version control system since forever has been smart enough to realize that, if I make changes only to foo/src/ and you make changes to bar/src/, our changes don't conflict and can be merged automatically without user intervention. (There might be technical difficulties; for example, if you're using Git, I would imagine that trying to view the list of commits of a small subtree of a gigantic repo might not be terribly efficient. But just like the issue of managing multiple repos, that's something that you can solve with better tool support, if you really need to and are motivated enough.)
    - Confusion 11y ago
      About merging (I'll get back to the other bit later): if I want to push my change, I already regularly encounter the situation where git tells me: you are not up to date, even if I pulled say 15 minutes ago. So I have to fetch and rebase before I can push. Even if that doesn't require manually solving any merge conflicts (and I think I have to in about 1/4th of the cases), it takes a bit of time. In that bit of time another developer can push a commit. If that already happens on occasion with 10 developers, I imagine it becomes very problematic with 1000 developers on the same repo. Of course you effectively organise your repos in a 'tree' like e.g. happens with the Linux kernel, but, well, then you again have many repos. One per component that has someone responsible for signing off on them. It reduces the amount of people committing to each specific repo, but in the end, it's still many repos. So that's not 'one monolithic repo' in my book.
      - teraflop 11y ago
        Still, that's a limitation of the specific tools and the way you're using them, not a fundamental problem with monolithic repositories. You're running into problems because you have multiple people all trying to push to the same branch, and Git errs on the side of extreme caution rather than creating a merge that you didn't specifically request. If you use merge requests instead, as supported by Github/Gitlab, you don't get blocked by other people's commits. (Actual merge conflicts are an orthogonal problem. They'll happen whenever you have multiple people editing the same code, regardless of what VCS you use or how it's organized.) The Linux kernel example is a bit of a tricky one. Yes, kernel.org hosts a lot of repos, but they're all different versions of the same codebase. They have a common commit ancestry, and commits get merged from one to another. So they're not really separate modules in the sense that we're talking about; they're more like namespaces that each contain a set of branches.
        Confusion 11y ago
        I guess it comes down to organisational style. Using merge requests requires some benevolent dictator (poor sod?) to perform the merges. For us, if your commit is approved in Gerrit, it's your task to merge it. If some other commit was merged in between, you have to rebase, possibly solving merge conflicts. So guess that with a monolithic repo, that wouldn't scale, but it could be made to scale by appointing someone to perform the merge requests. I'm not sure I would like to be that person...
- qznc 11y ago
  Google has a few departements that do their own thing. Android and Chrome for example. There is still a lot one the monolithic repo. I heared "1TB" when I asked a Googler. Also, they don't use Perforce anymore, at least for the central repo, because Perforce requires a single host. They built a distributed monolithic repo server themselves. (disclaimer: I have no proof for any of this and might misremember)

by Nitramp 11y ago

viraptor 11y ago

Do they really store youtube, self driving car and Android in the same repo? As far as I can tell, at least android bits live in many repositories: https://android.googlesource.com/ https://android.googlesource.com/ - I'd say over 300, with one for each utility.

deleted 11y ago
[deleted]
Confusion 11y ago
Exactly. I frankly have no clue what this article is about. I work at a company with < 10 developers. We started with one repo and now have over 20 repositories for various bits and pieces of our code. Each one maps to its own releasible component. I don't recognize the 'I don't want to waste time dealing with multiple commands to manage multiple repositories' at all. The only time there is a difference on the cli is when you clone a repo. If anything about this would hurt, we'd change it: optimizing our workflow is something we pay a lot of attention to. In fact, with hundreds of developers and everything in one repo, I don't see how you'd ever be able get a commit through : you'd be merging commits that others just did all the time and would have to get lucky?
- teraflop 11y ago
  The issue with multiple repositories has nothing do to with the number of commands you have to run. As you say, that's the kind of thing that can easily be automated. The problems arise when you have to combine code from different repositories into a single deployable product. Most of us don't take Amazon's hard-line stance of making absolutely everything a microservice, so we end up with libraries of reusable code that are referenced by multiple projects. But when you store those libraries in separate repositories, it becomes impossible to describe the state of your deployed code without listing the version of every single dependency. That makes it easy for subtle inconsistencies and bugs to creep in, especially when the dependencies are multiple levels deep and are owned by different teams. If everything lives in the same tree, then a single commit ID reproducibly describes a complete system from top to bottom. And you can atomically make changes that cross module boundaries, which is difficult to do safely with separate repositories. I don't really follow your comment about merging. Pretty much every version control system since forever has been smart enough to realize that, if I make changes only to foo/src/ and you make changes to bar/src/, our changes don't conflict and can be merged automatically without user intervention. (There might be technical difficulties; for example, if you're using Git, I would imagine that trying to view the list of commits of a small subtree of a gigantic repo might not be terribly efficient. But just like the issue of managing multiple repos, that's something that you can solve with better tool support, if you really need to and are motivated enough.)
  - Confusion 11y ago
    About merging (I'll get back to the other bit later): if I want to push my change, I already regularly encounter the situation where git tells me: you are not up to date, even if I pulled say 15 minutes ago. So I have to fetch and rebase before I can push. Even if that doesn't require manually solving any merge conflicts (and I think I have to in about 1/4th of the cases), it takes a bit of time. In that bit of time another developer can push a commit. If that already happens on occasion with 10 developers, I imagine it becomes very problematic with 1000 developers on the same repo. Of course you effectively organise your repos in a 'tree' like e.g. happens with the Linux kernel, but, well, then you again have many repos. One per component that has someone responsible for signing off on them. It reduces the amount of people committing to each specific repo, but in the end, it's still many repos. So that's not 'one monolithic repo' in my book.
    - teraflop 11y ago
      Still, that's a limitation of the specific tools and the way you're using them, not a fundamental problem with monolithic repositories. You're running into problems because you have multiple people all trying to push to the same branch, and Git errs on the side of extreme caution rather than creating a merge that you didn't specifically request. If you use merge requests instead, as supported by Github/Gitlab, you don't get blocked by other people's commits. (Actual merge conflicts are an orthogonal problem. They'll happen whenever you have multiple people editing the same code, regardless of what VCS you use or how it's organized.) The Linux kernel example is a bit of a tricky one. Yes, kernel.org hosts a lot of repos, but they're all different versions of the same codebase. They have a common commit ancestry, and commits get merged from one to another. So they're not really separate modules in the sense that we're talking about; they're more like namespaces that each contain a set of branches.
      - Confusion 11y ago
        I guess it comes down to organisational style. Using merge requests requires some benevolent dictator (poor sod?) to perform the merges. For us, if your commit is approved in Gerrit, it's your task to merge it. If some other commit was merged in between, you have to rebase, possibly solving merge conflicts. So guess that with a monolithic repo, that wouldn't scale, but it could be made to scale by appointing someone to perform the merge requests. I'm not sure I would like to be that person...
qznc 11y ago
Google has a few departements that do their own thing. Android and Chrome for example. There is still a lot one the monolithic repo. I heared "1TB" when I asked a Googler. Also, they don't use Perforce anymore, at least for the central repo, because Perforce requires a single host. They built a distributed monolithic repo server themselves. (disclaimer: I have no proof for any of this and might misremember)