6 ms·

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a s

• koonsolo 6y ago
  In git I fully agree, and I wonder which company successfully runs a monorepo in git. For me, I prefer git submodules, which seem to have the benefit of both monorepo and separate repo's.
  • Kipters 6y ago
    > I wonder which company successfully runs a monorepo in git. Microsoft
    • koonsolo 6y ago
      For which product is this? It can't be all of them right?
      • jedieaston 6y ago
        Windows is in one repo at Microsoft, I believe. https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
        • vorpalhex 6y ago
          Yeah but they basically wrote their own wrapper around it, which is to say Microsoft has fallen into one of their usual patterns: 1. Picking a trendy tool 2. Mis-using trendy tool 3. Rewriting trendy tool so it's no longer trendy tool but something custom and not quite standards-compliant with it's own weird behaviors and bugs 4. Complain the standard is wrong
          • Kipters 6y ago
            They wrote a wrapper around it because git took so much time for every basic operations, even a `git status` would take minutes. But still it's not a fork, you can either use it or not without impacting anyone else using the repo. And they also made it available to everyone, as how good open source citizens should do.
    • pjmlp 6y ago
      Microsoft does definitely use git, monorepo I am not so sure.
      • wikibob 6y ago
        https://www.google.com/search?q=microsoft+monorepo https://www.google.com/search?q=microsoft+monorepo https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
        • pjmlp 6y ago
          That doesn't say that everything related to Windows is developed on the same repo, just the kernel and several core components.
          • foobarian 6y ago
            How did I miss this! Can you imagine someone claiming Windows is going to be in git as few as 10 years ago? This world never ceases to amaze me. What happened to SourceSafe?
            • oblio 6y ago
              SourceSafe was never used for anything internal major. They had their own custom source control system, based on Perforce, I believe, and then on TFS and then they switched to git. SourceSafe was something inflicted on SMBs :-D
              • foobarian 6y ago
                Interesting, didn't realize that. I haven't worked there but knew some people who did, and they were very proud of dogfooding - I guess at least the OS and IDEs, though maybe not the VCS.
      • Kipters 6y ago
        There have been many posts in their tech blogs about why they chose a monorepo
    • bdcravens 6y ago
      Worth noting that due to the size of the Windows repo, they ended up having to extend git significantly https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/ https://devblogs.microsoft.com/bharry/scaling-git-and-some-b... https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit
  • michaelmcmillan 6y ago
    > I wonder which company successfully runs a monorepo in git. Google. They're not using git, but their own scm: https://research.google/pubs/pub45424/ https://research.google/pubs/pub45424/
  • dangoor 6y ago
    Khan Academy has a monorepo in git. With 10 years of history in it, it's not always pleasant but does come with some advantages.
  • bdcravens 6y ago
    I've been looking at that approach. We have a large "app" that has a lot of boundaries (parts even being in different languages), but everything is run (love being able to docker-compose up my entire stack) and deployed in unison. However, it does make for a messy single repo.
• auxym 6y ago
  I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.
  • bluGill 6y ago
    If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your answer is the right one for everyone else.
    • auxym 6y ago
      > If that happens often you have the wrong repo split. Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops. Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and multi- repos are the right solution for different problems. Just trying to expose problems I've seen happen in a massive multirepo approach. Everyone thought it was the correct solution after being burned by a massive monolith that over two decades became an entangled mess. They thought they had the perfect architecture, the "correct repo split". In practice it's almost impossible, which is a major tradeoff to be considered. Minimizin the number of repos and splitting only when absolutely necessary at least has a chance of reducing that complexity. The default should be "as few repos as possible", not "we're going multirepo, let us break this down into as many small parts as possible as our first step", which might be what the architects were used to in modular program architecture (small, reusable functions/classes).
      • vorpalhex 6y ago
• leethargo 6y ago
  Often there are implicit dependencies between (versions of) the many repos. E.g. where I work, it was decided to put test data in a separate from the code (to keep the latter repo small in size). But now, when you add a new test case, it will fail on older versions of the code. With a monorepo, you can always check out a consistent version of all parts.
  • oconnor663 6y ago
    And when your CI fails, it blames to a specific commit, rather than "the commit that triggered it, plus any commit in a dependency repo around the same time." And if you need to maintain old release branches, each one is a single branch, and your tooling doesn't need to know anything special about what branches of other repos to check out. And `git bisect` works. All of these things are possible without a monorepo, by using higher-level tooling to manage the relationship between repos. It could be `git submodules`, or Repo, or whatever. But all of those things have their own downsides.
    • pierrebai 6y ago
      git submodules tracks the version of the sub repos. So if you organize that your releases are a master repo with all needed repos as sub-modules, your sub-repo version tracking is already done fr you. All that while still allowing sub-repo to move forward faster than other repos. With a mono-repo, you can only allow some sub-system to go forward faster than others by making them live permanently in separate branches. Basically, in monorepo you have to use branches to do what separate repo do normally for free. (Then there is the issue that with a monorepo, any screw up screws everybody. You're all on the same boat, all the time.)
      • leethargo 6y ago
        Yeah, we have set up a "parent" repo with git submodules for all the individual repos. But it was an afterthought and our workflow has not really caught up to the new possibilities.
• xorcist 6y ago
  On the contrary, you should provide a (good) argument before splitting a project. One such argument might be that those parts, for example configuration code, is on such a difference cadence that it should have a different branch and release model. Or that some parts must be kept unreadable for most developers. Or that the project has simply grown too large. A good rule of thumb can be how much larger your project is than the Linux kernel itself. The kernel seems to constantly get near the number of objects that git can confortably handle without any major speedbumps. Until then, do not bother. Be prepared that splitting a project in multiple repos will always require some sort of external tooling as soon as tickets or pull request workflows span multiple repos.
  • wst_ 6y ago
    Depends what are you working on. Kernel is nothing alike enterprise environment. If you have services in your architecture, deployed separately, that is perfect argument to split your repository.
• namelosw 6y ago
  It's about having boundaries that make sense. The best scenario is having multiple codebases, while when someone wants to change something, it can be done by just modifying code in one repo. However, the worst scenario is also having multiple codebases, but when someone wants to change something, they end up having to go through many codebases and have to deal with all the environment setup, coordination, and possibly the worst of them all - the versions. Monorepo is a straightforward way to avoid the best and worst scenario at the same time. So you'll give up the codebase splitting game and never worry about them, then focus on things that matter more.
• cryptonector 6y ago
  The biggest barrier to monorepos is size, which is a problem for `git log` and related (e.g., `git blame`). The Microsoft work on Git will make a lot of that trouble go away, but I suppose there will always be some issues to working with a huge monorepo. Q: But what's the alternative? A: A repo forest, or a repo web. Well, repo forests/webs have similar issues crop up anyways. You could say that whether a mega-codebase is spread over a mega-monorepo or a mega-repoforest doesn't change the fact that it's a mega-codebase -- big comes with problems no matter what. And many enterprises can't help getting to have megacodebases. Operating systems (including distros in the Linux case) are huge. So are ecosystems for various popular programming languages. Enterprise apps easily add up to more in large enterprises. So once you're dealing with mega-codebases... if you can find a mega-monorepo that works, that's probably where you want to be.
• bregma 6y ago
  I read somewhere on the internet that Google uses a monorepo. I want to be like Google, so I need to use a monorepo.
• alexbanks 6y ago
  I feel like the library needed is actually the opposite. Take a monorepo, do some code analysis, split into many repositories. Monorepos are fool's gold unless you can have a team whose sole job is managing monorepo complexity, and even then I might ask "Why waste a team on monorepos?"
  • danenania 6y ago
    The problem is that we have crammed many related but distinct concepts--versioning, package management, access control, issue tracking, project management, licensing, etc.--into a single envelope called the "repo". Having a single top level version and commit-log for all an organization's code is a huge win. But that doesn't mean you necessarily want to manage those other more granular concerns at the top level too. With tooling that does a better job drawing these boundaries, we should be able to have the best of both worlds (instead of the worst, which is where we are with the currently dominant repo-management tools).
    • alexbanks 6y ago
      I don't really agree. Rather than stuffing all your code into one envelope and then building tooling to decipher it, why not build tooling that allows commit log and versioning information for an arbitrary amount of repositories? Why does code have to live side-by-side to manage metadata about it?
      • danenania 6y ago
        That works too. I think you could arrive at the ideal setup from either direction: a monorepo with appropriate boundaries or multi-repos with an “umbrella” versioning layer for all of them. The end result is more or less the same.
        • alexbanks 6y ago
          I guess I think one of those scenarios comes with all of the benefits but very few of the drawbacks than the other. But you're right, you can skin a cat however you want.