5 ms·

Very well written post and upstream work is always appreciated. I also really like monorepos, but Git and GitHub really don't work well at all for them. On th

Very well written post and upstream work is always appreciated. I also really like monorepos, but Git and GitHub really don't work well at all for them. On the Git side there is no way to clone only parts of a repo or to limit access by user. All the Git tooling out there, from CLI to the various IDE integrations, are all very ill adjusted to a huge repo with lots of unrelated commits. On the Github side there is no separation between the different parts of a monorepo in the UI (issues, prs, CI), the workflows, or the permission system. Sure, you can hack something together with labels and custom bots, but it always feels like a hack. Using Git(hub) for monorepos is really painful in my experience. There is a reason why Google, Facebook et all have heaps of custom tooling.

krasin 6y ago
I really like monorepos. But I find that it's almost never a good idea to hide parts of a source code from developers. And if there's some secret sauce that's so sensitive that only single-digit number of developers in the whole company can access it, then it's probably okay to have a separate repository just for it. Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.
Beowolve 6y ago
On this note, GitHub does reach out to customers with Monorepos and is aware of their short comings. I think overtime we will see them change to have better support. It's only a matter of time.
jeffbee 6y ago
It's funny that you mention this as if monorepos of course require custom tooling. Google started with off-the-shelf Perforce and that was fine for many years, long after their repo became truly huge. Only when it became monstrously huge did they need custom tools and even then they basically just re-implemented Perforce instead of adopting git concepts. You, too, can just use Perforce. It's even free for up to five users. You won't outgrow its limits until you get about a million engineer-years under your belt. The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.
- forrestthewoods 6y ago
  It’s somewhat mind boggling that no one has made a better Perforce. It has numerous issues and warts. But it’s much closer to what the majority of projects need that Git imho. And for bonus points I can teach an artist/designer how to safely and correctly use Perforce in about 10 minutes. I’ve been using Git/Hg for years and I still run into the occasional Gitastrophe where I have to Google how to unbreak myself.
- Chyzwar 6y ago
  git have added recently sparse checkout and there is also Virtual File System for Git from Microsoft. From my experience git/vcs is not an issue for monorepo. Build, test, automation, deployments, CI/CD are way harder. You will end with a bunch of shell scripts, make files, grunt and a combination of ugly hacks. If you are smart you will adopt something like bazel and have a dedicated tooling team. If you see everything as nail, you will split monorepo into an unmaintainable mess of small repos that slowly rot away.
  - no_wizard 6y ago
    I have had the exact opposite experience. Long term maintenance in a monorepo is much easier because all developers have visibility into changes. Tooling isn’t really all that complex (at least to me and I’m not a devops engineer either). It’s way more complex to me to have multiple repositories. I’d rather have hundreds of directories that follow uniformity rather than hundreds of git repositories that may not
    - KallDrexx 6y ago
      For me the complexity comes when you have many isolated projects sharing code. With a mono-repo that shared code is immediately consumed by all isolated projects, most of which the developer is not actively looking at to look for breakages. This means that every single commit every single test suite has to be run across the whole organization. For a small startup that's probably fine, but for a larger organization that adds a ton of issues and extra time. If your change to a shared project breaks another project that you may not be familiar with you end up not only with a delayed context switch (since you don't know about it until the CI has finished running all tests across the whole org), but now you have to figure out how to address it before someone else changes code in that shared project which causes a merge conflict. Compare that to a split-repo setup where projects are sharing code indirectly (via package management systems). When you change code other projects will only consume the new code when they actually upgrade their packages, and usually a developer that is familiar with the code base will actively hit that breakage in real time. Granted, this has its own trade offs such as all your systems may be running different versions of shared code, and the delay in breakage detection can cause some back and forth in breakage between isolated projects.
jayd16 6y ago
I wouldn't call it painful exactly but I'll be happy when shallow and sparse cloning become rock solid and boring.
throwaway894345 6y ago
I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them to support any interesting case, and Nix has too many usability issues to use productively (and I’ve been in an organization that gave it an earnest shot).
- krasin 6y ago
  Can you please give an example of such an interesting case? I am genuinely curious. And I agree with the general point that monorepos require a great build tooling as a match.
  - throwaway894345 6y ago
    An interesting case of what? A monorepo? Issues with Bazel? With Nix?
    - krasin 6y ago
      An interesting case that's hard to support with Bazel. Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.
      - throwaway894345 6y ago
        It’s been a while since I tried it (maybe 2018?), but Python 3 support straight up didn’t work despite the documentation. It also seemed like adding support for your own toolchain required you to write Java plugins in a very confusing, inheritance-for-it’s-own-sake, raptly-2000s-OOP way.
        krasin 6y ago
        Got it. I can relate: while Python 3 support is probably fixed now, I agree that Bazel tooling for adding toolchains has been always abysmal. Things somewhat get better (https://docs.bazel.build/versions/master/toolchains.html https://docs.bazel.build/versions/master/toolchains.html) since it's now in Skylark, not in Java, but I still don't like it (aka not productive) Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.

krasin 6y ago

I really like monorepos. But I find that it's almost never a good idea to hide parts of a source code from developers. And if there's some secret sauce that's so sensitive that only single-digit number of developers in the whole company can access it, then it's probably okay to have a separate repository just for it. Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.

Beowolve 6y ago

On this note, GitHub does reach out to customers with Monorepos and is aware of their short comings. I think overtime we will see them change to have better support. It's only a matter of time.

jeffbee 6y ago

It's funny that you mention this as if monorepos of course require custom tooling. Google started with off-the-shelf Perforce and that was fine for many years, long after their repo became truly huge. Only when it became monstrously huge did they need custom tools and even then they basically just re-implemented Perforce instead of adopting git concepts. You, too, can just use Perforce. It's even free for up to five users. You won't outgrow its limits until you get about a million engineer-years under your belt. The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.

forrestthewoods 6y ago
It’s somewhat mind boggling that no one has made a better Perforce. It has numerous issues and warts. But it’s much closer to what the majority of projects need that Git imho. And for bonus points I can teach an artist/designer how to safely and correctly use Perforce in about 10 minutes. I’ve been using Git/Hg for years and I still run into the occasional Gitastrophe where I have to Google how to unbreak myself.
Chyzwar 6y ago
git have added recently sparse checkout and there is also Virtual File System for Git from Microsoft. From my experience git/vcs is not an issue for monorepo. Build, test, automation, deployments, CI/CD are way harder. You will end with a bunch of shell scripts, make files, grunt and a combination of ugly hacks. If you are smart you will adopt something like bazel and have a dedicated tooling team. If you see everything as nail, you will split monorepo into an unmaintainable mess of small repos that slowly rot away.
- no_wizard 6y ago
  I have had the exact opposite experience. Long term maintenance in a monorepo is much easier because all developers have visibility into changes. Tooling isn’t really all that complex (at least to me and I’m not a devops engineer either). It’s way more complex to me to have multiple repositories. I’d rather have hundreds of directories that follow uniformity rather than hundreds of git repositories that may not
  - KallDrexx 6y ago
    For me the complexity comes when you have many isolated projects sharing code. With a mono-repo that shared code is immediately consumed by all isolated projects, most of which the developer is not actively looking at to look for breakages. This means that every single commit every single test suite has to be run across the whole organization. For a small startup that's probably fine, but for a larger organization that adds a ton of issues and extra time. If your change to a shared project breaks another project that you may not be familiar with you end up not only with a delayed context switch (since you don't know about it until the CI has finished running all tests across the whole org), but now you have to figure out how to address it before someone else changes code in that shared project which causes a merge conflict. Compare that to a split-repo setup where projects are sharing code indirectly (via package management systems). When you change code other projects will only consume the new code when they actually upgrade their packages, and usually a developer that is familiar with the code base will actively hit that breakage in real time. Granted, this has its own trade offs such as all your systems may be running different versions of shared code, and the delay in breakage detection can cause some back and forth in breakage between isolated projects.

jayd16 6y ago

I wouldn't call it painful exactly but I'll be happy when shallow and sparse cloning become rock solid and boring.

throwaway894345 6y ago

I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them to support any interesting case, and Nix has too many usability issues to use productively (and I’ve been in an organization that gave it an earnest shot).

krasin 6y ago
Can you please give an example of such an interesting case? I am genuinely curious. And I agree with the general point that monorepos require a great build tooling as a match.
- throwaway894345 6y ago
  An interesting case of what? A monorepo? Issues with Bazel? With Nix?
  - krasin 6y ago
    An interesting case that's hard to support with Bazel. Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.
    - throwaway894345 6y ago
      It’s been a while since I tried it (maybe 2018?), but Python 3 support straight up didn’t work despite the documentation. It also seemed like adding support for your own toolchain required you to write Java plugins in a very confusing, inheritance-for-it’s-own-sake, raptly-2000s-OOP way.
      - krasin 6y ago
        Got it. I can relate: while Python 3 support is probably fixed now, I agree that Bazel tooling for adding toolchains has been always abysmal. Things somewhat get better (https://docs.bazel.build/versions/master/toolchains.html https://docs.bazel.build/versions/master/toolchains.html) since it's now in Skylark, not in Java, but I still don't like it (aka not productive) Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.