9 ms·

Bring your monorepo down to size with sparse-checkout

137 points by Amorymeltzer 7y ago

alanfranz 7y ago
Monorepo + sparse-checkout looks a bit like a distributed subversion!
- rrmm 7y ago
  Did you ever try out SVK? It was based on svn's libraries and provided for a more disconnected workflow. Most of my code (dating from before git was around) for various projects is in a single subversion tree and I check it out using git or mercurial to provide for local version control. Features like sparse checkout are definitely welcome in git since the industry seems to have standardized on it.
- gdxhyrd 7y ago
  Not really, because commits don't go across the entire SVN, which is what makes monorepos so powerful.
  - krupan 7y ago
    What do you mean? When you commit to svn the whole repository goes up in version number.
    - gdxhyrd 7y ago
      You are right, I was thinking of CVS. In any case, with SVN you usually do not want to give write perms to everyone in all the tree, so you end up with effectively partitioned spaces, or you make several repos instead, or you put another layer on top. With Git, anyone can easily develop global commits.
tantalor 7y ago
I have to tell it which directories I want? That seems like work the tool could do. Also, the granularity should be at the file level, not directory.
- war1025 7y ago
  It's based on glob paths. I think you can specify it at whatever level you want. Also you can use wildcards.
- stolee 7y ago
  The sparse-checkout patterns match at the file level, so you can always use that (without “cone mode”) if you want. It becomes difficult to match an exact file list as people add files to projects: you require every other user to update their patterns to match the newly-added file.
  - WorldMaker 7y ago
    Just keep in mind as the article points out that without "cone mode" is potentially a lot slower, and that's why cone mode exists.
alexhutcheson 7y ago
sparse-checkout, partial-clone, and shallow seem like decent building blocks to make working with very large repos tractable in git. At the same time, the features and their interaction are pretty complicated, so I believe we'll need good "porcelain" abstractions over these building blocks to make the workflow reasonable for average users.
- allover 7y ago
  What is your bar for 'average users'? If we're talking project-wide repos rather than entire-org repos, I'd wager the vast majority of projects can use monorepos without special git tooling, and will retain huge productivity benefits vs app/package-per-repo organisation.
  - alexhutcheson 7y ago
    Honestly most orgs (with "most" weighted by org, not by headcount) could handle entire-org repos without using any of these features. It's still worth simplifying the workflow and training experience for projects and orgs that grow beyond that, though.
MayeulC 7y ago
Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.
- mappu 7y ago
  > For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.
  - monocasa 7y ago
    Sometimes you don't own the other repo.
    - scarejunba 7y ago
      Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.
      - phonebanshee 7y ago
        Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.
        scarejunba 7y ago
        Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.
      - 7y ago
ComputerGuru 7y ago
I have always done —depth=1 for projects I am not a core developer of, but ran into an issue with it being seemingly impossible to do the same with submodules. Golang should have figured this out from day 1 before shipping with a release system built around cloning a repo in its entirety, history and recursive submodules, and all.
- est31 7y ago
  depth=1 is an old feature but it only limits the clone in the history dimension. You still have to store the entire state of the tree as of last commit. This feature is about cloning parts of the tree.
  - ComputerGuru 7y ago
    My point is that even that basic functionality a) did not extend to all of git's core functionality, and b) went unused by major players in the industry self-procaimedly responsible for "optimizing" the internet.
- xyzzy_plugh 7y ago
  FWIW Go modules address this pretty well.
- alexhutcheson 7y ago
  Have you tried the --shallow_submodules flag? I haven't used it, but it seems like to should do what you want.
peterwwillis 7y ago
I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a repo of submodules not work?
- gdxhyrd 7y ago
  What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.
  - gen220 7y ago
    We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I think even a simple list of these sort of “gotchas” would be valuable for the aspirational mono repo company. My impression is that a lot of teams hit these early and painful roadblocks, and imagine that they’ll never go away (they do!!).
    - Hackbraten 7y ago
      Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.
      - gen220 7y ago
        I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “pip install”/“go get” fail on our CI system if the dependency isn’t hosted by us. This gives us reproducible builds, while allowing us to hold 3rd party libraries to a higher standard of entry than source code. For certain libraries we pin version numbers in our build system, but in general it allows us to update dependencies transparently. It also keeps our source repo size small, for developers, and allows for conflicting versions (example Kafka X.Y and X.Z) without cluttering the repo with duplicates. It’s definitely a smaller gotcha than the others I listed, maybe to the point where it’s not a gotcha, but I stand by it :)
divbzero 7y ago
This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
- wikibob 7y ago
  I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?
  - kyrra 7y ago
    2 reasons: VFS is a fork of normal Git. There is no Linux client. Also remember that there are many Git clients that work with normal Git repos. Like libgit and others. I doubt you'll see wide spread support for it unless MS can upstream it into the main Git implementation, and maybe some the the primary libraries. This is one nice argument for Mercurial, where there is only a single implementation, so adding big new changes can be easier.
    - deleted 7y ago
      [deleted]
  - microcolonel 7y ago
    VFS for Git tool only works on Windows, and is written in C#; it also relies on a fork of Git.
- WorldMaker 7y ago
  This sounds like a parallel effort (along with the commit graph work) to keep pushing that better. The article's writer (who also wrote most of the blog posts on the commit graph work) mentions a "three million file repository" used for testing in this article and that would of course sound like the Windows repo. It's also I'd imagine not mutually exclusive effort. It seems like exactly like something you would want in combination with something like VFS at scale, as it reduces the number of materialized versus virtual objects in both the git working copy and the git object database. If you've got millions or billions of objects and files, even reducing the number of virtual placeholders I would imagine is probably a big win.
gravypod 7y ago
I can't wait until there's tooling that takes advantage of this. Tying sparse checkout into Gradle or Bazel would make this a lot easier.
krupan 7y ago
How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM type company in he late '90s/early 2000's before you do that. Please!
- marcell 7y ago
  Continuous integration tools still check out and test the whole repo. Google has used this approach for over a decade.
  - andrewg 7y ago
    This would be impractical for really large monorepos like the ones Google and Microsoft have. They have virtual file system layers on top (MS open sourced theirs) to prevent checking out the whole repo. In fact, it’s not just useful for the CI/CD pipeline - any developers making significant changes to base libraries or core infrastructure should be able to use the VFS in combination with a system like Bazel to run all (or a significant sample of) affected tests across the company.
- qznc 7y ago
  They are hard to find. Do you know some? All I have is this thread: https://lobste.rs/s/fosip5/should_version_control_build_systems#c_bkq3ve https://lobste.rs/s/fosip5/should_version_control_build_syst...
  - krupan 7y ago
    I used clearcase many years ago, and this thread on lobste.rs is pretty accurate and interesting. They point out that the biggest problems were exclusive checkouts, file versioning instead of changesets, and the baked in out of date assumptions about networking. Getting your configspec wrong was a common problem too. At HP we had some in-house perl-script wrappers around the raw clearcase tools that fixed many of these problems. The developers of those scripts had all left to go work for Rational (makers of clearcase), and I don't think anyone really knew how they worked. We also had a full-time clearcase engineer that kept the servers running. Fortunately our smallish projects didn't need the full power of clearcase and those perl scripts kept working fine for us. I did alway wonder what would happen if the one guy who understood the servers left the company. In short, it's a complex and powerful tool that very few people really understood. Very few projects need all that power and complexity. I'm sure Microsoft and Google benefit from complex version control tools and have engineers to spare for managing and understanding them, but I don't think any open source projects or smaller companies are really going to benefit from "clearcase for the modern age" type tools.
fuckyah 7y ago
monorepos are just a disaster
mark_l_watson 7y ago
Partial checkout efficiency improvements makes mono repos more compelling for large projects and organizations. As an individual, I have switched to a mono repo for all of my Common Lisp code and with some adjustments to my Quicklisp configuration I am very happy with my setup. I am a programming language junkie, and I have it on my low priority todo list to switch to a mono repo for Haskell, Racket, and Hy language (Lisp with a Clojure syntax that sits on top of Python). I worked as a contractor at Google in 2013 and I absolutely loved their mono repo and web based development environment. I really miss that.

137 points by Amorymeltzer 7y ago

alanfranz 7y ago

Monorepo + sparse-checkout looks a bit like a distributed subversion!

rrmm 7y ago
Did you ever try out SVK? It was based on svn's libraries and provided for a more disconnected workflow. Most of my code (dating from before git was around) for various projects is in a single subversion tree and I check it out using git or mercurial to provide for local version control. Features like sparse checkout are definitely welcome in git since the industry seems to have standardized on it.
gdxhyrd 7y ago
Not really, because commits don't go across the entire SVN, which is what makes monorepos so powerful.
- krupan 7y ago
  What do you mean? When you commit to svn the whole repository goes up in version number.
  - gdxhyrd 7y ago
    You are right, I was thinking of CVS. In any case, with SVN you usually do not want to give write perms to everyone in all the tree, so you end up with effectively partitioned spaces, or you make several repos instead, or you put another layer on top. With Git, anyone can easily develop global commits.

tantalor 7y ago

I have to tell it which directories I want? That seems like work the tool could do. Also, the granularity should be at the file level, not directory.

war1025 7y ago
It's based on glob paths. I think you can specify it at whatever level you want. Also you can use wildcards.
stolee 7y ago
The sparse-checkout patterns match at the file level, so you can always use that (without “cone mode”) if you want. It becomes difficult to match an exact file list as people add files to projects: you require every other user to update their patterns to match the newly-added file.
- WorldMaker 7y ago
  Just keep in mind as the article points out that without "cone mode" is potentially a lot slower, and that's why cone mode exists.

alexhutcheson 7y ago

sparse-checkout, partial-clone, and shallow seem like decent building blocks to make working with very large repos tractable in git. At the same time, the features and their interaction are pretty complicated, so I believe we'll need good "porcelain" abstractions over these building blocks to make the workflow reasonable for average users.

allover 7y ago
What is your bar for 'average users'? If we're talking project-wide repos rather than entire-org repos, I'd wager the vast majority of projects can use monorepos without special git tooling, and will retain huge productivity benefits vs app/package-per-repo organisation.
- alexhutcheson 7y ago
  Honestly most orgs (with "most" weighted by org, not by headcount) could handle entire-org repos without using any of these features. It's still worth simplifying the workflow and training experience for projects and orgs that grow beyond that, though.

MayeulC 7y ago

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

mappu 7y ago
> For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.
- monocasa 7y ago
  Sometimes you don't own the other repo.
  - scarejunba 7y ago
    Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.
    - phonebanshee 7y ago
      Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.
      - scarejunba 7y ago
        Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.
    - 7y ago

ComputerGuru 7y ago

I have always done —depth=1 for projects I am not a core developer of, but ran into an issue with it being seemingly impossible to do the same with submodules. Golang should have figured this out from day 1 before shipping with a release system built around cloning a repo in its entirety, history and recursive submodules, and all.

est31 7y ago
depth=1 is an old feature but it only limits the clone in the history dimension. You still have to store the entire state of the tree as of last commit. This feature is about cloning parts of the tree.
- ComputerGuru 7y ago
  My point is that even that basic functionality a) did not extend to all of git's core functionality, and b) went unused by major players in the industry self-procaimedly responsible for "optimizing" the internet.
xyzzy_plugh 7y ago
FWIW Go modules address this pretty well.
alexhutcheson 7y ago
Have you tried the --shallow_submodules flag? I haven't used it, but it seems like to should do what you want.

peterwwillis 7y ago

I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a repo of submodules not work?

gdxhyrd 7y ago
What problems did you encounter with just a few services? Monorepos should be straightforward unless you are managing the code of >1k engineers.
- gen220 7y ago
  We’ve run into some nontrivial but totally solvable issues at about 100-200 engineers. IME, most consternation comes from people adopting a mono repo without adopting a build/dependency graph tool (like Bazel, buck or pants). An additional source of strain is from people abusing the repo (checking in large binaries, third party dependencies, etc). A third is when people try to do branch-based feature development, instead of the “correct” practice of only deploying master (or weekly cuts of master). I think even a simple list of these sort of “gotchas” would be valuable for the aspirational mono repo company. My impression is that a lot of teams hit these early and painful roadblocks, and imagine that they’ll never go away (they do!!).
  - Hackbraten 7y ago
    Checking in third-party dependencies is not always abuse. It can be a useful habit for certain kinds of reproducible builds. The Buck documentation even endorses keeping your dependencies in your monorepo along with your own sources.
    - gen220 7y ago
      I understand the reasoning, and agree that it’s not always abuse. At first blush it’s a good idea, but I’d maintain that it’s one of the things that balloons your repo size quite quickly. Plus, one have to draw a line somewhere on what to include (a Python interpreter? A Go version? awk and grep?), and third party vs in-house is a fairly robust one imo. We host a private mirror for third party dependencies, so that “pip install”/“go get” fail on our CI system if the dependency isn’t hosted by us. This gives us reproducible builds, while allowing us to hold 3rd party libraries to a higher standard of entry than source code. For certain libraries we pin version numbers in our build system, but in general it allows us to update dependencies transparently. It also keeps our source repo size small, for developers, and allows for conflicting versions (example Kafka X.Y and X.Z) without cluttering the repo with duplicates. It’s definitely a smaller gotcha than the others I listed, maybe to the point where it’s not a gotcha, but I stand by it :)

divbzero 7y ago

This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

wikibob 7y ago
I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?
- kyrra 7y ago
  2 reasons: VFS is a fork of normal Git. There is no Linux client. Also remember that there are many Git clients that work with normal Git repos. Like libgit and others. I doubt you'll see wide spread support for it unless MS can upstream it into the main Git implementation, and maybe some the the primary libraries. This is one nice argument for Mercurial, where there is only a single implementation, so adding big new changes can be easier.
  - deleted 7y ago
    [deleted]
- microcolonel 7y ago
  VFS for Git tool only works on Windows, and is written in C#; it also relies on a fork of Git.
WorldMaker 7y ago
This sounds like a parallel effort (along with the commit graph work) to keep pushing that better. The article's writer (who also wrote most of the blog posts on the commit graph work) mentions a "three million file repository" used for testing in this article and that would of course sound like the Windows repo. It's also I'd imagine not mutually exclusive effort. It seems like exactly like something you would want in combination with something like VFS at scale, as it reduces the number of materialized versus virtual objects in both the git working copy and the git object database. If you've got millions or billions of objects and files, even reducing the number of virtual placeholders I would imagine is probably a big win.

gravypod 7y ago

I can't wait until there's tooling that takes advantage of this. Tying sparse checkout into Gradle or Bazel would make this a lot easier.

krupan 7y ago

How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM type company in he late '90s/early 2000's before you do that. Please!

marcell 7y ago
Continuous integration tools still check out and test the whole repo. Google has used this approach for over a decade.
- andrewg 7y ago
  This would be impractical for really large monorepos like the ones Google and Microsoft have. They have virtual file system layers on top (MS open sourced theirs) to prevent checking out the whole repo. In fact, it’s not just useful for the CI/CD pipeline - any developers making significant changes to base libraries or core infrastructure should be able to use the VFS in combination with a system like Bazel to run all (or a significant sample of) affected tests across the company.
qznc 7y ago
They are hard to find. Do you know some? All I have is this thread: https://lobste.rs/s/fosip5/should_version_control_build_systems#c_bkq3ve https://lobste.rs/s/fosip5/should_version_control_build_syst...
- krupan 7y ago
  I used clearcase many years ago, and this thread on lobste.rs is pretty accurate and interesting. They point out that the biggest problems were exclusive checkouts, file versioning instead of changesets, and the baked in out of date assumptions about networking. Getting your configspec wrong was a common problem too. At HP we had some in-house perl-script wrappers around the raw clearcase tools that fixed many of these problems. The developers of those scripts had all left to go work for Rational (makers of clearcase), and I don't think anyone really knew how they worked. We also had a full-time clearcase engineer that kept the servers running. Fortunately our smallish projects didn't need the full power of clearcase and those perl scripts kept working fine for us. I did alway wonder what would happen if the one guy who understood the servers left the company. In short, it's a complex and powerful tool that very few people really understood. Very few projects need all that power and complexity. I'm sure Microsoft and Google benefit from complex version control tools and have engineers to spare for managing and understanding them, but I don't think any open source projects or smaller companies are really going to benefit from "clearcase for the modern age" type tools.

fuckyah 7y ago

monorepos are just a disaster

mark_l_watson 7y ago

Partial checkout efficiency improvements makes mono repos more compelling for large projects and organizations. As an individual, I have switched to a mono repo for all of my Common Lisp code and with some adjustments to my Quicklisp configuration I am very happy with my setup. I am a programming language junkie, and I have it on my low priority todo list to switch to a mono repo for Haskell, Racket, and Hy language (Lisp with a Clojure syntax that sits on top of Python). I worked as a contractor at Google in 2013 and I absolutely loved their mono repo and web based development environment. I really miss that.