24 ms·

Scaling monorepo maintenance

296 points by pimterry 5y ago

underdeserver 5y ago
30 minute read + the Git object model = mind boggled. I'd have appreciated a series of articles instead of one, for me it's way too much info to take in in one sitting.
- georgyo 5y ago
  It was a lot to digest, but it was also all one continuous thought. If it was broken up, I don't think it would have been nearly as good. And I don't think I would have been able to keep all the context to understand smaller chunks. I really enjoyed the whole thing.
- iudqnolq 5y ago
  I'm currently working through the book Building Git. Best $30 I've spent in a while. It's about 700 pages, but 200 pages in and I can stage files to/from the index, make commits, and see the current status (although not on a repo with packfiles). I'm thinking about writing a blog post where I write a git commit with hexdump, zlib, and vim.
whymauri 5y ago
I love when people use Git in ways I haven't thought about before. Reminds me of the first time I played around with 'blobs.'
cryptica 5y ago
To scale a monorepo, you need to split it up into multiple repos; that way each repo can be maintained independently by a separate team... We can call it a multi-monrepo, that way our brainwashed managers will agree to it.
- Orphis 5y ago
  And that way, you can't have atomic updates across the repositories and need to synchronize them all the time, great.
  - swiley 5y ago
    Yes you can, it happens when you bump the sub module reference. This is how reasonable people use git.
    - Denvercoder9 5y ago
      Submodules often provide a terrible user experience because they are locked to a single version. To propagate a single commit, you need to update every single dependent repository. In some contexts that can be helpful, but in my experience it's mostly an enormous hassle. Also it's awful that a simple git pull doesn't actually pull updated submodules, you need to run git submodule update (or sync or whatever it is) as well. I don't want to work with git submodules ever again. The idea is nice, but the user experience is really terrible.
      - mdaniel 5y ago
        And woe unto junior developers who change into the submodule directory and do a git commit, then made infinitely worse if it's followed by git push because now there's a sha hanging out in the repo which works on one machine but that no one else's submodule update will see without surgery I'm not at my computer to see if modern git prohibits that behavior, but it is indicative of the "watch out" that comes with advanced git usage: it is a very sharp knife
      - fpoling 5y ago
        Looking back I just do not understand why git came up with this awkward mess of submodules. Instead it should have a way to say that a particular directory is self-contained and any commit affecting it should be two objects. The first is the commit object for the directory using only relative paths. The second is commit for the rest of code with a reference to it. Then one can just pull any repository into the main repository without and use it normally. git subtree tries to emulate that, but it does not scale to huge repositories as it needs to change all commits in the subtree to use new nested paths.
WayToDoor 5y ago
The article is really impressive. It's nice to see GitHub contribute changes back to the git project, and to know that the two work closely together.
- slver 5y ago
  It's in their mutual interest. Imagine what happens to GIThub if GIT goes out of fashion.
  - infogulch 5y ago
    Yes, isn't it nice when interests of multiple parties are aligned such that they help each other make progress towards their shared goals?
    - slver 5y ago
      Well, it's nice to see they're rational, indeed.
      - jackbravo 5y ago
        Other rational companies could try to fix this without contributing upstream. Doing it upstream benefits competitors like gitlab. So yeah! It's nice seeing this kind of behavior
        slver 5y ago
        First, they not only contributed upstream, upstream developers contributed to this patch. I.e. they got help outside GitHub to make this patch possible. Second, if they had decided to fork Git, then they'd have to maintain this fork forever. Third, this fork could overtime become visibly or even worse subtly incompatible with stock Git which is still the Git running on GitHub users' machines, and both should interact with each other in 100% compatible manner. So, in this case, not contributing upstream was literally no-go. The only rational choice would be to not fork Git.
debarshri 5y ago
It is a great writeup. I wonder how gitlab solves this problem.
- masklinn 5y ago
  They might have yet to encounter it. Git is hosting some really big repos.
  - lbotos 5y ago
    Oh, we def have. I've seen some large repos (50+GB) in some GL installations.
- lbotos 5y ago
  GL packs refs at various times and frequencies depending on usage: https://docs.gitlab.com/ee/administration/housekeeping.html https://docs.gitlab.com/ee/administration/housekeeping.html It works well for most repos but as you start to get out to the edges of lots of commits it can cause slowness. GL admins can repack reasonably safely at various times to get access speedups, but the solution that is presented in the blog would def speed packing up. (I work as a Support Engineering Leader at GL but I'm reading HN for fun <3)
infogulch 5y ago
This was great! My summary: A git packfile is an aggregated and indexed collection of historical git objects which reduce the time it takes to serve requests to those objects, implemented as two files: .pack and .idx. GitHub was having issues maintaining packfiles for very large repos in particular because regular repacking always has to repack the entire history into a single new packfile every time -- which is an expensive quadratic algorithm. GitHub's engineering team ameliorated this problem in two steps: 1. Enable repos to be served from multiple packfiles at once, 2. Design a packfile maintenance strategy that uses multiple packfiles sustainably. Multi-pack indexes are a new git feature, but the initial implementation was missing performance-critial reachability bitmaps for multi-pack indexes. In general, index files store object names in lexicographic order and point to the named object's position in the associated packfile. As a first step to implement reachability bitmaps for multi-pack indexes, they introduced a reverse index file (.rev) which maps packfile object positions back to index file name offsets. This alone had a big performance improvement, but it also filled in the missing piece in order to implement multi-pack bitmaps. With the issues of serving repos from multiple packs solved, they need to efficiently utilize multiple packs to reduce maintenance overhead. They chose to maintain historical packfiles in a geometrically increasing size. I.e., during the maintenance job, consider the first N most recent packfiles, if the sum of the size of all packfiles from [1, N] is less than the size of packfile N+1, then packfiles [1, N] are repacked into a single packfile, done; however if their summed size is greater than the size of packfile N+1, then iterate and consider all the packfiles [1, N+1] compared to packfile N+2 etc. This results in a set of packfiles where each file is roughly double the size of the previous when ordered by age, which has a number of beneficial properties for both serving and the average case maintenance run. Funny enough, this selection procedure struck me as similar to the game "2048".
swiley 5y ago
Mono-repos are like having a flat directory structure. Sure it's simple but it makes it hard to find anything if you have a lot of stuff/people. submodules and package managers exist for a reason.
- Denvercoder9 5y ago
  > Sure it's simple but it makes it hard to find anything if you have a lot of stuff/people. I think this is a bad analogy. Looking up a file or directory in a monorepo isn't harder than looking up a repository. In fact, I'd argue it's easier, as we've developed decades of tooling for searching through filesystems, while for searching through remotely hosted repositories you're dependent on the search function of the repository host, which is often worse.
- no_wizard 5y ago
  Note: for the sake of discussion I’m assuming when we say monorepo we mean monorepo and associated tools used to manage them The trade off is simplified management of dependencies. With a monorepo, I can control every version of a given dependency so they’re uniform across packages. If I update one package it is always going to be linked to the other in its latest version. I can simplify releases and managing my infrastructure in the long term, though there is a trade off in initial complexity for certain things if you want to do something like say, only run tests for packages that have changed in CI (useful in some cases). It’s all trade offs, but the quality of code has been higher for our org in a monorepo on average
  - mr_tristan 5y ago
    I've found that many developers do not pay attention to dependency management, so this approach of "it's either in the repo or it doesn't exist" is actually a nice guard rail. I'm reading between the lines here, but, I'm assuming you've setup your tooling to enforce this. As in: the various projects in the repo don't just optionally decide to have external references, i.e., Maven central, npm, etc. This puts quite a lot of "stuff" in the repo, but with improvements like this article mentioned, makes monorepos in git much easier to use. I'd have to think, you could generate a lot of automation and reports triggering out of commits pretty easily, too. I'd say that would make the monorepo even easier to observe with a modicum of the tooling required to maintain independent repositories.
the_duke 5y ago
Very well written post and upstream work is always appreciated. I also really like monorepos, but Git and GitHub really don't work well at all for them. On the Git side there is no way to clone only parts of a repo or to limit access by user. All the Git tooling out there, from CLI to the various IDE integrations, are all very ill adjusted to a huge repo with lots of unrelated commits. On the Github side there is no separation between the different parts of a monorepo in the UI (issues, prs, CI), the workflows, or the permission system. Sure, you can hack something together with labels and custom bots, but it always feels like a hack. Using Git(hub) for monorepos is really painful in my experience. There is a reason why Google, Facebook et all have heaps of custom tooling.
- krasin 5y ago
  I really like monorepos. But I find that it's almost never a good idea to hide parts of a source code from developers. And if there's some secret sauce that's so sensitive that only single-digit number of developers in the whole company can access it, then it's probably okay to have a separate repository just for it. Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.
- Beowolve 5y ago
  On this note, GitHub does reach out to customers with Monorepos and is aware of their short comings. I think overtime we will see them change to have better support. It's only a matter of time.
- jeffbee 5y ago
  It's funny that you mention this as if monorepos of course require custom tooling. Google started with off-the-shelf Perforce and that was fine for many years, long after their repo became truly huge. Only when it became monstrously huge did they need custom tools and even then they basically just re-implemented Perforce instead of adopting git concepts. You, too, can just use Perforce. It's even free for up to five users. You won't outgrow its limits until you get about a million engineer-years under your belt. The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.

296 points by pimterry 5y ago

30 minute read + the Git object model = mind boggled. I'd have appreciated a series of articles instead of one, for me it's way too much info to take in in one sitting.

georgyo 5y ago
It was a lot to digest, but it was also all one continuous thought. If it was broken up, I don't think it would have been nearly as good. And I don't think I would have been able to keep all the context to understand smaller chunks. I really enjoyed the whole thing.
iudqnolq 5y ago
I'm currently working through the book Building Git. Best $30 I've spent in a while. It's about 700 pages, but 200 pages in and I can stage files to/from the index, make commits, and see the current status (although not on a repo with packfiles). I'm thinking about writing a blog post where I write a git commit with hexdump, zlib, and vim.

whymauri 5y ago

I love when people use Git in ways I haven't thought about before. Reminds me of the first time I played around with 'blobs.'

cryptica 5y ago

To scale a monorepo, you need to split it up into multiple repos; that way each repo can be maintained independently by a separate team... We can call it a multi-monrepo, that way our brainwashed managers will agree to it.

Orphis 5y ago
And that way, you can't have atomic updates across the repositories and need to synchronize them all the time, great.
- swiley 5y ago
  Yes you can, it happens when you bump the sub module reference. This is how reasonable people use git.
  - Denvercoder9 5y ago
    Submodules often provide a terrible user experience because they are locked to a single version. To propagate a single commit, you need to update every single dependent repository. In some contexts that can be helpful, but in my experience it's mostly an enormous hassle. Also it's awful that a simple git pull doesn't actually pull updated submodules, you need to run git submodule update (or sync or whatever it is) as well. I don't want to work with git submodules ever again. The idea is nice, but the user experience is really terrible.
    - mdaniel 5y ago
      And woe unto junior developers who change into the submodule directory and do a git commit, then made infinitely worse if it's followed by git push because now there's a sha hanging out in the repo which works on one machine but that no one else's submodule update will see without surgery I'm not at my computer to see if modern git prohibits that behavior, but it is indicative of the "watch out" that comes with advanced git usage: it is a very sharp knife
    - fpoling 5y ago
      Looking back I just do not understand why git came up with this awkward mess of submodules. Instead it should have a way to say that a particular directory is self-contained and any commit affecting it should be two objects. The first is the commit object for the directory using only relative paths. The second is commit for the rest of code with a reference to it. Then one can just pull any repository into the main repository without and use it normally. git subtree tries to emulate that, but it does not scale to huge repositories as it needs to change all commits in the subtree to use new nested paths.

WayToDoor 5y ago

The article is really impressive. It's nice to see GitHub contribute changes back to the git project, and to know that the two work closely together.

slver 5y ago
It's in their mutual interest. Imagine what happens to GIThub if GIT goes out of fashion.
- infogulch 5y ago
  Yes, isn't it nice when interests of multiple parties are aligned such that they help each other make progress towards their shared goals?
  - slver 5y ago
    Well, it's nice to see they're rational, indeed.
    - jackbravo 5y ago
      Other rational companies could try to fix this without contributing upstream. Doing it upstream benefits competitors like gitlab. So yeah! It's nice seeing this kind of behavior
      - slver 5y ago
        First, they not only contributed upstream, upstream developers contributed to this patch. I.e. they got help outside GitHub to make this patch possible. Second, if they had decided to fork Git, then they'd have to maintain this fork forever. Third, this fork could overtime become visibly or even worse subtly incompatible with stock Git which is still the Git running on GitHub users' machines, and both should interact with each other in 100% compatible manner. So, in this case, not contributing upstream was literally no-go. The only rational choice would be to not fork Git.

debarshri 5y ago

It is a great writeup. I wonder how gitlab solves this problem.

masklinn 5y ago
They might have yet to encounter it. Git is hosting some really big repos.
- lbotos 5y ago
  Oh, we def have. I've seen some large repos (50+GB) in some GL installations.
lbotos 5y ago
GL packs refs at various times and frequencies depending on usage: https://docs.gitlab.com/ee/administration/housekeeping.html https://docs.gitlab.com/ee/administration/housekeeping.html It works well for most repos but as you start to get out to the edges of lots of commits it can cause slowness. GL admins can repack reasonably safely at various times to get access speedups, but the solution that is presented in the blog would def speed packing up. (I work as a Support Engineering Leader at GL but I'm reading HN for fun <3)

infogulch 5y ago

This was great! My summary: A git packfile is an aggregated and indexed collection of historical git objects which reduce the time it takes to serve requests to those objects, implemented as two files: .pack and .idx. GitHub was having issues maintaining packfiles for very large repos in particular because regular repacking always has to repack the entire history into a single new packfile every time -- which is an expensive quadratic algorithm. GitHub's engineering team ameliorated this problem in two steps: 1. Enable repos to be served from multiple packfiles at once, 2. Design a packfile maintenance strategy that uses multiple packfiles sustainably. Multi-pack indexes are a new git feature, but the initial implementation was missing performance-critial reachability bitmaps for multi-pack indexes. In general, index files store object names in lexicographic order and point to the named object's position in the associated packfile. As a first step to implement reachability bitmaps for multi-pack indexes, they introduced a reverse index file (.rev) which maps packfile object positions back to index file name offsets. This alone had a big performance improvement, but it also filled in the missing piece in order to implement multi-pack bitmaps. With the issues of serving repos from multiple packs solved, they need to efficiently utilize multiple packs to reduce maintenance overhead. They chose to maintain historical packfiles in a geometrically increasing size. I.e., during the maintenance job, consider the first N most recent packfiles, if the sum of the size of all packfiles from [1, N] is less than the size of packfile N+1, then packfiles [1, N] are repacked into a single packfile, done; however if their summed size is greater than the size of packfile N+1, then iterate and consider all the packfiles [1, N+1] compared to packfile N+2 etc. This results in a set of packfiles where each file is roughly double the size of the previous when ordered by age, which has a number of beneficial properties for both serving and the average case maintenance run. Funny enough, this selection procedure struck me as similar to the game "2048".

swiley 5y ago

Mono-repos are like having a flat directory structure. Sure it's simple but it makes it hard to find anything if you have a lot of stuff/people. submodules and package managers exist for a reason.

Denvercoder9 5y ago
> Sure it's simple but it makes it hard to find anything if you have a lot of stuff/people. I think this is a bad analogy. Looking up a file or directory in a monorepo isn't harder than looking up a repository. In fact, I'd argue it's easier, as we've developed decades of tooling for searching through filesystems, while for searching through remotely hosted repositories you're dependent on the search function of the repository host, which is often worse.
no_wizard 5y ago
Note: for the sake of discussion I’m assuming when we say monorepo we mean monorepo and associated tools used to manage them The trade off is simplified management of dependencies. With a monorepo, I can control every version of a given dependency so they’re uniform across packages. If I update one package it is always going to be linked to the other in its latest version. I can simplify releases and managing my infrastructure in the long term, though there is a trade off in initial complexity for certain things if you want to do something like say, only run tests for packages that have changed in CI (useful in some cases). It’s all trade offs, but the quality of code has been higher for our org in a monorepo on average
- mr_tristan 5y ago
  I've found that many developers do not pay attention to dependency management, so this approach of "it's either in the repo or it doesn't exist" is actually a nice guard rail. I'm reading between the lines here, but, I'm assuming you've setup your tooling to enforce this. As in: the various projects in the repo don't just optionally decide to have external references, i.e., Maven central, npm, etc. This puts quite a lot of "stuff" in the repo, but with improvements like this article mentioned, makes monorepos in git much easier to use. I'd have to think, you could generate a lot of automation and reports triggering out of commits pretty easily, too. I'd say that would make the monorepo even easier to observe with a modicum of the tooling required to maintain independent repositories.

the_duke 5y ago

Very well written post and upstream work is always appreciated. I also really like monorepos, but Git and GitHub really don't work well at all for them. On the Git side there is no way to clone only parts of a repo or to limit access by user. All the Git tooling out there, from CLI to the various IDE integrations, are all very ill adjusted to a huge repo with lots of unrelated commits. On the Github side there is no separation between the different parts of a monorepo in the UI (issues, prs, CI), the workflows, or the permission system. Sure, you can hack something together with labels and custom bots, but it always feels like a hack. Using Git(hub) for monorepos is really painful in my experience. There is a reason why Google, Facebook et all have heaps of custom tooling.

krasin 5y ago
I really like monorepos. But I find that it's almost never a good idea to hide parts of a source code from developers. And if there's some secret sauce that's so sensitive that only single-digit number of developers in the whole company can access it, then it's probably okay to have a separate repository just for it. Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.
Beowolve 5y ago
On this note, GitHub does reach out to customers with Monorepos and is aware of their short comings. I think overtime we will see them change to have better support. It's only a matter of time.
jeffbee 5y ago
It's funny that you mention this as if monorepos of course require custom tooling. Google started with off-the-shelf Perforce and that was fine for many years, long after their repo became truly huge. Only when it became monstrously huge did they need custom tools and even then they basically just re-implemented Perforce instead of adopting git concepts. You, too, can just use Perforce. It's even free for up to five users. You won't outgrow its limits until you get about a million engineer-years under your belt. The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.