7 ms·

From the article: > Lots of branches – Users of Git create branches pretty prolifically. It’s not uncommon for an engineer to build up ~20 branches over time

by ekidd 10y ago

From the article: > Lots of branches – Users of Git create branches pretty prolifically. It’s not uncommon for an engineer to build up ~20 branches over time and multiply 20 by, say 5000 engineers and that’s 100,000 branches. Git just won’t be usable. To solve this, we built a feature we call “limited refs” into our Git service (Team Services and TFS) that will cause the service to pretend that only the branches “you care about” are projected to your Git client. You can favorite the branches you want and Git will be happy. This is almost certainly a result of trying to have one company-wide monolithic repository that holds the source code of hundreds or thousands of separate projects. Git is more pleasant when you break your codebase into isolated components. These can be pretty large—the Linux kernel has 16 million lines of code—but if your codebase is many times larger than a complete modern kernel, you might want to split it. If you have 5,000 engineers all pushing branches to a single master git repository, you may want to either rethink your repository structure, or at least have maintainer subtrees the way Linux does.

LeifCarrotson 10y ago
> This is almost certainly a result of trying to have one company-wide monolithic repository that holds the source code of hundreds or thousands of separate projects. Correct. But both Google and Microsoft use this method. And I'm sure they've put in a lot more man-years of investigation into it than you and I have!
- mjn 10y ago
  Google reportedly [1] also uses a virtual filesystem (FUSE-based, in their case) to let individual clients get a convenient/efficient view into their giant monorepo. Microsoft looks like they're doing something very similar here, except building their vfs/monorepo solution on top of git and open-sourcing it (Google's is built on top of an in-house VCS called Piper). [1] http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext http://cacm.acm.org/magazines/2016/7/204032-why-google-store...
  - bluejekyll 10y ago
    Isn't google's system inspired from their use of Perforce as well? Is Piper Git like, or more Perforce/SVN like?
    - iainmerrick 10y ago
      Piper is a reimplementation of Perforce, because they had an existing gigantic source repository in Perforce and needed something compatible. There was an earlier project to migrate to SVN but that failed. I believe Git was never in contention due to the obvious scalability problems, but GVFS does seem like it might potentially be usable at Google.
- cwyers 10y ago
  Facebook and I believe Twitter, too. It's almost as if anyone who actually has to contend with problems at this scale prefers a monorepo and anyone who likes to speculate on how to solve other people's problems on message boards thinks it's a dumb idea.
  - iainmerrick 10y ago
    True, but I believe a lot of both Facebook's and Twitter's dev culture comes from ex-Google employees. So there could still be some groupthink going on. (Alternatively, a victory for successful ideas?) Indirect evidence: both of them released open source clones of Google's proprietary build system "Blaze" -- Facebook's "Buck" and Twitter's "Pants". Google was actually last to the party, with "Bazel".
- deleted 10y ago
  [deleted]
bluejekyll 10y ago
I think GitHub/Gitlab (enterprise or public), do an excellent job of capturing the monorepo desires, but with lots of disparate code repositories. That is, mono repos are great for single company wide access, easy to fork and commit against different areas, have a single system for driving all CI changes across repos. I agree that having a single huge repo kinda sucks in Git, but it's also basically a nonstarter for companies that want all the features of git, but their repo and build is monolithic. Would you require people to first completely rebuild their software into separate Git repos before using Git? This could take years! I've been looking at this exact problem at my work, and there is no easy answer here. I think MS probably picked the easier path forward in their case.
izacus 10y ago
> Git is more pleasant when you break your codebase into isolated components. These can be pretty large—the Linux kernel has 16 million lines of code—but if your codebase is many times larger than a complete modern kernel, you might want to split it. As part of a company that switched to same kind monorepo structure form several separate repositories - there is nothing "pleasant" about having to deal with multiple Git repositories for connected components. Subtrees, submodules are utter hell of maintenance, checkout bugs (which hurt CI) and bad UX across the board ("why doesn't this build?" "you forgot to checkout submodules" "no you forgot to move the commit pointer" "no you forgot to change dependant tests because you didn't see them in an isolated repository"...) It's not a GOOD approach, but it's the best approach compared to all other more terrible ones.
- luckydude 10y ago
  Git would do well to copy BitKeeper's nested repository architecture. Unlike Git, a collection of nested repositories (what we call submodules) works exactly like a monolithic repository. Want to see diffs for your tree? Same command. Want to check in your changes? Same command. Want to update your collection? Same command. Etc. It's a lot more work than the hack that is submodules (we understand that hack, we did exactly the same thing for years to track outside source drops). But it is worth it, everything just works. http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/productline.pdf http://www.mcvoy.com/lm/bkdocs/productline.pdf
  - krupan 10y ago
    I'm really intrigued by this. I love mercurial (and tolerate git), but they both handle sub repositories poorly. How much of my mercurial (and/or git) muscle memory would I have to unlearn to switch to bk. Just for basic commands, I mean. You could tell me to go try it out myself, but a nice side-by-side comparison showing that bk does all that git/hg do (if that's true) could go a long way to attracting git/mercurial people over to bk.
    - luckydude 10y ago
blktiger 10y ago
_Programming_ is more pleasant when you break your codebase into isolated components. Instead of trying to force git to adapt to these monolithic repositories, we should be breaking them down into manageable pieces for developers to work on. If you still need 10 years of history, just keep the old repository around for archival purposes. [Edit] Not that I'm against GVFS, there is nothing wrong with not wanting to download the entire repository in some cases.
- aseipp 10y ago
  Many monolithic repositories still have very clear isolation and component separation. There's a difference between "I'm doing this out of good taste" and "I'm doing this because this tool makes my life insufferable, otherwise". For dedicated organizations, with active development -- singular repos have some keen benefits, IMO. Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. I didn't even work on a multi-million line codebase. Few hundred kLOC. But with like 20 submodules though (library components), and the amount of organizational overhead is annoying as shit. (Not to mention the tools we developed to stop the active ~20 developers from footgunning themselves when they committed non-existent pointers, explaining carefully how they worked, etc). We have a binary or two, but we are anxious and careful to control and update them when we do (they were added before our time, and every update at this point on such a long-lived repo can visibly affect performance. Splitting it out is even more annoying ultimately and loses our history, and does not reduce the size of the original repo). Git is just bad here. Almost everyone is a volunteer. This kind of shit is a waste of time for everyone, frankly. What if git was 5x as slow as it was now? Would that mean we should create 5x more repositories, divide every repository up into 5 more? Delete your historic repositores with 5x frequency? Would trying to improve that mean we're trying to "force git to adapt for monolithic repositories", as you claim? There are very real limits to this kind of reasoning, bending backwards for your tools. Tools like Bitkeeper are far better at this and have a way better UI, out of the box. The "Product Lines" feature means subrepositories are much more transparent for all users (with full push/pull bidirectional interoperation, which is absolutely critical, and it does crazy things like -- gasp, cloning all subrepositores by default). But BitKeeper can also handle 50GB repositories fairly easily and quickly, including large binary files. No multi-repos necessary, no LFS. It "just works". That's excellent and it removes concerns and worries you might have later. I also think you understate the true value of continuous history. It's not something to toss aside so quickly, and any extra effort to use it means you lose a lot of insight. Maybe if you don't have to dig old code constantly (or your company throws away/rewrites its product constantly I guess). In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. New developers cannot possibly understand the rational of an obscure change 10 years ago by someone with a PhD without this kind of help, at least not easily. Some files may not have been touched in like 8 years! This repository is also relatively quick to clone, luckily. The 20 submodules make it a damper. One of my last jobs was working on a product that was 15 years old, and had only one repository. They stored binaries in it, but it had had many of them updated over the years (small JAR files, mostly, one copy of GCC). It took 6 hours to checkout, but honestly that was the main thing that sucked - performance. However, all of the same benefits applied. And that one truly was continuous -- you could get a look at the repo, as it was built for customers, years ago, to look at the behavior. Atomic refactorings across years of code, over a million lines of code, and 50+ developers and libraries was possible in a single commit. That's a good thing. But the performance is a tool deficiency, not something to be hand waved away. That second repository, the 15 year one, was like, 50GB total. That's peanuts in today's terms. Other VCSs would have been much more tolerable (including BK, or probably something like Perforce, in terms of pure speed).
aseipp 10y ago
This reasoning has very serious limits, however. Every tool influences your behavior. Imagine if Git was somehow 10x slower. It's already quite fast still, but still way slower (e.g. on Windows for some cases). Does this mean you should now use 10x the amount of repositories to compensate, because "it's more pleasant to use Git with smaller repositories"? There's a line between bending your back for the tool, vs the other way around. Put another way: what if git was suddenly 100x faster, at every scale for every repository? Would you put more stuff in there? Some limits are real (e.g. Google has like an 80TB source repository, you're screwed there). But some of the limits are absolutely artificial in a sense. There's definitely something to be said for misusing Git in ways it shouldn't be used. And if you're Google or Microsoft or Facebook -- whatever, you have a thousand other problems. But honestly, "I want it to be able to handle large repositories" has never seemed like a fundamental "misuse" of any version control tool. It actually seems like something we've only told ourselves is a misuse, because literally every available tool is just incredibly bad at, it fundamentally.

by ekidd 10y ago

LeifCarrotson 10y ago

> This is almost certainly a result of trying to have one company-wide monolithic repository that holds the source code of hundreds or thousands of separate projects. Correct. But both Google and Microsoft use this method. And I'm sure they've put in a lot more man-years of investigation into it than you and I have!

mjn 10y ago
Google reportedly [1] also uses a virtual filesystem (FUSE-based, in their case) to let individual clients get a convenient/efficient view into their giant monorepo. Microsoft looks like they're doing something very similar here, except building their vfs/monorepo solution on top of git and open-sourcing it (Google's is built on top of an in-house VCS called Piper). [1] http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext http://cacm.acm.org/magazines/2016/7/204032-why-google-store...
- bluejekyll 10y ago
  Isn't google's system inspired from their use of Perforce as well? Is Piper Git like, or more Perforce/SVN like?
  - iainmerrick 10y ago
    Piper is a reimplementation of Perforce, because they had an existing gigantic source repository in Perforce and needed something compatible. There was an earlier project to migrate to SVN but that failed. I believe Git was never in contention due to the obvious scalability problems, but GVFS does seem like it might potentially be usable at Google.
cwyers 10y ago
Facebook and I believe Twitter, too. It's almost as if anyone who actually has to contend with problems at this scale prefers a monorepo and anyone who likes to speculate on how to solve other people's problems on message boards thinks it's a dumb idea.
- iainmerrick 10y ago
  True, but I believe a lot of both Facebook's and Twitter's dev culture comes from ex-Google employees. So there could still be some groupthink going on. (Alternatively, a victory for successful ideas?) Indirect evidence: both of them released open source clones of Google's proprietary build system "Blaze" -- Facebook's "Buck" and Twitter's "Pants". Google was actually last to the party, with "Bazel".
deleted 10y ago
[deleted]

bluejekyll 10y ago

I think GitHub/Gitlab (enterprise or public), do an excellent job of capturing the monorepo desires, but with lots of disparate code repositories. That is, mono repos are great for single company wide access, easy to fork and commit against different areas, have a single system for driving all CI changes across repos. I agree that having a single huge repo kinda sucks in Git, but it's also basically a nonstarter for companies that want all the features of git, but their repo and build is monolithic. Would you require people to first completely rebuild their software into separate Git repos before using Git? This could take years! I've been looking at this exact problem at my work, and there is no easy answer here. I think MS probably picked the easier path forward in their case.

izacus 10y ago

> Git is more pleasant when you break your codebase into isolated components. These can be pretty large—the Linux kernel has 16 million lines of code—but if your codebase is many times larger than a complete modern kernel, you might want to split it. As part of a company that switched to same kind monorepo structure form several separate repositories - there is nothing "pleasant" about having to deal with multiple Git repositories for connected components. Subtrees, submodules are utter hell of maintenance, checkout bugs (which hurt CI) and bad UX across the board ("why doesn't this build?" "you forgot to checkout submodules" "no you forgot to move the commit pointer" "no you forgot to change dependant tests because you didn't see them in an isolated repository"...) It's not a GOOD approach, but it's the best approach compared to all other more terrible ones.

luckydude 10y ago
Git would do well to copy BitKeeper's nested repository architecture. Unlike Git, a collection of nested repositories (what we call submodules) works exactly like a monolithic repository. Want to see diffs for your tree? Same command. Want to check in your changes? Same command. Want to update your collection? Same command. Etc. It's a lot more work than the hack that is submodules (we understand that hack, we did exactly the same thing for years to track outside source drops). But it is worth it, everything just works. http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/productline.pdf http://www.mcvoy.com/lm/bkdocs/productline.pdf
- krupan 10y ago
  I'm really intrigued by this. I love mercurial (and tolerate git), but they both handle sub repositories poorly. How much of my mercurial (and/or git) muscle memory would I have to unlearn to switch to bk. Just for basic commands, I mean. You could tell me to go try it out myself, but a nice side-by-side comparison showing that bk does all that git/hg do (if that's true) could go a long way to attracting git/mercurial people over to bk.
  - luckydude 10y ago

blktiger 10y ago

_Programming_ is more pleasant when you break your codebase into isolated components. Instead of trying to force git to adapt to these monolithic repositories, we should be breaking them down into manageable pieces for developers to work on. If you still need 10 years of history, just keep the old repository around for archival purposes. [Edit] Not that I'm against GVFS, there is nothing wrong with not wanting to download the entire repository in some cases.

aseipp 10y ago
Many monolithic repositories still have very clear isolation and component separation. There's a difference between "I'm doing this out of good taste" and "I'm doing this because this tool makes my life insufferable, otherwise". For dedicated organizations, with active development -- singular repos have some keen benefits, IMO. Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. I didn't even work on a multi-million line codebase. Few hundred kLOC. But with like 20 submodules though (library components), and the amount of organizational overhead is annoying as shit. (Not to mention the tools we developed to stop the active ~20 developers from footgunning themselves when they committed non-existent pointers, explaining carefully how they worked, etc). We have a binary or two, but we are anxious and careful to control and update them when we do (they were added before our time, and every update at this point on such a long-lived repo can visibly affect performance. Splitting it out is even more annoying ultimately and loses our history, and does not reduce the size of the original repo). Git is just bad here. Almost everyone is a volunteer. This kind of shit is a waste of time for everyone, frankly. What if git was 5x as slow as it was now? Would that mean we should create 5x more repositories, divide every repository up into 5 more? Delete your historic repositores with 5x frequency? Would trying to improve that mean we're trying to "force git to adapt for monolithic repositories", as you claim? There are very real limits to this kind of reasoning, bending backwards for your tools. Tools like Bitkeeper are far better at this and have a way better UI, out of the box. The "Product Lines" feature means subrepositories are much more transparent for all users (with full push/pull bidirectional interoperation, which is absolutely critical, and it does crazy things like -- gasp, cloning all subrepositores by default). But BitKeeper can also handle 50GB repositories fairly easily and quickly, including large binary files. No multi-repos necessary, no LFS. It "just works". That's excellent and it removes concerns and worries you might have later. I also think you understate the true value of continuous history. It's not something to toss aside so quickly, and any extra effort to use it means you lose a lot of insight. Maybe if you don't have to dig old code constantly (or your company throws away/rewrites its product constantly I guess). In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. New developers cannot possibly understand the rational of an obscure change 10 years ago by someone with a PhD without this kind of help, at least not easily. Some files may not have been touched in like 8 years! This repository is also relatively quick to clone, luckily. The 20 submodules make it a damper. One of my last jobs was working on a product that was 15 years old, and had only one repository. They stored binaries in it, but it had had many of them updated over the years (small JAR files, mostly, one copy of GCC). It took 6 hours to checkout, but honestly that was the main thing that sucked - performance. However, all of the same benefits applied. And that one truly was continuous -- you could get a look at the repo, as it was built for customers, years ago, to look at the behavior. Atomic refactorings across years of code, over a million lines of code, and 50+ developers and libraries was possible in a single commit. That's a good thing. But the performance is a tool deficiency, not something to be hand waved away. That second repository, the 15 year one, was like, 50GB total. That's peanuts in today's terms. Other VCSs would have been much more tolerable (including BK, or probably something like Perforce, in terms of pure speed).

aseipp 10y ago

This reasoning has very serious limits, however. Every tool influences your behavior. Imagine if Git was somehow 10x slower. It's already quite fast still, but still way slower (e.g. on Windows for some cases). Does this mean you should now use 10x the amount of repositories to compensate, because "it's more pleasant to use Git with smaller repositories"? There's a line between bending your back for the tool, vs the other way around. Put another way: what if git was suddenly 100x faster, at every scale for every repository? Would you put more stuff in there? Some limits are real (e.g. Google has like an 80TB source repository, you're screwed there). But some of the limits are absolutely artificial in a sense. There's definitely something to be said for misusing Git in ways it shouldn't be used. And if you're Google or Microsoft or Facebook -- whatever, you have a thousand other problems. But honestly, "I want it to be able to handle large repositories" has never seemed like a fundamental "misuse" of any version control tool. It actually seems like something we've only told ourselves is a misuse, because literally every available tool is just incredibly bad at, it fundamentally.