7 ms·

On Monolithic Repositories

84 points by tristanz 11y ago

jinushaun 11y ago
Someone needs to do for subrepos what git did for branches. Working with subrepos is a nightmare.
- saurik 11y ago
  Subversion ;P (the reason their branches were so awkward was because they were essentially sub repositories...)
  - lfowles 11y ago
    So we combine the best of git, subversion, and perforce. perverted-git, anyone?
    - pm 11y ago
      What's the advantage of Perforce (I don't know as I've never used it before)?
      - to3m 11y ago
        It's quite fast and it seems to scale well to large (though not necessarily to ridiculously large) repositories. The largest one I've used it with had a 1TB head revision - I don't know how large the history was - and a zillion files. Performance was still fine. (This is probably its main advantage, really: you can just put all your files in it. Then it's easy for everybody to get them, and you didn't have to think too hard about it.) It uses the check in/check out model, so there's no problem with unmergeable binary files. There are per-user access permissions. Branches are folder copies. There's a GUI tool, but you can do everything from the command line as well (I believe that is exactly how the GUI tool does everything). (UX is a bit hit or miss though. There's no git-style index, and the command line tool's output isn't as convenient to parse as you might like. On the other hand the diff/merge tool is alright and the UI for keeping your branches in sync is fine.) I've never minded using it.
tristanz 11y ago
Interesting comment on that article by Siddharth Agarwal: "(I work on source control at Facebook.) Every problem you mentioned with monolithic repositories is a well-known problem with Git (though some of them do have workarounds, such as clone with --depth). None of them are issues in principle. With Mercurial we're aiming to provide tooling that scales well while still maintaining DVCS workflows like local commits."
- lfowles 11y ago
  Related: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc...
jacques_chester 11y ago
Another way to look at this is that repos quickly ossify into unplanned Conway boundaries. Where possible, the goal is to decouple software components by design, not by backpressure from the toolchain. I've seen the many-repo approach. It's particularly frustrating on distributed systems when a shared component migrates from repo to repo like a sad ronin, sometimes alighting in some of them more than once.
- jamesrom 11y ago
  > Where possible, the goal is to decouple software components by design, not by backpressure from the toolchain. Taken to the extreme, this produces a single main.c file for an entire organisation. Good software design should dictate the types toolchain backpressure tradeoffs that need to be managed.
gnoway 11y ago
I'm sitting here running git gc and repack on about 50 repos right now, of varying sizes. We just actually combined two of our larger repos into one for productivity reasons, so this article resonates with me a little on that front. I spend a lot more time on the build and administration side of things than the code side, and I personally prefer more smaller repos. Builds are faster and less error prone, less disk space is used overall (regardless of cloning scheme - I have used them all), and I do believe the separation and inherent difficulty aids quality at the expense of productivity. I'm about the only person in the company who does, though, and that tells me this discussion depends more on how you personally interact with source control than any abstract 'monolithic vs. not' ideal. Or that I'm crazy, but I refuse to accept that.
wtbob 11y ago
I've worked with monolithic and project-based repositories, and in practice I've found the problems with monolithic repos are less than the problems with project-based repos, and the benefits of monolithic repos are more than the benefits of project-based repos. Certainly there can be issues at a very large organisation with very many extremely large projects—but most of us don't work at those organisations with that many projects that large. I think that having one large repo helps identify cross-project dependency breakage faster, e.g. on a small team without fully-automated integration tests by increasing the likelihood that the person or team who broke the integration notices rather than the person or team who maintains the affected components. There's also the issue, as jacques_chester notes, of shared components, some of which are far too small too small the be their own projects and which don't necessarily make sense thrown into a pile with other projects. Project-specific repos make a lot of sense from an organised, a-place-for-everything-and-everything-in-its-place perspective, but real life is often quite messy and mutable, and the proper organisation for a project can change frequently (as the article notes); there's no sense chiselling it into stone.
- viraptor 11y ago
  I think there's a missing distinction here: is it a monolithic repo for products available separately, or monolithic final deployable thing(s) (google has multiple products, but it's still "google services"). This may be better for them, but it wouldn't work for example for OpenStack, which has many projects available and released separately. Putting nova, stevedore, anchor, bandit, etc. in one repository just wouldn't make sense - they have their own versioning and live their own lives, even if they will be frozen/released in one go as a single working deployment in October. So when the author writes "When I am interacting with version control, I just want to get stuff done. I don't want to waste time dealing with multiple commands to manage multiple repositories." they just don't have that use case. They don't have to care about releasing different bits separately.
  - Nitramp 11y ago
    Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g. OpenStack, and different products certainly have extremely different release cycles and even deployment platforms (hardware in cars, Android phones, web servers). The question really is whether you can scale your version control practices, build tools, and source organization habits across many diverse projects.
pcwalton 11y ago
Small repositories are great for open source because they encourage code sharing. When your project consists of many pieces all owned by different, often volunteer, teams, then having one big repository is a barrier to that. For example, Servo, the project I work on, consists of around 150 small repositories (in contrast to every other browser engine). Lots of them are maintained by the Servo project, but lots of them aren't. They're separate projects of which Servo is only one user of many. The fact that we can take advantage of the fantastic work of the Rust community by simply adding a couple of lines to our Cargo files has been invaluable in helping our small team get the browser off the ground. A culture of monolithic repositories would discourage code sharing, leading to more wheel reinvention and less code collaboration overall. I think the model works well for Google and Facebook because they're big centralized companies with thousands of engineers under one management and reporting structure. But that's a far cry from the aggressive, fine-grained code reuse culture that the Ruby, JS, and Rust (for example) have fostered.
- indygreg2 11y ago
  Thought experiment: the entire github.com URL space is a single repository. Each organization/user has a top-level directory and projects/forks exist under them. Does that prevent/encourage code sharing? Why or why not?
  - viraptor 11y ago
    Prevents. Instead of cloning just one of your repos and expecting it to work, I clone `indygreg2` and need to answer: - is there one build system for everything, or one per project? - are there dependencies between projects? - are they symlink-vendored? (which means that potentially I need more than one toolchain if projects A and B are in different languages) - are they completely separate? (do they always assume latest version of your projects, or do they have reasonable version qualifiers) - are any elements included cross-project, or can I just copy one directory and package it separately? Those and other similar questions just don't exist (in well maintained projects) that have separate repos. I know that I can clone one project and build it.
TheAceOfHearts 11y ago
How do people do CI with monolithic repositories? Do they run ALL the tests or is there an easy way around that?
- justinmk 11y ago
  Your build agents should use a cached .git, and only pull updates. Different test groups can be a different build target in your build/make file, or you can have nested build files.
- brandonbloom 11y ago
  With SVN (or similary) you'd simply configure your CI to 1) checkout 1 or more files and subdirectories and 2) config your CI to run a particular build script. The notion of zero-config CI is a recent invention, thanks to conventions of small repos with well-known project file names per language/platform ecosystem.
- QuercusMax 11y ago
  At Google, the dependencies for everything are very well specified, so in most cases the code affected by a single change is quite small. Dependencies are often specified on a per file basis. It's kind of a pain in the ass, but being able to know for sure you won't break stuff is very powerful. Also, they have a ridiculous number of machines dedicated to running automated tests. My tech lead told me "feel free to add as many tests as long as they're under 30 seconds each". I was super skeptical of perforce before starting at Google, but it works really well, especially in conjunction with the code review tools and processes
  - cpeterso 11y ago
    > Dependencies are often specified on a per file basis. It's kind of a pain in the ass, but being able to know for sure you won't break stuff is very powerful. How are the dependencies specified? Is this a language-specific tool?
    - gefh 11y ago
vicapow 11y ago
I've worked both at a large company (Facebook) that used the mono repo approach as well as a large company that uses the per project repo approach (Uber) and I have to say I'm personally a VERY big fan of the project based repo approach but every company is different. So is every team. If you're a small company with primarily one service and primarily in one programming language, the mono-repo way seems to be the best approach. On the other hand, if you're a company that has embraced a service oriented architecture, the per project repo approach is likely the way to go. Especially if your company is OK with services being written in a variety of different languages and so long as it is as easy to use open source code as it is to use third party code written within your org. It also goes a long way in supporting local (ie., laptop) development. Otherwise, the entire codebase would be too big to fit in RAM. Disclaimer, these opinions do not necessarily represent the opinions of my employer.
- indygreg2 11y ago
  If your concerns are driven by resource requirements, then I posit your concerns are driven by limitations of fully distributed version control tools of today. Shallow and/or narrow clone (like the Subversion model) limit the amount of data required on clients and thus facilitate monolithic repositories without the extreme resource requirements on clients. I posit that if Git or Mercurial allowed you to clone a subset of directories, the differences between a monolithic repository and a set of smaller repositories becomes indistinguishable, as a clone of a sub-directory is functionally equivalent to a standalone repository! The problem is that narrow clone is not implemented in any popular DVCS tool today (but Mercurial is working on it).
  - glandium 11y ago
    I posit that if Git or Mercurial had better workflows for submodules, monolithic repositories would seem less attractive.
    - minot 11y ago
      atlassian recommends you consider sub tree as an option if you are looking at Git submodule
jamesrom 11y ago
Option C: Custom tooling built for the purpose of managing collections of repositories. It appears the main argument for monolithic repositories is that it improves developer productivity by giving the developer access to the entire organisation's codebase. What a terrible hack to provide something that could be better managed with other tools. No DCVS is an island.
- cpeterso 11y ago
  The Android team wrote the "repo" to manage multiple git repositories: https://source.android.com/source/developing.html https://source.android.com/source/developing.html
- stinos 11y ago
  There's also the multiple repository tool 'mr' http://linux.die.net/man/1/mr http://linux.die.net/man/1/mr which adresses exactly the I don't want to waste time dealing with multiple commands to manage multiple repositories complaint of the author and moreover can do so for repositories using different version control systems. I've been using it for years mainly because I have a few repos shared between projects and jobs, and using one monolithic repo would mean having to copy those repos around. (At least AFAIK - or how else do people using one big repository use external projects?)
rdtsc 11y ago
As someone already said, it is not "many repositores vs one" but often "many repositories + custom tooling to merge, agregate, diff, checkout, etc. vs one". We've tried both, had one large repo (2G in size, mostly due to historical mistakes of checking in large binaries). Now we have many little repos but we've spend I don't know how many man months writing custom scripts (based on gyp) to manage all these collection of repos. So far I would say it is toss-up. I would say for completely separate products that do not share almost any code it might makes sense to have separate repos. But if you find yourself building a custom version of git porcelains on top of your multiple repos -- you have probably gone too far.
KB1JWQ 11y ago
I feel like I'm in the minority here; even my dotfiles live in separate repositories, managed by vcsh (https://www.github.com/richih/vcsh https://www.github.com/richih/vcsh).
- mirceal 11y ago
  minority as in multiple repositories? eh... I don't think so. I would say a lot of people, given the choice, would go for this approach.
mirceal 11y ago
I am definitely in the multiple repository camp, but with one caveat (below). With the monolithic approach everybody is forced to be on the latest version (or to update to the latest version). There can be a lot of contention when trying to get things out - especially when things are moving fast. You need a lot of discipline to make it work and you also are forced to constantly update parts of the code you own. Also, we tend to forget about deployment and management of the build artifacts. I am willing to bet that bigCo's that do this have a separate system for managing the artifacts and performing the deployments. In the presence of a decent build and deployment system the only way to go is multiple repos - a logical separation per component / service. The build system can easily track dependencies via components via metadata associated with each of them. You can easily orchestrate builds when something happens, it's easier to rebuild only what changes and you get granular control over build artifacts.
ElHacker 11y ago
It all comes down to the scale factor. The monolithic approach makes more sense for big companies that have a large portfolio of projects, an army of software engineers and the resources to develop their own tools around a SCM. This solves several problems like multiple commits per minute, resolving ever occurring merge conflicts, code search queries and code sharing. Someone might argue that it can slowdown the development process by having to load tons of data on a local machine but this issue can be solved by creating the correct custom tooling to download a sub node of the overall repo. At this scale level this is the only feasible solution to empower developer productiveness and to avoid headaches in keeping track of a long array of projects. The project-based repositories solution has nice qualities as well: Service oriented projects, clearcut responsibilities and clear dependencies. This seems to be a good solution for a small to middle scale organization. Can design a well documented interface for every service and even expose some of those services to external clients when needed https://www.nginx.com/blog/microservices-at-netflix-architectural-best-practices/ https://www.nginx.com/blog/microservices-at-netflix-architec... . Not that this service design solution is incompatible with a monolithic approach, is just that is easier to come up with a simpler answer when having silos of knowledge. At the lower scale, where most startups reside. Either one seems to be a reasonable approach. Albeit a startup doesn't have the resources nor the time to spend on developing custom tools to manage a monolithic repo, it turns out they don't need to, because they have a small amount of developers and ongoing commit traffic. A plain simple git repo works as well as a multi repo. This is a matter of taste and the organization the founders/early employees wish to create for their project.
wpeterson 11y ago
Architecture is just dividing pain into different buckets. Large repositories are painful, as each developer bears the pain of integration every time they have to make a change to the repository integrating with all of the other code. Small, distributed repos are painful when change accrues in larger increments and must be resolved at release/integration time. As a developer in one of the largest monolithic repos in the world, I feel the pain every day of that pattern.
- wpeterson 11y ago
  Perhaps a better way of framing this is: Distributed repos make writing software easier. Monolithic repos make testing and running software easier. Large monolithic repos are an artifact of scale where the individual cost of lost productivity in development is worth the increased performance at deployment and testing.
- itsdrewmiller 11y ago
  Why do you think integration is more painful in larger repositories? Presumably the architecture of the code is unchanged.
  - wpeterson 11y ago
    When you have separate repos, the external code and integration points remain relatively static. IE: You may consume an API, but it's versioned/snapshotted at a given version. With a monolithic repo, especially without development branching - everyone is running against HEAD all the time. So while you are developing your component, everything else is changing around you.
sandGorgon 11y ago
can someone talk about how you keep track of commit messages when committing to different subfolder/subprojects of a monolithic repository ? We have 6 projects on bitbucket - and we would love to move to a monolithic model (we have already seen integration issues because of developer carelessness with deployments, that would go away)... but I'm just not sure on how we will do things like Slack/Hipchat integration, emails on commits, etc.
jcranmer 11y ago
I'm very much a fan of monolithic repositories, because I have had horrible experiences working with project-based repos. I've been around long enough to recall discussions of why people should stop using CVS, and invariably one of the bullet points on that list is "CVS lacks atomic commits." Projects that use multiple repositories invariably fall into the trap of having non-atomic commits, and people who advocate multiple repositories have likely never had the fun task of trying to do archaeology on those repositories where the sudden non-atomicity is suddenly painful. When I've brought this up before, people occasionally mention submodule or subrepositories, and those are equally broken. It makes big assumptions about how you're going to be organizing repositories (i.e., a strict tree), and if your design is not in that organization scheme, you're up a creek. For practical development, the subrepo tree effectively becomes one monorepo anyways: touch the innermost subrepo, and now you need to add a commit to all the outer repos to update the newer version of the innermost subrepo. A saner way to handle repos is to recognize that it's not necessary for people to have the full history of everything in the repository stored on their local machine most of the time. This is something that SVN does better--you can checkout subdirectories of an SVN repo, but the commits are still atomic across the entire repository.
- warkid 11y ago
  > A saner way to handle repos is to recognize that it's not necessary for people to have the full history of everything in the repository stored on their local machine most of the time. Local history enables rebasing. Of course you can do it without local history already present, say in SVN you'd use separate branches for changes and 'rebased' changes where you merge your work on top of some new state of trunk. But this involvs creating branches(and at cleast one checkout for separate branch folder) and communicating with the server(takes time). This all means that people almost never do this. With git rebase is a snap. But of course one can live without rebase, it's not oxygen or something.
  - jcranmer 11y ago
    You can download the history you need for rebasing when you do the rebase command, in effect turning the local .git (or .hg) store into basically a local cache of a remote repository.
comex 11y ago
The most interesting takeaway from me is that Mercurial is planning to support "narrow clones" soon, i.e. sparse fetches (i.e. you only have to download some subtree of the whole repository). It would be great if Git followed suit at some point - the whole monolithic versus project-based repository debate would be a lot more interesting if it were just a matter of convention, as opposed to the former being associated with "ew, SVN/Perforce/[insert old fashioned feeling tool]".
yellowapple 11y ago
I'd agree with this article's conclusions if the current-gen VCSes (like Git and Mercurial) actually provided the necessary features and tooling to effectively manage a monolithic repo. Comparing to SVN's ability to checkout portions of a repo instead of the whole repo doesn't do much good in systems that aren't designed to handle that sort of workflow. Whether or not one should go monolithic or project-based depends on how tightly integrated those projects are. Folks like Google and Facebook and OpenBSD's dev team develop all their things under a single source tree because those things are pretty tightly integrated and designed to interoperate. Other folks develop each of their things in a separate repo because they expect those things to be independently useful (OpenBSD's subprojects do happen to be independently useful, but AFAIK the priority is generally "OpenBSD first, everyone else second"). Multi-repository environments can also be very manageable depending on the language, runtime, etc. Dividing things into do-one-thing-and-do-it-well chunks (perhaps gems in Ruby land, or crates in Rust land, or packages in Perl land, or whatever) goes a long way toward alleviating the typical non-atomicity of multi-repo setups. Of course, this tends to be easier for open-source projects than closed-source projects (though the closed-source camp can have some of this fun, too; using Ruby as an example, one can run a private gem server just by running the "gem server" command, or - if using Bundler - can even install dependencies directly from Git repos), but it's still a possibility. Plus, the whole "well Google says it's more productive for them, so we should take it as a general rule" vibe doesn't really sit well with me.
Zardoz84 11y ago
Were I work we have a monolithic subversion repo that contains : - DBs ddl - Selenium automatic tests - A hierarchy with our Java code were we have our libraries, modules, final products and client personalization of our products. Our build solution relay on Maven with a nexus server and Jenkins to autodeploy maven artifacts. Pros : - We can do a single commit across all to update many modules/libs/final product to fix a issue or implement something new. So tracking a issue fix or a new feature across all is more easy. Contras : - We can't work like in git, where we can create a branch to develop a new feature or fix a issue isolated from other changes. - Enforces big commits. Actually I managed to use git svn to map our subversion repository, but I need to create many small git local repositories to track every module/lib/product/personalization that work over. Also, I have a few scripts to update my git local repos (master branch) against the subversion repo. This allowed me to be more flexible when I work on a new feature or a potential dangerous change, as I can create local branches, do local commits (and do a commit squash of they before send to subversion), stash , etc. I think that with this I grab the best of both ways of working. I can be more flexible and more productive, as I can change of branch of a project/module/lib with a few keystrokes, and do local code versioning. At same time. I only miss doing a commit across modules/libs/products when to resolve a issue I need to do change across many modules/libs. On this case, I try to use the same commit message or at least, put the issue number on the commit message, so make easy track it on fisheye.
reissbaker 11y ago
I work on the Developer Infrastructure team at Airbnb — essentially our tools team — and have some experience with both sides of the coin. Airbnb's monolithic vs project-based repository organization is currently split along language lines on the backend: the Java folks prefer a single monorepo (and have one), and the Ruby engineers use project-based repos (and have many). There are a lot of good points made about the benefits of monorepos, and at Airbnb we enjoy several of them. What hasn't been mentioned is the effort required to do them well: you need specialized build and dependency tools to ensure that you only run builds+tests for the single project that's being changed; engineers have to check out extremely large amounts of data to work on a single subdirectory, or else you need custom tooling to allow them to only check out portions while still contributing to the larger whole; if someone mistakenly breaks a piece of shared code and merges it to master, every project is now broken and engineering work may be stalled unless you have very good debugging tools and testing frameworks to quickly recover from and prevent these kinds of issues. The upfront costs of doing monorepos well are high, and doing them poorly is in my experience a net productivity loss. For large companies with established business models (and I'd consider Facebook, Google, and Airbnb to be some degree of "large" and with "established business models," although Airbnb is clearly still much smaller and pre-IPO), the tradeoff of allocating some number of engineers to work on in-house tools for a much larger engineering team is usually worthwhile, and monolithic repos start to become an attractive option. I'd caution small companies or early-stage startups against monorepos, though: when you're a twenty-person team, that amount of tools work just isn't worth it. Use open-source tools, and spend the rest of your time shipping product. TL;DR: Facebook and Google have optimized their workflows for their size; if you're not a Facebook or a Google, your mileage will probably vary.
- qznc 11y ago
  Facebook uses Mercurial. Mercurial works well for small projects as well. Mercurial has no clear technical advantage over git though. That might change if Mercurial gets narrow clones (only check out subdirectories) working and git fails to clone (haha) that.
  - glandium 11y ago
_pmf_ 11y ago
I was a proponent of project specific repositories before I actually had to work with large scale system that consistently handled development in this way. This has me taught the hard way that managing transitive SVN externals (project A refers project B as external which refers to project C as external; I've had a hierarchy of 5 levels) is an incredibly tedious affair (tagging to get a consistent, pegged state of the main project is now a multi hour process instead of a single mouse click). That alone justifies a single repository. TortoiseSVN has a feature of pegging externals at a specific release when tagging the main project, but this only works at one level.
barries1 11y ago
We use Perforce (since before DVCSs came along) and follow the monolithic approach with 100s of projects. We use git and mercurial when interfacing to clients' repositories and for small, local temporary repos to structure work that shouldn't clutter our main repository. The monolithic approach works so smoothly because Perforce makes it trivial to only check out the directories we're working on. It has huge advantages when we're working on interdependent projects. Perforce does have some historical oddities (top level directories are called "depots" and are slightly different than normal directories), but the ability to branch, merge, and check out using normal filesystem concepts is a huge usability boon.
flagZ 11y ago
I agree with the author... In the context of startups that are very much in "discovery" mode (still doing big changes to their product) breaking the repositories in subparts is just a source of confusion for the tech lead. I would also argue it is bad for devs because they then need to manage version-binding between different repos. Many repos approach is only justifiable if you have many teams with many tech leads each independently doing releases.

84 points by tristanz 11y ago

jinushaun 11y ago

Someone needs to do for subrepos what git did for branches. Working with subrepos is a nightmare.

saurik 11y ago
Subversion ;P (the reason their branches were so awkward was because they were essentially sub repositories...)
- lfowles 11y ago
  So we combine the best of git, subversion, and perforce. perverted-git, anyone?
  - pm 11y ago
    What's the advantage of Perforce (I don't know as I've never used it before)?
    - to3m 11y ago
      It's quite fast and it seems to scale well to large (though not necessarily to ridiculously large) repositories. The largest one I've used it with had a 1TB head revision - I don't know how large the history was - and a zillion files. Performance was still fine. (This is probably its main advantage, really: you can just put all your files in it. Then it's easy for everybody to get them, and you didn't have to think too hard about it.) It uses the check in/check out model, so there's no problem with unmergeable binary files. There are per-user access permissions. Branches are folder copies. There's a GUI tool, but you can do everything from the command line as well (I believe that is exactly how the GUI tool does everything). (UX is a bit hit or miss though. There's no git-style index, and the command line tool's output isn't as convenient to parse as you might like. On the other hand the diff/merge tool is alright and the UI for keeping your branches in sync is fine.) I've never minded using it.

tristanz 11y ago

Interesting comment on that article by Siddharth Agarwal: "(I work on source control at Facebook.) Every problem you mentioned with monolithic repositories is a well-known problem with Git (though some of them do have workarounds, such as clone with --depth). None of them are issues in principle. With Mercurial we're aiming to provide tooling that scales well while still maintaining DVCS workflows like local commits."

lfowles 11y ago
Related: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc...

jacques_chester 11y ago

Another way to look at this is that repos quickly ossify into unplanned Conway boundaries. Where possible, the goal is to decouple software components by design, not by backpressure from the toolchain. I've seen the many-repo approach. It's particularly frustrating on distributed systems when a shared component migrates from repo to repo like a sad ronin, sometimes alighting in some of them more than once.

jamesrom 11y ago
> Where possible, the goal is to decouple software components by design, not by backpressure from the toolchain. Taken to the extreme, this produces a single main.c file for an entire organisation. Good software design should dictate the types toolchain backpressure tradeoffs that need to be managed.

gnoway 11y ago

I'm sitting here running git gc and repack on about 50 repos right now, of varying sizes. We just actually combined two of our larger repos into one for productivity reasons, so this article resonates with me a little on that front. I spend a lot more time on the build and administration side of things than the code side, and I personally prefer more smaller repos. Builds are faster and less error prone, less disk space is used overall (regardless of cloning scheme - I have used them all), and I do believe the separation and inherent difficulty aids quality at the expense of productivity. I'm about the only person in the company who does, though, and that tells me this discussion depends more on how you personally interact with source control than any abstract 'monolithic vs. not' ideal. Or that I'm crazy, but I refuse to accept that.

wtbob 11y ago

I've worked with monolithic and project-based repositories, and in practice I've found the problems with monolithic repos are less than the problems with project-based repos, and the benefits of monolithic repos are more than the benefits of project-based repos. Certainly there can be issues at a very large organisation with very many extremely large projects—but most of us don't work at those organisations with that many projects that large. I think that having one large repo helps identify cross-project dependency breakage faster, e.g. on a small team without fully-automated integration tests by increasing the likelihood that the person or team who broke the integration notices rather than the person or team who maintains the affected components. There's also the issue, as jacques_chester notes, of shared components, some of which are far too small too small the be their own projects and which don't necessarily make sense thrown into a pile with other projects. Project-specific repos make a lot of sense from an organised, a-place-for-everything-and-everything-in-its-place perspective, but real life is often quite messy and mutable, and the proper organisation for a project can change frequently (as the article notes); there's no sense chiselling it into stone.

viraptor 11y ago
I think there's a missing distinction here: is it a monolithic repo for products available separately, or monolithic final deployable thing(s) (google has multiple products, but it's still "google services"). This may be better for them, but it wouldn't work for example for OpenStack, which has many projects available and released separately. Putting nova, stevedore, anchor, bandit, etc. in one repository just wouldn't make sense - they have their own versioning and live their own lives, even if they will be frozen/released in one go as a single working deployment in October. So when the author writes "When I am interacting with version control, I just want to get stuff done. I don't want to waste time dealing with multiple commands to manage multiple repositories." they just don't have that use case. They don't have to care about releasing different bits separately.
- Nitramp 11y ago
  Google has self driving cars, YouTube, various Android apps, and web search in the same repository. That does seem a lot more varied at the first look than e.g. OpenStack, and different products certainly have extremely different release cycles and even deployment platforms (hardware in cars, Android phones, web servers). The question really is whether you can scale your version control practices, build tools, and source organization habits across many diverse projects.

pcwalton 11y ago

Small repositories are great for open source because they encourage code sharing. When your project consists of many pieces all owned by different, often volunteer, teams, then having one big repository is a barrier to that. For example, Servo, the project I work on, consists of around 150 small repositories (in contrast to every other browser engine). Lots of them are maintained by the Servo project, but lots of them aren't. They're separate projects of which Servo is only one user of many. The fact that we can take advantage of the fantastic work of the Rust community by simply adding a couple of lines to our Cargo files has been invaluable in helping our small team get the browser off the ground. A culture of monolithic repositories would discourage code sharing, leading to more wheel reinvention and less code collaboration overall. I think the model works well for Google and Facebook because they're big centralized companies with thousands of engineers under one management and reporting structure. But that's a far cry from the aggressive, fine-grained code reuse culture that the Ruby, JS, and Rust (for example) have fostered.

indygreg2 11y ago
Thought experiment: the entire github.com URL space is a single repository. Each organization/user has a top-level directory and projects/forks exist under them. Does that prevent/encourage code sharing? Why or why not?
- viraptor 11y ago
  Prevents. Instead of cloning just one of your repos and expecting it to work, I clone `indygreg2` and need to answer: - is there one build system for everything, or one per project? - are there dependencies between projects? - are they symlink-vendored? (which means that potentially I need more than one toolchain if projects A and B are in different languages) - are they completely separate? (do they always assume latest version of your projects, or do they have reasonable version qualifiers) - are any elements included cross-project, or can I just copy one directory and package it separately? Those and other similar questions just don't exist (in well maintained projects) that have separate repos. I know that I can clone one project and build it.

TheAceOfHearts 11y ago

How do people do CI with monolithic repositories? Do they run ALL the tests or is there an easy way around that?

justinmk 11y ago
Your build agents should use a cached .git, and only pull updates. Different test groups can be a different build target in your build/make file, or you can have nested build files.
brandonbloom 11y ago
With SVN (or similary) you'd simply configure your CI to 1) checkout 1 or more files and subdirectories and 2) config your CI to run a particular build script. The notion of zero-config CI is a recent invention, thanks to conventions of small repos with well-known project file names per language/platform ecosystem.
QuercusMax 11y ago
At Google, the dependencies for everything are very well specified, so in most cases the code affected by a single change is quite small. Dependencies are often specified on a per file basis. It's kind of a pain in the ass, but being able to know for sure you won't break stuff is very powerful. Also, they have a ridiculous number of machines dedicated to running automated tests. My tech lead told me "feel free to add as many tests as long as they're under 30 seconds each". I was super skeptical of perforce before starting at Google, but it works really well, especially in conjunction with the code review tools and processes
- cpeterso 11y ago
  > Dependencies are often specified on a per file basis. It's kind of a pain in the ass, but being able to know for sure you won't break stuff is very powerful. How are the dependencies specified? Is this a language-specific tool?
  - gefh 11y ago

vicapow 11y ago

I've worked both at a large company (Facebook) that used the mono repo approach as well as a large company that uses the per project repo approach (Uber) and I have to say I'm personally a VERY big fan of the project based repo approach but every company is different. So is every team. If you're a small company with primarily one service and primarily in one programming language, the mono-repo way seems to be the best approach. On the other hand, if you're a company that has embraced a service oriented architecture, the per project repo approach is likely the way to go. Especially if your company is OK with services being written in a variety of different languages and so long as it is as easy to use open source code as it is to use third party code written within your org. It also goes a long way in supporting local (ie., laptop) development. Otherwise, the entire codebase would be too big to fit in RAM. Disclaimer, these opinions do not necessarily represent the opinions of my employer.

indygreg2 11y ago
If your concerns are driven by resource requirements, then I posit your concerns are driven by limitations of fully distributed version control tools of today. Shallow and/or narrow clone (like the Subversion model) limit the amount of data required on clients and thus facilitate monolithic repositories without the extreme resource requirements on clients. I posit that if Git or Mercurial allowed you to clone a subset of directories, the differences between a monolithic repository and a set of smaller repositories becomes indistinguishable, as a clone of a sub-directory is functionally equivalent to a standalone repository! The problem is that narrow clone is not implemented in any popular DVCS tool today (but Mercurial is working on it).
- glandium 11y ago
  I posit that if Git or Mercurial had better workflows for submodules, monolithic repositories would seem less attractive.
  - minot 11y ago
    atlassian recommends you consider sub tree as an option if you are looking at Git submodule

jamesrom 11y ago

Option C: Custom tooling built for the purpose of managing collections of repositories. It appears the main argument for monolithic repositories is that it improves developer productivity by giving the developer access to the entire organisation's codebase. What a terrible hack to provide something that could be better managed with other tools. No DCVS is an island.

cpeterso 11y ago
The Android team wrote the "repo" to manage multiple git repositories: https://source.android.com/source/developing.html https://source.android.com/source/developing.html
stinos 11y ago
There's also the multiple repository tool 'mr' http://linux.die.net/man/1/mr http://linux.die.net/man/1/mr which adresses exactly the I don't want to waste time dealing with multiple commands to manage multiple repositories complaint of the author and moreover can do so for repositories using different version control systems. I've been using it for years mainly because I have a few repos shared between projects and jobs, and using one monolithic repo would mean having to copy those repos around. (At least AFAIK - or how else do people using one big repository use external projects?)

rdtsc 11y ago

As someone already said, it is not "many repositores vs one" but often "many repositories + custom tooling to merge, agregate, diff, checkout, etc. vs one". We've tried both, had one large repo (2G in size, mostly due to historical mistakes of checking in large binaries). Now we have many little repos but we've spend I don't know how many man months writing custom scripts (based on gyp) to manage all these collection of repos. So far I would say it is toss-up. I would say for completely separate products that do not share almost any code it might makes sense to have separate repos. But if you find yourself building a custom version of git porcelains on top of your multiple repos -- you have probably gone too far.

KB1JWQ 11y ago

I feel like I'm in the minority here; even my dotfiles live in separate repositories, managed by vcsh (https://www.github.com/richih/vcsh https://www.github.com/richih/vcsh).

mirceal 11y ago
minority as in multiple repositories? eh... I don't think so. I would say a lot of people, given the choice, would go for this approach.

mirceal 11y ago

I am definitely in the multiple repository camp, but with one caveat (below). With the monolithic approach everybody is forced to be on the latest version (or to update to the latest version). There can be a lot of contention when trying to get things out - especially when things are moving fast. You need a lot of discipline to make it work and you also are forced to constantly update parts of the code you own. Also, we tend to forget about deployment and management of the build artifacts. I am willing to bet that bigCo's that do this have a separate system for managing the artifacts and performing the deployments. In the presence of a decent build and deployment system the only way to go is multiple repos - a logical separation per component / service. The build system can easily track dependencies via components via metadata associated with each of them. You can easily orchestrate builds when something happens, it's easier to rebuild only what changes and you get granular control over build artifacts.

ElHacker 11y ago

It all comes down to the scale factor. The monolithic approach makes more sense for big companies that have a large portfolio of projects, an army of software engineers and the resources to develop their own tools around a SCM. This solves several problems like multiple commits per minute, resolving ever occurring merge conflicts, code search queries and code sharing. Someone might argue that it can slowdown the development process by having to load tons of data on a local machine but this issue can be solved by creating the correct custom tooling to download a sub node of the overall repo. At this scale level this is the only feasible solution to empower developer productiveness and to avoid headaches in keeping track of a long array of projects. The project-based repositories solution has nice qualities as well: Service oriented projects, clearcut responsibilities and clear dependencies. This seems to be a good solution for a small to middle scale organization. Can design a well documented interface for every service and even expose some of those services to external clients when needed https://www.nginx.com/blog/microservices-at-netflix-architectural-best-practices/ https://www.nginx.com/blog/microservices-at-netflix-architec... . Not that this service design solution is incompatible with a monolithic approach, is just that is easier to come up with a simpler answer when having silos of knowledge. At the lower scale, where most startups reside. Either one seems to be a reasonable approach. Albeit a startup doesn't have the resources nor the time to spend on developing custom tools to manage a monolithic repo, it turns out they don't need to, because they have a small amount of developers and ongoing commit traffic. A plain simple git repo works as well as a multi repo. This is a matter of taste and the organization the founders/early employees wish to create for their project.

wpeterson 11y ago

Architecture is just dividing pain into different buckets. Large repositories are painful, as each developer bears the pain of integration every time they have to make a change to the repository integrating with all of the other code. Small, distributed repos are painful when change accrues in larger increments and must be resolved at release/integration time. As a developer in one of the largest monolithic repos in the world, I feel the pain every day of that pattern.

wpeterson 11y ago
Perhaps a better way of framing this is: Distributed repos make writing software easier. Monolithic repos make testing and running software easier. Large monolithic repos are an artifact of scale where the individual cost of lost productivity in development is worth the increased performance at deployment and testing.
itsdrewmiller 11y ago
Why do you think integration is more painful in larger repositories? Presumably the architecture of the code is unchanged.
- wpeterson 11y ago
  When you have separate repos, the external code and integration points remain relatively static. IE: You may consume an API, but it's versioned/snapshotted at a given version. With a monolithic repo, especially without development branching - everyone is running against HEAD all the time. So while you are developing your component, everything else is changing around you.

sandGorgon 11y ago

can someone talk about how you keep track of commit messages when committing to different subfolder/subprojects of a monolithic repository ? We have 6 projects on bitbucket - and we would love to move to a monolithic model (we have already seen integration issues because of developer carelessness with deployments, that would go away)... but I'm just not sure on how we will do things like Slack/Hipchat integration, emails on commits, etc.

jcranmer 11y ago

I'm very much a fan of monolithic repositories, because I have had horrible experiences working with project-based repos. I've been around long enough to recall discussions of why people should stop using CVS, and invariably one of the bullet points on that list is "CVS lacks atomic commits." Projects that use multiple repositories invariably fall into the trap of having non-atomic commits, and people who advocate multiple repositories have likely never had the fun task of trying to do archaeology on those repositories where the sudden non-atomicity is suddenly painful. When I've brought this up before, people occasionally mention submodule or subrepositories, and those are equally broken. It makes big assumptions about how you're going to be organizing repositories (i.e., a strict tree), and if your design is not in that organization scheme, you're up a creek. For practical development, the subrepo tree effectively becomes one monorepo anyways: touch the innermost subrepo, and now you need to add a commit to all the outer repos to update the newer version of the innermost subrepo. A saner way to handle repos is to recognize that it's not necessary for people to have the full history of everything in the repository stored on their local machine most of the time. This is something that SVN does better--you can checkout subdirectories of an SVN repo, but the commits are still atomic across the entire repository.

warkid 11y ago
> A saner way to handle repos is to recognize that it's not necessary for people to have the full history of everything in the repository stored on their local machine most of the time. Local history enables rebasing. Of course you can do it without local history already present, say in SVN you'd use separate branches for changes and 'rebased' changes where you merge your work on top of some new state of trunk. But this involvs creating branches(and at cleast one checkout for separate branch folder) and communicating with the server(takes time). This all means that people almost never do this. With git rebase is a snap. But of course one can live without rebase, it's not oxygen or something.
- jcranmer 11y ago
  You can download the history you need for rebasing when you do the rebase command, in effect turning the local .git (or .hg) store into basically a local cache of a remote repository.

comex 11y ago

The most interesting takeaway from me is that Mercurial is planning to support "narrow clones" soon, i.e. sparse fetches (i.e. you only have to download some subtree of the whole repository). It would be great if Git followed suit at some point - the whole monolithic versus project-based repository debate would be a lot more interesting if it were just a matter of convention, as opposed to the former being associated with "ew, SVN/Perforce/[insert old fashioned feeling tool]".

yellowapple 11y ago

I'd agree with this article's conclusions if the current-gen VCSes (like Git and Mercurial) actually provided the necessary features and tooling to effectively manage a monolithic repo. Comparing to SVN's ability to checkout portions of a repo instead of the whole repo doesn't do much good in systems that aren't designed to handle that sort of workflow. Whether or not one should go monolithic or project-based depends on how tightly integrated those projects are. Folks like Google and Facebook and OpenBSD's dev team develop all their things under a single source tree because those things are pretty tightly integrated and designed to interoperate. Other folks develop each of their things in a separate repo because they expect those things to be independently useful (OpenBSD's subprojects do happen to be independently useful, but AFAIK the priority is generally "OpenBSD first, everyone else second"). Multi-repository environments can also be very manageable depending on the language, runtime, etc. Dividing things into do-one-thing-and-do-it-well chunks (perhaps gems in Ruby land, or crates in Rust land, or packages in Perl land, or whatever) goes a long way toward alleviating the typical non-atomicity of multi-repo setups. Of course, this tends to be easier for open-source projects than closed-source projects (though the closed-source camp can have some of this fun, too; using Ruby as an example, one can run a private gem server just by running the "gem server" command, or - if using Bundler - can even install dependencies directly from Git repos), but it's still a possibility. Plus, the whole "well Google says it's more productive for them, so we should take it as a general rule" vibe doesn't really sit well with me.

Zardoz84 11y ago

Were I work we have a monolithic subversion repo that contains : - DBs ddl - Selenium automatic tests - A hierarchy with our Java code were we have our libraries, modules, final products and client personalization of our products. Our build solution relay on Maven with a nexus server and Jenkins to autodeploy maven artifacts. Pros : - We can do a single commit across all to update many modules/libs/final product to fix a issue or implement something new. So tracking a issue fix or a new feature across all is more easy. Contras : - We can't work like in git, where we can create a branch to develop a new feature or fix a issue isolated from other changes. - Enforces big commits. Actually I managed to use git svn to map our subversion repository, but I need to create many small git local repositories to track every module/lib/product/personalization that work over. Also, I have a few scripts to update my git local repos (master branch) against the subversion repo. This allowed me to be more flexible when I work on a new feature or a potential dangerous change, as I can create local branches, do local commits (and do a commit squash of they before send to subversion), stash , etc. I think that with this I grab the best of both ways of working. I can be more flexible and more productive, as I can change of branch of a project/module/lib with a few keystrokes, and do local code versioning. At same time. I only miss doing a commit across modules/libs/products when to resolve a issue I need to do change across many modules/libs. On this case, I try to use the same commit message or at least, put the issue number on the commit message, so make easy track it on fisheye.

reissbaker 11y ago

I work on the Developer Infrastructure team at Airbnb — essentially our tools team — and have some experience with both sides of the coin. Airbnb's monolithic vs project-based repository organization is currently split along language lines on the backend: the Java folks prefer a single monorepo (and have one), and the Ruby engineers use project-based repos (and have many). There are a lot of good points made about the benefits of monorepos, and at Airbnb we enjoy several of them. What hasn't been mentioned is the effort required to do them well: you need specialized build and dependency tools to ensure that you only run builds+tests for the single project that's being changed; engineers have to check out extremely large amounts of data to work on a single subdirectory, or else you need custom tooling to allow them to only check out portions while still contributing to the larger whole; if someone mistakenly breaks a piece of shared code and merges it to master, every project is now broken and engineering work may be stalled unless you have very good debugging tools and testing frameworks to quickly recover from and prevent these kinds of issues. The upfront costs of doing monorepos well are high, and doing them poorly is in my experience a net productivity loss. For large companies with established business models (and I'd consider Facebook, Google, and Airbnb to be some degree of "large" and with "established business models," although Airbnb is clearly still much smaller and pre-IPO), the tradeoff of allocating some number of engineers to work on in-house tools for a much larger engineering team is usually worthwhile, and monolithic repos start to become an attractive option. I'd caution small companies or early-stage startups against monorepos, though: when you're a twenty-person team, that amount of tools work just isn't worth it. Use open-source tools, and spend the rest of your time shipping product. TL;DR: Facebook and Google have optimized their workflows for their size; if you're not a Facebook or a Google, your mileage will probably vary.

qznc 11y ago
Facebook uses Mercurial. Mercurial works well for small projects as well. Mercurial has no clear technical advantage over git though. That might change if Mercurial gets narrow clones (only check out subdirectories) working and git fails to clone (haha) that.
- glandium 11y ago

_pmf_ 11y ago

I was a proponent of project specific repositories before I actually had to work with large scale system that consistently handled development in this way. This has me taught the hard way that managing transitive SVN externals (project A refers project B as external which refers to project C as external; I've had a hierarchy of 5 levels) is an incredibly tedious affair (tagging to get a consistent, pegged state of the main project is now a multi hour process instead of a single mouse click). That alone justifies a single repository. TortoiseSVN has a feature of pegging externals at a specific release when tagging the main project, but this only works at one level.

barries1 11y ago

We use Perforce (since before DVCSs came along) and follow the monolithic approach with 100s of projects. We use git and mercurial when interfacing to clients' repositories and for small, local temporary repos to structure work that shouldn't clutter our main repository. The monolithic approach works so smoothly because Perforce makes it trivial to only check out the directories we're working on. It has huge advantages when we're working on interdependent projects. Perforce does have some historical oddities (top level directories are called "depots" and are slightly different than normal directories), but the ability to branch, merge, and check out using normal filesystem concepts is a huge usability boon.

flagZ 11y ago

I agree with the author... In the context of startups that are very much in "discovery" mode (still doing big changes to their product) breaking the repositories in subparts is just a source of confusion for the tech lead. I would also argue it is bad for devs because they then need to manage version-binding between different repos. Many repos approach is only justifiable if you have many teams with many tech leads each independently doing releases.