4 ms·
I work on the Developer Infrastructure team at Airbnb — essentially our tools team — and have some experience with both sides of the coin. Airbnb's monolithic v
by reissbaker 11y ago
I work on the Developer Infrastructure team at Airbnb — essentially our tools team — and have some experience with both sides of the coin. Airbnb's monolithic vs project-based repository organization is currently split along language lines on the backend: the Java folks prefer a single monorepo (and have one), and the Ruby engineers use project-based repos (and have many).
There are a lot of good points made about the benefits of monorepos, and at Airbnb we enjoy several of them. What hasn't been mentioned is the effort required to do them well: you need specialized build and dependency tools to ensure that you only run builds+tests for the single project that's being changed; engineers have to check out extremely large amounts of data to work on a single subdirectory, or else you need custom tooling to allow them to only check out portions while still contributing to the larger whole; if someone mistakenly breaks a piece of shared code and merges it to master, every project is now broken and engineering work may be stalled unless you have very good debugging tools and testing frameworks to quickly recover from and prevent these kinds of issues.
The upfront costs of doing monorepos well are high, and doing them poorly is in my experience a net productivity loss. For large companies with established business models (and I'd consider Facebook, Google, and Airbnb to be some degree of "large" and with "established business models," although Airbnb is clearly still much smaller and pre-IPO), the tradeoff of allocating some number of engineers to work on in-house tools for a much larger engineering team is usually worthwhile, and monolithic repos start to become an attractive option. I'd caution small companies or early-stage startups against monorepos, though: when you're a twenty-person team, that amount of tools work just isn't worth it. Use open-source tools, and spend the rest of your time shipping product.
TL;DR: Facebook and Google have optimized their workflows for their size; if you're not a Facebook or a Google, your mileage will probably vary.
- qznc 11y agoFacebook uses Mercurial. Mercurial works well for small projects as well. Mercurial has no clear technical advantage over git though. That might change if Mercurial gets narrow clones (only check out subdirectories) working and git fails to clone (haha) that.
- glandium 11y agoIIRC there were experimental patches implementing it for git a few months ago. Ironically, to make narrow clones happen, Mercurial is going to change its internal structure representing the files in the repository (the manifest) to use a separate manifest for each sub-directory... like git has been doing for trees from day one.
- qznc 11y agoI found patches from 2008, but nothing recent. http://article.gmane.org/gmane.comp.version-control.git/96349/match=narrow+clone http://article.gmane.org/gmane.comp.version-control.git/9634...
- glandium 11y agoYour link is about narrow checkout, not narrow clone (so the .git data is still transfered completely, as opposed to transfered partially). Maybe what I'm thinking about was just a discussion with no formal patch, I don't remember.
- davvid 11y agoThe two features of interest are "shallow clone" and "partial checkout". Both of these features are already part of git. A current experiment is an untracked files cache, which speeds up stuff on large repos considerably. This stuff is actively being worked on -- the git project has always valued performance.
- qznc 11y agoNo, "shallow clone" and "partial checkout" are both different to "narrow clone". Maybe "partial clone" would fit the git terminology better. A shallow clone misses history after a certain point in time (well, commit order). A narrow clone misses history except in a subtree. A partial clone looks like a narrow clone in the workspace, but a narrow clone should have substantially less objects to download and store.