4 ms·
> Git is more pleasant when you break your codebase into isolated components. These can be pretty large—the Linux kernel has 16 million lines of code—but if you
by izacus 10y ago
> Git is more pleasant when you break your codebase into isolated components. These can be pretty large—the Linux kernel has 16 million lines of code—but if your codebase is many times larger than a complete modern kernel, you might want to split it.
As part of a company that switched to same kind monorepo structure form several separate repositories - there is nothing "pleasant" about having to deal with multiple Git repositories for connected components. Subtrees, submodules are utter hell of maintenance, checkout bugs (which hurt CI) and bad UX across the board ("why doesn't this build?" "you forgot to checkout submodules" "no you forgot to move the commit pointer" "no you forgot to change dependant tests because you didn't see them in an isolated repository"...)
It's not a GOOD approach, but it's the best approach compared to all other more terrible ones.
- luckydude 10y agoGit would do well to copy BitKeeper's nested repository architecture. Unlike Git, a collection of nested repositories (what we call submodules) works exactly like a monolithic repository. Want to see diffs for your tree? Same command. Want to check in your changes? Same command. Want to update your collection? Same command. Etc. It's a lot more work than the hack that is submodules (we understand that hack, we did exactly the same thing for years to track outside source drops). But it is worth it, everything just works. http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/nested.html http://www.mcvoy.com/lm/bkdocs/productline.pdf http://www.mcvoy.com/lm/bkdocs/productline.pdf
- krupan 10y agoI'm really intrigued by this. I love mercurial (and tolerate git), but they both handle sub repositories poorly. How much of my mercurial (and/or git) muscle memory would I have to unlearn to switch to bk. Just for basic commands, I mean. You could tell me to go try it out myself, but a nice side-by-side comparison showing that bk does all that git/hg do (if that's true) could go a long way to attracting git/mercurial people over to bk.
- luckydude 10y agoWell we support fast-import so you can get your history into BK and play trivially. BK does more and less than git/hg. It's got richer history, it was trivial to write a bk fast-export that git could import. Writing a bk fast-import wasn't so bad but an incremental one is hard because git doesn't store as much history (no per file history, actually no file object, just pathnames that have some data, not the same, we have a DAG per file, makes history better, makes merges faster and easier). Mercurial copied our UI so for a lot of stuff if you know hg <cmd> then bk <cmd> will just work. BK predates all these systems, they picked up some of bk's commands. There is a cheatsheet (somewhat out of date but it will get you going) at: http://mcvoy.com/lm/bkdocs/bk_refcard.pdf http://mcvoy.com/lm/bkdocs/bk_refcard.pdf And you can email dev <at> bitkeeper.com or hit up http://bitkeeper.org http://bitkeeper.org for more info. Cheers.
- iainmerrick 10y agoSeconded. Submodules especially are a tempting feature but never, never, ever work the way you want. Submodules used to be incredibly buggy and terrible; they've improved in recent versions of Git so now they're merely really bad.
- luckydude 10y agoThey are pretty profoundly broken in my opinion in that they take what is a distributed scm and turn it, for the submodules, into a centralized system. Here's an example. Everyone who has used a good SCM discovers a pretty useful work flow, we call it merge and test. The idea is you have N pull request sitting around. You don't want to just pull them in, you want to see if they all work together. So you clone your integration tree to a test tree. In BitKeeper, you'd then add each of the pull requests as an incoming parent (bk parent -i <repo>) and then you just merge them (bk pull). Once it's all merged, you build it, test it. Facebook does this, a lot of people do this. Why do they do it that way? To avoid polluting the main tree with garbage. It's the opposite of continuous integration (aka shoveling shit into your integration as fast as possible). Smart companies want their integration tree to be a stable base on which to build, not a layer of continuously integrated quicksand. If the tests all pass you can push the whole wad to the integration tree, or rebase each to tip (we hate this, doing that means you turned your history into CVS history and you lose all sorts of useful information). So how do submodules break that workflow? Easy, they don't support sideways pulls. If two people are working on a submodule and they have not pushed to the main tree, try and sideways pull from Jane to John and you won't get Jane's work. As a dev who worked on the nested collections, I get why that doesn't work, there are a zillion corner cases where it gets complicated dealing with that state (and there is a goldmine of test cases in the open source BK code, see src/t/t.nested* and start reading). Here's an example of an obvious thing you have to deal with (if you don't think in DAGs this is gonna hurt a little). Suppose you have a subrepo called libc. Jane has a clone of the collection as does John. Their clones are just the top repo, no subrepos are present. They both pull from different clones that have libc present and each of those clones have modified libc. So that means there is implied (different) work in Jane's missing libc and John's missing libc. Which means if Jane pulls John's clone then the libc DAG has forked and needs to be merged. BK recognizes that, even though the subrepos are not present, and tells Jane when she pulls John's collection that she needs to populate libc so she can merge it. The centralized model of submodules side steps that entire class of problems, at the cost of no sideways pulls, no workflow other than the centralized CVS like model. Submodules are the quick hack to get you semi-coupled collections, BK/nested is the hard work to make a collection have the semantics of a monolithic repo.
- woodrowbarlow 10y agodoes anyone know if mercurial is more pleasant in this regard?
- luckydude 10y agoSo one of the main developers of hg is on record saying don't use hg's answer to this problem. I haven't looked at what they have, that was enough for me to move on.
- e12e 10y agoNo, not really. Apart from the reply above about bitkeeper, I'm not aware of any new or old (d)vcs that does "sub-modules" well. There's AFAIK been some orgs working on mercurial in order to support large mono-repos - I don't know if it's better or worse than git today. I see that Illumos (opensolaris) have shifted from mercurial to git, for example: https://wiki.illumos.org/display/illumos/Mercurial+Workflow https://wiki.illumos.org/display/illumos/Mercurial+Workflow (Solaris/Sun was a somewhat early adopter of "big" mercurial repos). Working with big mono repos are not without challenges: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc... FWIW I tried using sub-modules for mercurial some years ago - in the end I realized it was probably not worth it. I'm also skeptical about go's use of "vendoring" dependencies for a similar reason.
- ekidd 10y ago> Subtrees, submodules are utter hell of maintenance, checkout bugs (which hurt CI) and bad UX across the board ("why doesn't this build?" "you forgot to checkout submodules" "no you forgot to move the commit pointer" "no you forgot to change dependant tests because you didn't see them in an isolated repository"...) I do not recommend using submodules, except in very limited cases. What I'm suggesting is that if your code base is big enough to break git (considerably larger than the Linux kernel's 16 million lines of code), you might want to break it into multiple projects with their own release cycles. Build them separately, release them via an internal release process, etc. Personally, I like largish repos. But "an order of magnitude larger than the Linux kernel" is just too big, in my opinion, at least for git-based projects. If you have 100 or 200 million lines of code, you probably have natural points to split up your architecture.