7 ms·

Why your company shouldn't use Git submodules

64 points by aiiane 15y ago

irrationalidiom 15y ago
having a controlled state for foreign repos is absolutely essential. having worked with all-in-one repos, where external stuff is thown in... then rots... submodules are a better way, making keeping external code up to date. simple yet controlled.
- andywhite37 15y ago
  Yeah, I agree, submodules prevent the copy and paste rot that can happen when you copy library code directly into your repo. We've been using submodules for most of our projects, and I don't think they are that bad as long as you are just a little extra careful with managing them.
sharken 15y ago
Agree with most of the posters, that Git submodules are very hard to work with. The key to submodules is that you should not update them on a regular basis. A good example is the Gitflow project that uses the shFlags repo as a submodule. A small gotcha is that you need to use --recursive when cloning the repo, so that you get the submodule cloned as well.
cpt1138 15y ago
gitslave probably is closer to what you want
- zoul 15y ago
  The problem with all wrappers and extensions is that you're no longer using vanilla Git, which probably breaks other tools like GUI clients. Plus the wrappers and extensions come with their own caveats, so it's still perfectly possible to get the repo into some broken state, only this time your scenario is even more esoteric. I so wish submodules got a better treatment in the vanilla Git.
rogerbinns 15y ago
I've also tried and failed several times to use submodules. Live would be a lot easier for my situations if the parent could always point to the head of the child instead of a specific revision.
- enobrev 15y ago
  Absolutely. I certainly understand Why it works the way it does. Especially when dealing with 3rd party modules, it's good to be explicit about which version of a submodule should be included. But having a setting that allows me to keep a submodule at HEAD at all times would certainly help me save quite a few keystrokes for my own internal libraries - most of which I quite regularly find myself crawling up the tree to add, commit, and push.
- exDM69 15y ago
  No it would not be a good idea for the submodules to be always pointing to the HEAD of their repos. It might make sense for syncing with some very stable external projects but that is a very limited use case. I need to have a consistent set of all the submodules I'm working with. I need to reliably get the exact same versions of all the modules in the big repository. This allows me to do a "git bisect" to search for problems in submodules. Doing "git submodule update" is not as bad as the OP suggests it is. It makes perfect sense to have it as it is.
  - etherealG 15y ago
    I think there's usefulness in being able to specify both things. I like the way it is now for a default, but like you say for some stable branch of another repository, it would be nice if submodules could be updated automatically.
    - buddydvd 15y ago
      You could specify a directory in .gitignore and create a script that clones/pulls the repository of your choosing into that directory.
saurik 15y ago
There are serious problems with git submodules. This article, however, is simply concentrating on "you can forget to do X or not understand Y, in which case you can cause yourself minor irritations", which is just silly: if you understand how to use submodules all of the problems in this article go away and get replaced with more serious issues like "the submodule update mechanism doesn't get rid of obsolete submodules", "submodules can only exist in the root folder", "the mechanism for migrating between different upstream sources of a submodule (which will happen: this is a distributed version control system) require coordination with people using the code", and "for many people, who are attempting to use this in a context of a unified company, the lack of a solution to moving code back/forth between submodules makes moving to git a major step down from Subversion, where the subtree repository support made it entirely reasonable to store an entire organization's worth off projects, with binary art assets, together in a single master repository".
- ryanpetrich 15y ago
  git submodule add <remote> <subpath/into/the/project> will create a submodule in a subfolder.
  - saurik 15y ago
    Ah, touche: I forgot that that one was just an issue with the tool and not a fundamental limitation. (Sadly, the same kind of "that isn't a real problem: just a 'you don't know enough'" that I'm complaining about from the article ;P.)
  - pyre 15y ago
    I think that the issue is that <subpath/into/the/project> always has to be relative to the root directory. You can't do something like: % cd subpath % git submodule add <remote> <into/the/project>
- pyre 15y ago
  > a major step down from Subversion, where the subtree > repository support You could always checkout git-subtree. The functionality is there under the hood in git (git-subtree is literally a shell script, though not a minor one).
exDM69 15y ago
I've said this before but I'll say it again. Stay the hell away from Google's Repo tool. It's a half-baked badly maintained piece of ad-hoc software. It will completely destroy your git workflow. You'll also be married to the crappy review tool called Gerrit. Repo was made prior to Git submodules to do the exact same thing for Android. Now that Git has submodule support, Repo is useless. It does pretty much the same thing as submodules, but it does it in a very crappy way. For example, you cannot go back to a specific set of versions of subdirectories with Repo. In other words there's no "global" git bisect as there would be with submodules. This stops you from automatically finding a problem in one of the submodule repos if there are dependencies betweeen the problematic repo and other repositories.
- valley_guy_12 15y ago
  Repo seems to work the way it does on purpose: "...the reason we made repo was because we didn't want to deal with commits in the super project, or trying to merge concurrent branches in the super project. Instead we wanted each subproject to use a floating branch as the revision it is tracking." https://groups.google.com/d/msg/repo-discuss/ZpqOOE5mLXo/Sw0NHh8VibYJ https://groups.google.com/d/msg/repo-discuss/ZpqOOE5mLXo/Sw0...
  - exDM69 15y ago
    If you ask me, it was the wrong choice. I work on a huge project that's hosted on Repo (full Android OS + big proprietary driver tree), and the lack of this feature has really cost me lots of time doing manual labor that could be automated. Another decision the Repo designers made that I disagree with is the silly Change-Id added to every commit message(!). Since when do globally unique identifiers (GUIDs) solve problems instead of creating more?
    - duskwuff 15y ago
      That's an artifact of Gerrit, not Repo.
- mindjiver 15y ago
buddydvd 15y ago
Git's submodule feature definitely has rough edges, however, I think the benefits outweigh the cons. One of the best benefits I see is that submodules make embedding forks more manageble. For example, when you include an open source library in your own project, it's common that you'd want to modify the library in some way. If you commit the modified library into your own project's repository, you'll have a harder time absorbing bug fixes/features from upstream later on. Instead of a simple merge, you'd have to check out the updated library somewhere else and use a diff tool to compare the changes. And, if the library changed much, you may need to find the specific commit that your modification were based off so you can understand how to rebase your modifications. In addition, submodules makes it easier to: contribute bug fixes/patches, share modified open source library across different projects, and identify bugs introduced in updated submodules (since the history is preserved).
- alexchamberlain 15y ago
  If you want to make changes to the upstream, I suggest adding an extra mirror in between the upstream and the repo you want to add the submodule in. That way, you can maintain your version without worrying about how your other code is affected. Only after it is tested do you then merge it into your other repo.
  - buddydvd 15y ago
    The projects I add as submodules are all hosted on Github. For those that require modification, I would fork (mirror) the project, commit the changes to my fork, and add the fork as a submodule to the parent project I'm working on. So, yes, I do add an extra mirror in between but it's really because I lack contributor permission to the main repository. Also, if repository A and B reference repository C as a submodule and I update repository C for A, B would not be affected since submodule references are just commit IDs. There's no need to create two additional mirrors (C-for-A and C-for-B) if that's what you meant.
etherealG 15y ago
I think git subtree seems like the most sane solution here, the ability to split out updates to the subtree back upstream makes them just as useful as submodules without all the pain.
aoprisan 15y ago
So "if you forgot to run git submodule update, you’ve just reverted any submodule commits the branch you merged in might have made"... Yes, if you forget to type the correct commands then undesired effects will happen. Now that's a git design flaw? Give me a break.
- brazzy 15y ago
  So you'd also say that a car that explodes if you forget to press the "don't explode" button after refuelling is a perfectly good design? Even if you think you're perfect and never make mistakes, it's not a good idea to add easy opportunities to make mistakes.
  - exDM69 15y ago
    Unlike an exploding car, Git has undo. See git reflog.
moe 15y ago
Very good article. The question is, will it ever be fixed? As it stands submodules don't even work sanely for the most trivial use-case of tracking a (slow changing) vendor-repo. Whoever designed this (Linus?) had a real brainfart here.
- harshreality 15y ago
  I wonder. How difficult can it be to mimic mercurial's subrepos? http://blog.codekills.net/2011/07/14/nested-repository-handling-in-git-and-mercurial/ http://blog.codekills.net/2011/07/14/nested-repository-handl...
  - moe 15y ago
    Yes! That should be built straight into git. The abstraction leaks are no worse than with submodules (it's just making different trade-offs) but the usability is so much more sane, it's not funny. The importance of having this in core-git can not be overstated. When it's not in core then it's not getting used. Case in point: count the number of projects on github using one of the external training wheels (subtree etc.). Submodules has proven inadequate for reality (again: count the number of subprojects on github). I wish one of the core-dev's would make a kickstarter for a solution...
- exDM69 15y ago
  > As it stands submodules don't even work sanely for the most trivial use-case of tracking a (slow changing) vendor-repo. That is not the use case it was designed for. And submodules still work just fine for that. The fact that you have to do a separate submodule update does not make it broken. In fact, it makes it better when you start thinking about all the corner cases there are. If you truly want to build a software out of subrepositories, you need to have a sane way of tracking working sets of revisions of all the submodules there are. And that is what git submodules does. If you were blindly tracking the head of a repository, it would be useful _only_ for the use case you mentioned. Tracking slowly changing external vendor repos. That would be a whole lot less useful.
  - 15y ago
jacobr 15y ago
Submodules are pretty annoying if you modify the submodule repo a lot, but a pre-commit hook would help against the "forgot to push in submodule" issue, no?
- exDM69 15y ago
  Or a simple git or shell alias which would do the submodule update when you want it. That would solve most of the sources of complaints in the article and in this thread. However, most likely you don't want an automatic submodule update because of all the issues there are. It would only be useful for tracking a very stable slow moving external dependency.
bryanlarsen 15y ago
It appears that one of the solutions he recommends, git-subtree, is going to be merged into git soon: http://git.661346.n2.nabble.com/git-subtree-Next-Round-Ready-td7404309.html http://git.661346.n2.nabble.com/git-subtree-Next-Round-Ready... This is excellent news. I've been using git subtree for a couple of years now without incident, and highly recommend it.
- aiiane 15y ago
  she.
- harshreality 15y ago
  Recent discussions about submodules and the subtree script on the mailing list: http://thread.gmane.org/gmane.comp.version-control.git/196484 http://thread.gmane.org/gmane.comp.version-control.git/19648... http://thread.gmane.org/gmane.comp.version-control.git/195604 http://thread.gmane.org/gmane.comp.version-control.git/19560...
- bryanlarsen 14y ago
  WOOT: https://github.com/apenwarr/git-subtree/commit/8cd698957f57f62ec3d313403bebced2fdf751e5 https://github.com/apenwarr/git-subtree/commit/8cd698957f57f...

64 points by aiiane 15y ago

having a controlled state for foreign repos is absolutely essential. having worked with all-in-one repos, where external stuff is thown in... then rots... submodules are a better way, making keeping external code up to date. simple yet controlled.

andywhite37 15y ago
Yeah, I agree, submodules prevent the copy and paste rot that can happen when you copy library code directly into your repo. We've been using submodules for most of our projects, and I don't think they are that bad as long as you are just a little extra careful with managing them.

sharken 15y ago

Agree with most of the posters, that Git submodules are very hard to work with. The key to submodules is that you should not update them on a regular basis. A good example is the Gitflow project that uses the shFlags repo as a submodule. A small gotcha is that you need to use --recursive when cloning the repo, so that you get the submodule cloned as well.

cpt1138 15y ago

gitslave probably is closer to what you want

zoul 15y ago
The problem with all wrappers and extensions is that you're no longer using vanilla Git, which probably breaks other tools like GUI clients. Plus the wrappers and extensions come with their own caveats, so it's still perfectly possible to get the repo into some broken state, only this time your scenario is even more esoteric. I so wish submodules got a better treatment in the vanilla Git.

rogerbinns 15y ago

I've also tried and failed several times to use submodules. Live would be a lot easier for my situations if the parent could always point to the head of the child instead of a specific revision.

enobrev 15y ago
Absolutely. I certainly understand Why it works the way it does. Especially when dealing with 3rd party modules, it's good to be explicit about which version of a submodule should be included. But having a setting that allows me to keep a submodule at HEAD at all times would certainly help me save quite a few keystrokes for my own internal libraries - most of which I quite regularly find myself crawling up the tree to add, commit, and push.
exDM69 15y ago
No it would not be a good idea for the submodules to be always pointing to the HEAD of their repos. It might make sense for syncing with some very stable external projects but that is a very limited use case. I need to have a consistent set of all the submodules I'm working with. I need to reliably get the exact same versions of all the modules in the big repository. This allows me to do a "git bisect" to search for problems in submodules. Doing "git submodule update" is not as bad as the OP suggests it is. It makes perfect sense to have it as it is.
- etherealG 15y ago
  I think there's usefulness in being able to specify both things. I like the way it is now for a default, but like you say for some stable branch of another repository, it would be nice if submodules could be updated automatically.
  - buddydvd 15y ago
    You could specify a directory in .gitignore and create a script that clones/pulls the repository of your choosing into that directory.

saurik 15y ago

There are serious problems with git submodules. This article, however, is simply concentrating on "you can forget to do X or not understand Y, in which case you can cause yourself minor irritations", which is just silly: if you understand how to use submodules all of the problems in this article go away and get replaced with more serious issues like "the submodule update mechanism doesn't get rid of obsolete submodules", "submodules can only exist in the root folder", "the mechanism for migrating between different upstream sources of a submodule (which will happen: this is a distributed version control system) require coordination with people using the code", and "for many people, who are attempting to use this in a context of a unified company, the lack of a solution to moving code back/forth between submodules makes moving to git a major step down from Subversion, where the subtree repository support made it entirely reasonable to store an entire organization's worth off projects, with binary art assets, together in a single master repository".

ryanpetrich 15y ago
git submodule add <remote> <subpath/into/the/project> will create a submodule in a subfolder.
- saurik 15y ago
  Ah, touche: I forgot that that one was just an issue with the tool and not a fundamental limitation. (Sadly, the same kind of "that isn't a real problem: just a 'you don't know enough'" that I'm complaining about from the article ;P.)
- pyre 15y ago
  I think that the issue is that <subpath/into/the/project> always has to be relative to the root directory. You can't do something like: % cd subpath % git submodule add <remote> <into/the/project>
pyre 15y ago
> a major step down from Subversion, where the subtree > repository support You could always checkout git-subtree. The functionality is there under the hood in git (git-subtree is literally a shell script, though not a minor one).

exDM69 15y ago

I've said this before but I'll say it again. Stay the hell away from Google's Repo tool. It's a half-baked badly maintained piece of ad-hoc software. It will completely destroy your git workflow. You'll also be married to the crappy review tool called Gerrit. Repo was made prior to Git submodules to do the exact same thing for Android. Now that Git has submodule support, Repo is useless. It does pretty much the same thing as submodules, but it does it in a very crappy way. For example, you cannot go back to a specific set of versions of subdirectories with Repo. In other words there's no "global" git bisect as there would be with submodules. This stops you from automatically finding a problem in one of the submodule repos if there are dependencies betweeen the problematic repo and other repositories.

valley_guy_12 15y ago
Repo seems to work the way it does on purpose: "...the reason we made repo was because we didn't want to deal with commits in the super project, or trying to merge concurrent branches in the super project. Instead we wanted each subproject to use a floating branch as the revision it is tracking." https://groups.google.com/d/msg/repo-discuss/ZpqOOE5mLXo/Sw0NHh8VibYJ https://groups.google.com/d/msg/repo-discuss/ZpqOOE5mLXo/Sw0...
- exDM69 15y ago
  If you ask me, it was the wrong choice. I work on a huge project that's hosted on Repo (full Android OS + big proprietary driver tree), and the lack of this feature has really cost me lots of time doing manual labor that could be automated. Another decision the Repo designers made that I disagree with is the silly Change-Id added to every commit message(!). Since when do globally unique identifiers (GUIDs) solve problems instead of creating more?
  - duskwuff 15y ago
    That's an artifact of Gerrit, not Repo.
mindjiver 15y ago

buddydvd 15y ago

Git's submodule feature definitely has rough edges, however, I think the benefits outweigh the cons. One of the best benefits I see is that submodules make embedding forks more manageble. For example, when you include an open source library in your own project, it's common that you'd want to modify the library in some way. If you commit the modified library into your own project's repository, you'll have a harder time absorbing bug fixes/features from upstream later on. Instead of a simple merge, you'd have to check out the updated library somewhere else and use a diff tool to compare the changes. And, if the library changed much, you may need to find the specific commit that your modification were based off so you can understand how to rebase your modifications. In addition, submodules makes it easier to: contribute bug fixes/patches, share modified open source library across different projects, and identify bugs introduced in updated submodules (since the history is preserved).

alexchamberlain 15y ago
If you want to make changes to the upstream, I suggest adding an extra mirror in between the upstream and the repo you want to add the submodule in. That way, you can maintain your version without worrying about how your other code is affected. Only after it is tested do you then merge it into your other repo.
- buddydvd 15y ago
  The projects I add as submodules are all hosted on Github. For those that require modification, I would fork (mirror) the project, commit the changes to my fork, and add the fork as a submodule to the parent project I'm working on. So, yes, I do add an extra mirror in between but it's really because I lack contributor permission to the main repository. Also, if repository A and B reference repository C as a submodule and I update repository C for A, B would not be affected since submodule references are just commit IDs. There's no need to create two additional mirrors (C-for-A and C-for-B) if that's what you meant.

etherealG 15y ago

I think git subtree seems like the most sane solution here, the ability to split out updates to the subtree back upstream makes them just as useful as submodules without all the pain.

aoprisan 15y ago

So "if you forgot to run git submodule update, you’ve just reverted any submodule commits the branch you merged in might have made"... Yes, if you forget to type the correct commands then undesired effects will happen. Now that's a git design flaw? Give me a break.

brazzy 15y ago
So you'd also say that a car that explodes if you forget to press the "don't explode" button after refuelling is a perfectly good design? Even if you think you're perfect and never make mistakes, it's not a good idea to add easy opportunities to make mistakes.
- exDM69 15y ago
  Unlike an exploding car, Git has undo. See git reflog.

moe 15y ago

Very good article. The question is, will it ever be fixed? As it stands submodules don't even work sanely for the most trivial use-case of tracking a (slow changing) vendor-repo. Whoever designed this (Linus?) had a real brainfart here.

harshreality 15y ago
I wonder. How difficult can it be to mimic mercurial's subrepos? http://blog.codekills.net/2011/07/14/nested-repository-handling-in-git-and-mercurial/ http://blog.codekills.net/2011/07/14/nested-repository-handl...
- moe 15y ago
  Yes! That should be built straight into git. The abstraction leaks are no worse than with submodules (it's just making different trade-offs) but the usability is so much more sane, it's not funny. The importance of having this in core-git can not be overstated. When it's not in core then it's not getting used. Case in point: count the number of projects on github using one of the external training wheels (subtree etc.). Submodules has proven inadequate for reality (again: count the number of subprojects on github). I wish one of the core-dev's would make a kickstarter for a solution...
exDM69 15y ago
> As it stands submodules don't even work sanely for the most trivial use-case of tracking a (slow changing) vendor-repo. That is not the use case it was designed for. And submodules still work just fine for that. The fact that you have to do a separate submodule update does not make it broken. In fact, it makes it better when you start thinking about all the corner cases there are. If you truly want to build a software out of subrepositories, you need to have a sane way of tracking working sets of revisions of all the submodules there are. And that is what git submodules does. If you were blindly tracking the head of a repository, it would be useful _only_ for the use case you mentioned. Tracking slowly changing external vendor repos. That would be a whole lot less useful.
- 15y ago

jacobr 15y ago

Submodules are pretty annoying if you modify the submodule repo a lot, but a pre-commit hook would help against the "forgot to push in submodule" issue, no?

exDM69 15y ago
Or a simple git or shell alias which would do the submodule update when you want it. That would solve most of the sources of complaints in the article and in this thread. However, most likely you don't want an automatic submodule update because of all the issues there are. It would only be useful for tracking a very stable slow moving external dependency.

bryanlarsen 15y ago

It appears that one of the solutions he recommends, git-subtree, is going to be merged into git soon: http://git.661346.n2.nabble.com/git-subtree-Next-Round-Ready-td7404309.html http://git.661346.n2.nabble.com/git-subtree-Next-Round-Ready... This is excellent news. I've been using git subtree for a couple of years now without incident, and highly recommend it.

aiiane 15y ago
she.
harshreality 15y ago
Recent discussions about submodules and the subtree script on the mailing list: http://thread.gmane.org/gmane.comp.version-control.git/196484 http://thread.gmane.org/gmane.comp.version-control.git/19648... http://thread.gmane.org/gmane.comp.version-control.git/195604 http://thread.gmane.org/gmane.comp.version-control.git/19560...
bryanlarsen 14y ago
WOOT: https://github.com/apenwarr/git-subtree/commit/8cd698957f57f62ec3d313403bebced2fdf751e5 https://github.com/apenwarr/git-subtree/commit/8cd698957f57f...