8 ms·

Here's the lesson I've learned from my experience with submodules in git in several different companies: avoid them like the plague. NEVER USE THEM FOR ANYTHING

by OskarS 6y ago

Here's the lesson I've learned from my experience with submodules in git in several different companies: avoid them like the plague. NEVER USE THEM FOR ANYTHING. Use any other solution available to you: use package managers, use monorepos, use subtrees, just copy/paste the files in, avoid the dependency entirely, do ANYTHING it takes to avoid using submodules. They just become a constant source of friction. Basically every action you do in git, there's some tiny bit of annoyance caused by submodules, which adds up to tearing your hair out. Like, read this StackOverflow question and answer, and tell me this is something you want to be dealing with on daily basis (and you will be, you'll regularly be dealing with far worse): https://stackoverflow.com/questions/9314365/git-clean-is-not-removing-a-submodule-added-to-a-branch-when-switching-branches https://stackoverflow.com/questions/9314365/git-clean-is-not... The correct way to handle dependencies in general is package managers, but if for some reason that is not available to you, and you wish to avoid git subtree for some reason, copy/pasting the code from the other repository and making a note in the commit message which commit in the dependency repo you copied from is a far preferable solution. Yeah, you lose the history of the sub-repo, but it's well worth it to avoid the complications from using submodule (and you never examine the history of the submodule anyway, how often do you look at the history of your dependencies?). Submodules: not even once.

m_eiman 6y ago
Don't use subtrees, they're even worse. At least when we tried using them for a while before moving to submodules; perhaps they're less of a broken hack these days. For example, whenever we did something with the subtree it reprocessed all commits in the repo - obviously taking longer and longer as time went on.
junon 6y ago
Disagree. Package managers cause more headache than anything. I have used submodules for almost all of my projects (aside from e.g. Node.js projects). Not once have had issues with them. I don't know why people raise such a stink about them.
- dataflow 6y ago
  > I have used submodules for almost all of my projects (aside from e.g. Node.js projects). Not once have had issues with them. Do you do things with git beside the trivial operations (add/commit/etc.)? Like rebasing/merging and such across commits that introduce or remove submodules? Do you never encounter friction when e.g. removing submodules? Do you never run into the errors people complain about?
  - P_I_Staker 6y ago
    I absolutely despise submodules, but under many use cases you don't add or remove that often. We definitely run into errors people complain about. In fact there are errors that the average user (that doesn't really care about SVC) pretty much can't fix... or with much difficulty. I don't know that I'd still say it's a dealbreaker, but that's definitely very bad. Very much does break some of the spirit of git and what makes it "nice". We've been using them, and our system is still very usable (obviously this is a ridiculous thing to brag about, but worth pointing out). I'm wondering if there's a better way to handle shared code between two repos, where the users of either don't have access to the other parent? You might be able to use subtrees? Email source code and just create n separate commits for n projects, with a standardized commit message? I'd be happy to do something different, but I never had some idea that I could sell in a meeting, and didn't really have the energy or clout to fight this.
    - sibrahim 6y ago
      My preferred solution has been git-subrepo: https://github.com/ingydotnet/git-subrepo https://github.com/ingydotnet/git-subrepo Basically is what (I think) submodules should have been. Creates a vendored copy with metadata about what commit it came from. Normal operations like clone, commit (even touching multiple main/subrepo files which subtree struggles with) are unaffected (normal files from git's point of view). Pull, push, branching all work as expected on main project (most devs don't even need to know there's a subrepo). If you want to pull/push vendored changes from subrepo, there's a new command, but that's it.
petepete 6y ago
This is my experience too, I wouldn't recommend people use them unless they really understand the costs. The last time I used them was to help manage my dotfiles, so I could use them with Pathogen to pull in my selection of vim plugins. It was just a headache, to the point where I bit the bullet and went down the package manager (vim-plug) route instead.
- Anthony-G 6y ago
  This is also the only use-case I have for git sub-modules. However, I haven’t encountered any serious problems. I mostly leave my (few) Vim plugins alone for so long that I forget the git commands to update the plugins when I do decide to update them. Having to check my notes or the relevant section of the Git Book is the worst of the friction. I’m using Pathogen as it works fine for me. I believe Vim 8.0 introduced something similar but I still have to use older Vim releases on various CentOS 6 and 7 systems.
atoav 6y ago
I agree. In principle the promise of submodules is a very good, logical and clear one – it is totally understandable why anybody would like to use them, because it would just make sense in a lot of cases. However the way they are implemented is just so horrible, I side with you here: Don't use them for anything serious. It is not worth it. Invest the time into writing a bash script that does the copying for you, use private package managers for shared dependecies or whatever, but don't use submodules – this way you will at least know what bites you when it does.
- OskarS 6y ago
  Exactly: they're so appealing the first time you find out about the feature. "Oh, i can just go `git submodule add <repo> <path>` and it includes another repo! Very cool!", and then two months later you're staring at the 427th time submodules have caused weird issues when changing branches or whatever, and you just want to hurl your computer into the sea. A hacked-together bash script like you suggested is a far better solution.
dataflow 6y ago
Yeah, I agree. At some point in time I came up with a clear rule on when exactly submodules are the best solution, and it was such a narrow set of circumstances that I forget what it was at this point. I think it might have been in situations similar to Boost, where the project is actually modular? I forget. But even in those cases the friction is painful. (For the life of me I don't understand why they don't try to polish all the rough edges. Anyone know?) One alternative solution I like: have a build tool that pulls dependencies in automatically. Bazel supports this natively, but I've also done this in plain ol' Make. Like you can build download rules for deps/github.com/% to automatically pull from GitHub, and the rest of the script can just treat them like local files. Having the build tool take care of it often seems to fit the problem space better too.
- coryrc 6y ago
  Your later half is essentially what subtree does, but it can also merge changes if you have any.
  - dataflow 6y ago
    Wait really? I thought subtree stores a copy of the other repo?
    - coryrc 6y ago
      Maybe I misunderstood and you aren't checking in those files, in which case it isn't the same.
yakubin 6y ago
> and you never examine the history of the submodule anyway, how often do you look at the history of your dependencies? All the time. Software is made of bugs, each of your dependencies has bugs, it's on you to diagnose them and, quite often, fix them. Git history is invaluable in those scenarios. Generally, I'm in favor of pasting the dependencies in your repo. It protects you against things like the leftpad incident and also keeps things nicely in one place, you don't have to go to multiple places to fetch the full source of your program. It also ensures that you have an accurate view of how complex your creation actually is. But history of dependencies is also useful. At work, we have one giant, heavily patched third-party dependency and a lot of small ones. The big one lives in a submodule. The small ones are pasted in the repo. There is no end to the bugs in them, history is then very useful. And before you say anything, they aren't some random npm packages not vetted by anyone. They're widely-regarded foundational open source projects. Package managers on the other hand discourage e.g. modifying the dependencies to debug problems in the application. There are more and less correct ways to do that, but generally it's a hoop to jump. With submodules or vendoring - no problem. They also create an illusion of your project being simpler than it really is and are prone to leftpad. And downloading things at build-time is an anti-pattern. My perspective on software is this: all software has bugs, now the question is: how do we deal with it? How will we debug it? Sadly, there is a lot of "let's assume this works perfectly all the time and do something else" mindset going around as well. Like... How do you debug a malfunctioning GC in your managed language of choice? Will you be able to debug the dependency you've just pulled into your project? Is this language feature a compile-time nicety that will make the interactive debugging later miserable?
- pvorb 6y ago
  If you control the dependency and if you're the primary user, just move it into your repository by merging with unrelated histories. If you're not controlling the dependency, you can still merge the dependency with an unrelated history on every version upgrade. This will still likely consume less time than fighting submodules every day.
temac 6y ago
I agree with that. Both the ergonomics and the quality of implementation of git submodules are currently atrocious. Special mention if you try to mix them with worktrees. I think there is a small possibility that they will become usable in 10 years, and an even smaller in 5. But for now: don't use git submodule. I did not believed that once, because they kinda look nice on paper. Don't do the same mistake. At least wait 5 years before trying them. And test them appart during at least 6 months, actively each day, before taking a dependency and actually making your project use them.
- pvorb 6y ago
  What makes you think that submodules will improve? I've used them for the last ten years or so (when I was forced to) and haven't seen any advances in UX.
globular-toast 6y ago
Submodules have been working very well for my emacs config for years now. However, I've used them for software projects and they are indeed awful. I joined a company where the lead engineer had set up a build system that was to be included as a submodule in every repository. It was a nightmare.
user-the-name 6y ago
I don't understand why submodules are so bad. They are a feature where maybe 60% is implemented, and then just abandoned. Mercurial has subrepos, which are the EXACT same feature, but they are actually fully implemented, they work, and they don't make you shoot yourself in the foot constantly. They are mostly a joy to use.
coding123 6y ago
SAME exact experience here. If you can create either a build workflow or some other method of managing a relationship like this, do that instead of submodules. I've been on teams that used it quite excessively and it was a PITA. I've used it in much smaller ways to try to avoid a build step or two and it worked until it was responsible for the world falling and no one knew it was that damn sub module. Just because the tech exists, doesn't mean it's a good idea.
jayd16 6y ago
Submodules used to be terrible to use but now that every tool supports it they are usable. A lot of dependency managers are moving to supporting git repos which I think is great. An annoying thing about this though it is causes several round trips to origin that can be saved with submodules. It's never easy... Its better to treat submodules as src dependencies and not wannabe mono-repo tool. Try not to change them. If you read-only they're somewhat painless.
gmadsen 6y ago
This is beyond dramatic. To actually suggest copy/paste is preferable is asinine. Once you get used to them, they are fine. I never have issues with submodules and I get to easily change revisions of dependencies.
- deleted 6y ago
  [deleted]
pedrolamarao 6y ago
In my experience, such things tend to be misused when dependency management is not available. There are many tools today which sell themselves as "dependency managers" which are in fact just glorified downloaders. Such things should be called "download managers" instead.
60secz 6y ago
Agree 100%. Submodules should be instead be implemented through auto-published library semver jars which are auto-incremented.
whatever1 6y ago
I have same experience with you, when I tried to deal with a c++ library that was co-developed with the main project. In lieu of an ubiquitous package manager / module support what other options are there for c++? Keep a text file in the main project with the version of the library you need to compile it ?

by OskarS 6y ago

m_eiman 6y ago

Don't use subtrees, they're even worse. At least when we tried using them for a while before moving to submodules; perhaps they're less of a broken hack these days. For example, whenever we did something with the subtree it reprocessed all commits in the repo - obviously taking longer and longer as time went on.

junon 6y ago

Disagree. Package managers cause more headache than anything. I have used submodules for almost all of my projects (aside from e.g. Node.js projects). Not once have had issues with them. I don't know why people raise such a stink about them.

dataflow 6y ago
> I have used submodules for almost all of my projects (aside from e.g. Node.js projects). Not once have had issues with them. Do you do things with git beside the trivial operations (add/commit/etc.)? Like rebasing/merging and such across commits that introduce or remove submodules? Do you never encounter friction when e.g. removing submodules? Do you never run into the errors people complain about?
- P_I_Staker 6y ago
  I absolutely despise submodules, but under many use cases you don't add or remove that often. We definitely run into errors people complain about. In fact there are errors that the average user (that doesn't really care about SVC) pretty much can't fix... or with much difficulty. I don't know that I'd still say it's a dealbreaker, but that's definitely very bad. Very much does break some of the spirit of git and what makes it "nice". We've been using them, and our system is still very usable (obviously this is a ridiculous thing to brag about, but worth pointing out). I'm wondering if there's a better way to handle shared code between two repos, where the users of either don't have access to the other parent? You might be able to use subtrees? Email source code and just create n separate commits for n projects, with a standardized commit message? I'd be happy to do something different, but I never had some idea that I could sell in a meeting, and didn't really have the energy or clout to fight this.
  - sibrahim 6y ago
    My preferred solution has been git-subrepo: https://github.com/ingydotnet/git-subrepo https://github.com/ingydotnet/git-subrepo Basically is what (I think) submodules should have been. Creates a vendored copy with metadata about what commit it came from. Normal operations like clone, commit (even touching multiple main/subrepo files which subtree struggles with) are unaffected (normal files from git's point of view). Pull, push, branching all work as expected on main project (most devs don't even need to know there's a subrepo). If you want to pull/push vendored changes from subrepo, there's a new command, but that's it.

petepete 6y ago

This is my experience too, I wouldn't recommend people use them unless they really understand the costs. The last time I used them was to help manage my dotfiles, so I could use them with Pathogen to pull in my selection of vim plugins. It was just a headache, to the point where I bit the bullet and went down the package manager (vim-plug) route instead.

Anthony-G 6y ago
This is also the only use-case I have for git sub-modules. However, I haven’t encountered any serious problems. I mostly leave my (few) Vim plugins alone for so long that I forget the git commands to update the plugins when I do decide to update them. Having to check my notes or the relevant section of the Git Book is the worst of the friction. I’m using Pathogen as it works fine for me. I believe Vim 8.0 introduced something similar but I still have to use older Vim releases on various CentOS 6 and 7 systems.

atoav 6y ago

I agree. In principle the promise of submodules is a very good, logical and clear one – it is totally understandable why anybody would like to use them, because it would just make sense in a lot of cases. However the way they are implemented is just so horrible, I side with you here: Don't use them for anything serious. It is not worth it. Invest the time into writing a bash script that does the copying for you, use private package managers for shared dependecies or whatever, but don't use submodules – this way you will at least know what bites you when it does.

OskarS 6y ago
Exactly: they're so appealing the first time you find out about the feature. "Oh, i can just go `git submodule add <repo> <path>` and it includes another repo! Very cool!", and then two months later you're staring at the 427th time submodules have caused weird issues when changing branches or whatever, and you just want to hurl your computer into the sea. A hacked-together bash script like you suggested is a far better solution.

dataflow 6y ago

Yeah, I agree. At some point in time I came up with a clear rule on when exactly submodules are the best solution, and it was such a narrow set of circumstances that I forget what it was at this point. I think it might have been in situations similar to Boost, where the project is actually modular? I forget. But even in those cases the friction is painful. (For the life of me I don't understand why they don't try to polish all the rough edges. Anyone know?) One alternative solution I like: have a build tool that pulls dependencies in automatically. Bazel supports this natively, but I've also done this in plain ol' Make. Like you can build download rules for deps/github.com/% to automatically pull from GitHub, and the rest of the script can just treat them like local files. Having the build tool take care of it often seems to fit the problem space better too.

coryrc 6y ago
Your later half is essentially what subtree does, but it can also merge changes if you have any.
- dataflow 6y ago
  Wait really? I thought subtree stores a copy of the other repo?
  - coryrc 6y ago
    Maybe I misunderstood and you aren't checking in those files, in which case it isn't the same.

yakubin 6y ago

> and you never examine the history of the submodule anyway, how often do you look at the history of your dependencies? All the time. Software is made of bugs, each of your dependencies has bugs, it's on you to diagnose them and, quite often, fix them. Git history is invaluable in those scenarios. Generally, I'm in favor of pasting the dependencies in your repo. It protects you against things like the leftpad incident and also keeps things nicely in one place, you don't have to go to multiple places to fetch the full source of your program. It also ensures that you have an accurate view of how complex your creation actually is. But history of dependencies is also useful. At work, we have one giant, heavily patched third-party dependency and a lot of small ones. The big one lives in a submodule. The small ones are pasted in the repo. There is no end to the bugs in them, history is then very useful. And before you say anything, they aren't some random npm packages not vetted by anyone. They're widely-regarded foundational open source projects. Package managers on the other hand discourage e.g. modifying the dependencies to debug problems in the application. There are more and less correct ways to do that, but generally it's a hoop to jump. With submodules or vendoring - no problem. They also create an illusion of your project being simpler than it really is and are prone to leftpad. And downloading things at build-time is an anti-pattern. My perspective on software is this: all software has bugs, now the question is: how do we deal with it? How will we debug it? Sadly, there is a lot of "let's assume this works perfectly all the time and do something else" mindset going around as well. Like... How do you debug a malfunctioning GC in your managed language of choice? Will you be able to debug the dependency you've just pulled into your project? Is this language feature a compile-time nicety that will make the interactive debugging later miserable?

pvorb 6y ago
If you control the dependency and if you're the primary user, just move it into your repository by merging with unrelated histories. If you're not controlling the dependency, you can still merge the dependency with an unrelated history on every version upgrade. This will still likely consume less time than fighting submodules every day.

temac 6y ago

I agree with that. Both the ergonomics and the quality of implementation of git submodules are currently atrocious. Special mention if you try to mix them with worktrees. I think there is a small possibility that they will become usable in 10 years, and an even smaller in 5. But for now: don't use git submodule. I did not believed that once, because they kinda look nice on paper. Don't do the same mistake. At least wait 5 years before trying them. And test them appart during at least 6 months, actively each day, before taking a dependency and actually making your project use them.

pvorb 6y ago
What makes you think that submodules will improve? I've used them for the last ten years or so (when I was forced to) and haven't seen any advances in UX.

globular-toast 6y ago

Submodules have been working very well for my emacs config for years now. However, I've used them for software projects and they are indeed awful. I joined a company where the lead engineer had set up a build system that was to be included as a submodule in every repository. It was a nightmare.

user-the-name 6y ago

I don't understand why submodules are so bad. They are a feature where maybe 60% is implemented, and then just abandoned. Mercurial has subrepos, which are the EXACT same feature, but they are actually fully implemented, they work, and they don't make you shoot yourself in the foot constantly. They are mostly a joy to use.

coding123 6y ago

SAME exact experience here. If you can create either a build workflow or some other method of managing a relationship like this, do that instead of submodules. I've been on teams that used it quite excessively and it was a PITA. I've used it in much smaller ways to try to avoid a build step or two and it worked until it was responsible for the world falling and no one knew it was that damn sub module. Just because the tech exists, doesn't mean it's a good idea.

jayd16 6y ago

Submodules used to be terrible to use but now that every tool supports it they are usable. A lot of dependency managers are moving to supporting git repos which I think is great. An annoying thing about this though it is causes several round trips to origin that can be saved with submodules. It's never easy... Its better to treat submodules as src dependencies and not wannabe mono-repo tool. Try not to change them. If you read-only they're somewhat painless.

gmadsen 6y ago

This is beyond dramatic. To actually suggest copy/paste is preferable is asinine. Once you get used to them, they are fine. I never have issues with submodules and I get to easily change revisions of dependencies.

deleted 6y ago
[deleted]

pedrolamarao 6y ago

In my experience, such things tend to be misused when dependency management is not available. There are many tools today which sell themselves as "dependency managers" which are in fact just glorified downloaders. Such things should be called "download managers" instead.

60secz 6y ago

Agree 100%. Submodules should be instead be implemented through auto-published library semver jars which are auto-incremented.

whatever1 6y ago

I have same experience with you, when I tried to deal with a c++ library that was co-developed with the main project. In lieu of an ubiquitous package manager / module support what other options are there for c++? Keep a text file in the main project with the version of the library you need to compile it ?