9 ms·

Git submodules are fine and can be really useful, but they are really hard. I've run into problems like: 1. Git clone not cloning submodules. You need `git sub

Git submodules are fine and can be really useful, but they are really hard. I've run into problems like: 1. Git clone not cloning submodules. You need `git submodule update` or `git clone --recursive`, I think 2. Git submodules being out-of-sync because I forgot to pull them specifically. I'm pretty sure `git submodule update` doesn't always work with this but maybe only when 3) 3. Git diff returns something even after I commit, because the submodule has a change. I have to go into the submodule, and either commit / push that as well or revert it. Basically every operation I do on the git main I need to also do on the submodule if I modified files in both 4. Fixing merge conflicts and using git in one repo is already hard enough. The team I was working on kept having issues with using the wrong submodule commit, not having the same commit / push requirements on submodules, etc. All of these can be fixed by tools and smart techniques like putting `git submodule update` in the makefile. Git submodules aren't "bad" and honestly they're an essential feature of git. But they are a struggle, and lots of people use monorepos instead (which have their own problems...).

lelandfe 4y ago
Agreed with all of this! In git parlance, the submodule porcelain is hard to use (but the plumbing is good)
- 42e6e8c8-f7b8-4 4y ago
  That's the entirety of git. Extremely fast. Bummer UI. It's status quo and no changing. Despite using git for 10+ years, I frequently have to look up commands and then I end up scratching my head as to why the CLI UI is like that.
  - oneplane 4y ago
    I suppose the biggest problem is that the concept of SCM/VCS is just not simple enough to make both easy and useful/advanced at the same time. You can have a 'pull, merge, push'-only system, but at that point we're re-inventing subversion. So making it more advanced would mean we also need to have the knowledge and skills to do other activities correctly and that means the tooling can't make as many choices for you because there simply isn't a default way that works all the time. Most efforts at git-alternatives run in to the same problems and either they'll be just as advanced and have the samen benefits and downsides or they end up less advanced but now it's not equally useful and you can't really make it work right.
    - pan69 4y ago
      > but at that point we're re-inventing subversion So? Maybe Subversion is all most developers need?
      - dalyons 4y ago
        Solo devs maybe. Merging and branching are so much worse in svn that it’s not good enough for “most developers”, aka those working on professional projects with a team of developers. Sure we used to make do with svn, but I have no desire to go back.
        mnd999 4y ago
        Honestly I’d take CVS over subversion. Merging was just bad on subversion and I found the IDE integrations to be confusing, obtuse and buggy. Sure, CVS was limited but it was reliable and straightforward.
        DonHopkins 4y ago
        ...And better than SCCS! ;)
        johannes1234321 4y ago
        > Sure, CVS was limited but it was reliable and straightforward. It was so reliable that people complained about SVN using a DB (Berkeley DB) as backend, as manual fixing of CVS files was a "normal" part of operation and people didn't believe that might not be needed ...
        avgcorrection 4y ago
        > Solo devs maybe. Like I’m gonna bother to set up a Subversion remote for every little repo that I create for myself.
        theamk 4y ago
        The most beautiful part of svn, and the one I miss the most in git, is the fact that there is no need to set up tons of separate repo. Every subdirectory acts as a separate git repo. This means you usually only have one svn repo, and you set it up the way you like. As an example, you may set things up so you can checkout: server:/proj/small/hello - to get a single project server:/proj/small- to get all small projects server:/proj - to get all projects If you already have one of those checked out, adding new project is as simple as "mkdir bar", "svn add bar", "svn commit". So juch easier than making new github repo. And multi-level hierarchical project nesting is still something impossible in git.
  - ajkjk 4y ago
    It's slowly changing. They finally added `git restore`, for example.
  - ximm 4y ago
    > It's status quo and no changing Not entirely true: checkout was split into switch and restore, which is something I guess.
fburnaby 4y ago
I agree with all of this. Submodules aren't easy but they perform a useful job. It's hard to see how they could be made significantly easier. Where else in software is dependency management easy and convenient?
- pm215 4y ago
  "Make 'git checkout' of the top level repo also set the submodules to the contents they should have for that top level commit hash" is probably the main change I'd want. The current setup means that checking out a branch or doing a git bisect gives you an inconsistent source tree, which seems like a really unhelpful behaviour.
andi999 4y ago
What tools fix this?
- abathur 4y ago
  Using something like Nix to specify the dependencies instead.
- oreally 4y ago
  If you're on windows, it just takes a few clicks with tortoisegit. I never have to remember these command line git commands.
- alephnan 4y ago
  The irony of needing a tool to make the first tool work well
dtgriscom 4y ago
`git submodule update --init --recursive` is the magic phrase. And, yes: submodules are really useful, as well as a PIA.
- acadapter 4y ago
  And you can also --recurse-submodules when cloning
agilob 4y ago
There are git options for managing these difficulties like: git config --global submodule.recurse true https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules search for "git config"
- elcritch 4y ago
  Thanks! That one option simplifies like 2/3 of the pain of using submodules. Doing a pull or checkout would just be regular commands. Though it’d be nice if `git commit’ supported it too and just did a `git submodule foreach git commit …`.
  - jhugo 4y ago
    Interesting! I've never thought of doing commits in submodules. We use them a lot at work but only in a "import a specific revision of this other repo into our repo" sense, if we needed to make changes we would do them in the repo that the submodule points to and then just update the ref that the submodule points to. What's your use case for making and committing changes inside the submodule tree?
    - s__s 4y ago
      I naively thought that was the whole point of sub modules. Otherwise why not use a package manager? The use case being that you can work on and update some independent system/repo while getting real-time feedback on how the changes interact with the system as a whole.
      - elcritch 4y ago
        That's how I've come to use git submodules. It's been helpful when working with various embedded projects. They often don't get updated for months or years (ideally), and then you only want to pull in specific changes.
    - chrsig 4y ago
      It's easy to overlook that the submodule is a complete git repository in its own right. When working on both, it's really annoying to have to commit to a dependency's repo, push, and pull down in the dependent repo. If you do the commit in the submodule, it's just immediately available...and you can still have whatever remote to push to. So it just cuts down on the number of checkouts you have to have.
Phurist 4y ago
That sounds like a problem that exists between the chair and the keyboard.
- onion2k 4y ago
  In 25 years of software dev, and most of that in developing user interfaces to things, I have never found a commonly repeated error that could be attributed to user error. It's always badly designed software that ignores the user's mental model that's developed based on using the software. In this case, "git clone" clones a repo 99% of the time, because 99% of repos are shallow and simple. For "git clone" to only clone the top level when you have su modules instead of prompting to ask if you want a deep clone, or doing a deep clone by default because that's the expected behavior, is pretty poor design IMO.
fanf2 4y ago
Switching branches in a repository with submodules is a huge pain, especially if (like the Ansible repo) some branches have the subdirectory in the same repo like normal, and some branches have the same subdirectory in a submodule.
esperent 4y ago
> Git submodules are fine and can be really useful, but they are really hard If an important software tool is hard to use to the point that most people avoid it, then it's not fine. It's broken.

lelandfe 4y ago

Agreed with all of this! In git parlance, the submodule porcelain is hard to use (but the plumbing is good)

42e6e8c8-f7b8-4 4y ago
That's the entirety of git. Extremely fast. Bummer UI. It's status quo and no changing. Despite using git for 10+ years, I frequently have to look up commands and then I end up scratching my head as to why the CLI UI is like that.
- oneplane 4y ago
  I suppose the biggest problem is that the concept of SCM/VCS is just not simple enough to make both easy and useful/advanced at the same time. You can have a 'pull, merge, push'-only system, but at that point we're re-inventing subversion. So making it more advanced would mean we also need to have the knowledge and skills to do other activities correctly and that means the tooling can't make as many choices for you because there simply isn't a default way that works all the time. Most efforts at git-alternatives run in to the same problems and either they'll be just as advanced and have the samen benefits and downsides or they end up less advanced but now it's not equally useful and you can't really make it work right.
  - pan69 4y ago
    > but at that point we're re-inventing subversion So? Maybe Subversion is all most developers need?
    - dalyons 4y ago
      Solo devs maybe. Merging and branching are so much worse in svn that it’s not good enough for “most developers”, aka those working on professional projects with a team of developers. Sure we used to make do with svn, but I have no desire to go back.
      - mnd999 4y ago
        Honestly I’d take CVS over subversion. Merging was just bad on subversion and I found the IDE integrations to be confusing, obtuse and buggy. Sure, CVS was limited but it was reliable and straightforward.
        DonHopkins 4y ago
        ...And better than SCCS! ;)
        johannes1234321 4y ago
        > Sure, CVS was limited but it was reliable and straightforward. It was so reliable that people complained about SVN using a DB (Berkeley DB) as backend, as manual fixing of CVS files was a "normal" part of operation and people didn't believe that might not be needed ...
      - avgcorrection 4y ago
        > Solo devs maybe. Like I’m gonna bother to set up a Subversion remote for every little repo that I create for myself.
        theamk 4y ago
        The most beautiful part of svn, and the one I miss the most in git, is the fact that there is no need to set up tons of separate repo. Every subdirectory acts as a separate git repo. This means you usually only have one svn repo, and you set it up the way you like. As an example, you may set things up so you can checkout: server:/proj/small/hello - to get a single project server:/proj/small- to get all small projects server:/proj - to get all projects If you already have one of those checked out, adding new project is as simple as "mkdir bar", "svn add bar", "svn commit". So juch easier than making new github repo. And multi-level hierarchical project nesting is still something impossible in git.
- ajkjk 4y ago
  It's slowly changing. They finally added `git restore`, for example.
- ximm 4y ago
  > It's status quo and no changing Not entirely true: checkout was split into switch and restore, which is something I guess.

fburnaby 4y ago

I agree with all of this. Submodules aren't easy but they perform a useful job. It's hard to see how they could be made significantly easier. Where else in software is dependency management easy and convenient?

pm215 4y ago
"Make 'git checkout' of the top level repo also set the submodules to the contents they should have for that top level commit hash" is probably the main change I'd want. The current setup means that checking out a branch or doing a git bisect gives you an inconsistent source tree, which seems like a really unhelpful behaviour.

andi999 4y ago

What tools fix this?

abathur 4y ago
Using something like Nix to specify the dependencies instead.
oreally 4y ago
If you're on windows, it just takes a few clicks with tortoisegit. I never have to remember these command line git commands.
alephnan 4y ago
The irony of needing a tool to make the first tool work well

dtgriscom 4y ago

`git submodule update --init --recursive` is the magic phrase. And, yes: submodules are really useful, as well as a PIA.

acadapter 4y ago
And you can also --recurse-submodules when cloning

agilob 4y ago

There are git options for managing these difficulties like: git config --global submodule.recurse true https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules search for "git config"

elcritch 4y ago
Thanks! That one option simplifies like 2/3 of the pain of using submodules. Doing a pull or checkout would just be regular commands. Though it’d be nice if `git commit’ supported it too and just did a `git submodule foreach git commit …`.
- jhugo 4y ago
  Interesting! I've never thought of doing commits in submodules. We use them a lot at work but only in a "import a specific revision of this other repo into our repo" sense, if we needed to make changes we would do them in the repo that the submodule points to and then just update the ref that the submodule points to. What's your use case for making and committing changes inside the submodule tree?
  - s__s 4y ago
    I naively thought that was the whole point of sub modules. Otherwise why not use a package manager? The use case being that you can work on and update some independent system/repo while getting real-time feedback on how the changes interact with the system as a whole.
    - elcritch 4y ago
      That's how I've come to use git submodules. It's been helpful when working with various embedded projects. They often don't get updated for months or years (ideally), and then you only want to pull in specific changes.
  - chrsig 4y ago
    It's easy to overlook that the submodule is a complete git repository in its own right. When working on both, it's really annoying to have to commit to a dependency's repo, push, and pull down in the dependent repo. If you do the commit in the submodule, it's just immediately available...and you can still have whatever remote to push to. So it just cuts down on the number of checkouts you have to have.

Phurist 4y ago

That sounds like a problem that exists between the chair and the keyboard.

onion2k 4y ago
In 25 years of software dev, and most of that in developing user interfaces to things, I have never found a commonly repeated error that could be attributed to user error. It's always badly designed software that ignores the user's mental model that's developed based on using the software. In this case, "git clone" clones a repo 99% of the time, because 99% of repos are shallow and simple. For "git clone" to only clone the top level when you have su modules instead of prompting to ask if you want a deep clone, or doing a deep clone by default because that's the expected behavior, is pretty poor design IMO.

fanf2 4y ago

Switching branches in a repository with submodules is a huge pain, especially if (like the Ansible repo) some branches have the subdirectory in the same repo like normal, and some branches have the same subdirectory in a submodule.

esperent 4y ago

> Git submodules are fine and can be really useful, but they are really hard If an important software tool is hard to use to the point that most people avoid it, then it's not fine. It's broken.