14 ms·

Ask HN: Why are Git submodules so bad?

213 points by gavinhoward 4y ago

I have been a git user for a long time, but I've never used Subversion or any other VCS more than a little. I also hardly use Git submodules, but when I do, I don't struggle. Yet people talk about Git submodules as though they are really hard. I presume I'm just not using them as much as other people, or that my use case for them happens to be on their happy path. So why are Git submodules so bad?

armchairhacker 4y ago
Git submodules are fine and can be really useful, but they are really hard. I've run into problems like: 1. Git clone not cloning submodules. You need `git submodule update` or `git clone --recursive`, I think 2. Git submodules being out-of-sync because I forgot to pull them specifically. I'm pretty sure `git submodule update` doesn't always work with this but maybe only when 3) 3. Git diff returns something even after I commit, because the submodule has a change. I have to go into the submodule, and either commit / push that as well or revert it. Basically every operation I do on the git main I need to also do on the submodule if I modified files in both 4. Fixing merge conflicts and using git in one repo is already hard enough. The team I was working on kept having issues with using the wrong submodule commit, not having the same commit / push requirements on submodules, etc. All of these can be fixed by tools and smart techniques like putting `git submodule update` in the makefile. Git submodules aren't "bad" and honestly they're an essential feature of git. But they are a struggle, and lots of people use monorepos instead (which have their own problems...).
- lelandfe 4y ago
  Agreed with all of this! In git parlance, the submodule porcelain is hard to use (but the plumbing is good)
  - 42e6e8c8-f7b8-4 4y ago
    That's the entirety of git. Extremely fast. Bummer UI. It's status quo and no changing. Despite using git for 10+ years, I frequently have to look up commands and then I end up scratching my head as to why the CLI UI is like that.
    - oneplane 4y ago
      I suppose the biggest problem is that the concept of SCM/VCS is just not simple enough to make both easy and useful/advanced at the same time. You can have a 'pull, merge, push'-only system, but at that point we're re-inventing subversion. So making it more advanced would mean we also need to have the knowledge and skills to do other activities correctly and that means the tooling can't make as many choices for you because there simply isn't a default way that works all the time. Most efforts at git-alternatives run in to the same problems and either they'll be just as advanced and have the samen benefits and downsides or they end up less advanced but now it's not equally useful and you can't really make it work right.
      - pan69 4y ago
        > but at that point we're re-inventing subversion So? Maybe Subversion is all most developers need?
        dalyons 4y ago
        Solo devs maybe. Merging and branching are so much worse in svn that it’s not good enough for “most developers”, aka those working on professional projects with a team of developers. Sure we used to make do with svn, but I have no desire to go back.
        mnd999 4y ago
        Honestly I’d take CVS over subversion. Merging was just bad on subversion and I found the IDE integrations to be confusing, obtuse and buggy. Sure, CVS was limited but it was reliable and straightforward.
        DonHopkins 4y ago
        ...And better than SCCS! ;)
        johannes1234321 4y ago
- fburnaby 4y ago
  I agree with all of this. Submodules aren't easy but they perform a useful job. It's hard to see how they could be made significantly easier. Where else in software is dependency management easy and convenient?
  - pm215 4y ago
    "Make 'git checkout' of the top level repo also set the submodules to the contents they should have for that top level commit hash" is probably the main change I'd want. The current setup means that checking out a branch or doing a git bisect gives you an inconsistent source tree, which seems like a really unhelpful behaviour.
- andi999 4y ago
  What tools fix this?
  - abathur 4y ago
    Using something like Nix to specify the dependencies instead.
  - oreally 4y ago
    If you're on windows, it just takes a few clicks with tortoisegit. I never have to remember these command line git commands.
  - alephnan 4y ago
    The irony of needing a tool to make the first tool work well
- dtgriscom 4y ago
  `git submodule update --init --recursive` is the magic phrase. And, yes: submodules are really useful, as well as a PIA.
  - acadapter 4y ago
    And you can also --recurse-submodules when cloning
- agilob 4y ago
  There are git options for managing these difficulties like: git config --global submodule.recurse true https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules search for "git config"
  - elcritch 4y ago
    Thanks! That one option simplifies like 2/3 of the pain of using submodules. Doing a pull or checkout would just be regular commands. Though it’d be nice if `git commit’ supported it too and just did a `git submodule foreach git commit …`.
    - jhugo 4y ago
      Interesting! I've never thought of doing commits in submodules. We use them a lot at work but only in a "import a specific revision of this other repo into our repo" sense, if we needed to make changes we would do them in the repo that the submodule points to and then just update the ref that the submodule points to. What's your use case for making and committing changes inside the submodule tree?
      - s__s 4y ago
        I naively thought that was the whole point of sub modules. Otherwise why not use a package manager? The use case being that you can work on and update some independent system/repo while getting real-time feedback on how the changes interact with the system as a whole.
        elcritch 4y ago
        That's how I've come to use git submodules. It's been helpful when working with various embedded projects. They often don't get updated for months or years (ideally), and then you only want to pull in specific changes.
      - chrsig 4y ago
        It's easy to overlook that the submodule is a complete git repository in its own right. When working on both, it's really annoying to have to commit to a dependency's repo, push, and pull down in the dependent repo. If you do the commit in the submodule, it's just immediately available...and you can still have whatever remote to push to. So it just cuts down on the number of checkouts you have to have.
- Phurist 4y ago
  That sounds like a problem that exists between the chair and the keyboard.
  - onion2k 4y ago
    In 25 years of software dev, and most of that in developing user interfaces to things, I have never found a commonly repeated error that could be attributed to user error. It's always badly designed software that ignores the user's mental model that's developed based on using the software. In this case, "git clone" clones a repo 99% of the time, because 99% of repos are shallow and simple. For "git clone" to only clone the top level when you have su modules instead of prompting to ask if you want a deep clone, or doing a deep clone by default because that's the expected behavior, is pretty poor design IMO.
- fanf2 4y ago
  Switching branches in a repository with submodules is a huge pain, especially if (like the Ansible repo) some branches have the subdirectory in the same repo like normal, and some branches have the same subdirectory in a submodule.
- esperent 4y ago
  > Git submodules are fine and can be really useful, but they are really hard If an important software tool is hard to use to the point that most people avoid it, then it's not fine. It's broken.
al2o3cr 4y ago
A couple things off the top of my head: * some folks had to deal with a LOT of submodules back in the day; for instance, it wasn't uncommon to have a dozen+ in your "vendor/plugins" directory in a Rails 1.x app. More submodules, more problems * sometimes submodules get used to "decompose a monolith" into a confusing tangle of interdependent repos. Actually changing stuff winds up requiring coordination across multiple repos (and depending on the org, multiple teams) and velocity drops to zero. Eventually somebody declares "submodules SUCK!!!!!one!!!" and fixes things by re-monolithing...
andix 4y ago
I always ran into issues when switching branches, merging or rebasing. And then you have to figure out what’s going on. If you’re not used to work with submodules, that’s the moment where you have to learn it. And a lot of people get overwhelmed then.
- grogenaut 4y ago
  This is what killed it for me. Changes to submodules don't seem to do well across commits/rebases, or bisect type operations. After fixing it on so many people's computers when using modules I just had us give up and move the stuff out.
d_watt 4y ago
Git submodules aren't bad in that they're buggy, they do what the documentation suggests. I think they're difficult to use, because it breaks my mental model of how I expect a repository to work. It creates a new level of abstraction for figuring out how everything is related, and what commands you need to be able to keep things in sync (as opposed to just a normal pull/branch/push flow). It creates a whole new layer to the way your VCS works the consumer needs to understand. The two alternatives are 1. Have a bunch of repositories with an understanding of what an expected file structure is, ala ./projects/repo_1, ./projects/repo_2. You have a master repo with a readme instructing people on how to set it up. In theory, there's a disadvantage here in that it puts more work on the end user to manually set that up, but the advantage is there's a simpler understanding of how everything works together. 2. A mono repo. If you absolutely want all of the files to be linked together in a large repo, why not just put them in the same repo, rather than forking everything out across many repos. You lose a little flexibility in being able to mix and match branches, but nothing a good cherry-pick when needed can't fix. Either of these strategies solve the same problem sub-modules are usually used to solve, without creating a more burdensome mental model, in my opinion. So the question becomes why use them and add more to understand, if there are simpler patterns to use instead.
- aslilac 4y ago
  You completely missed the problem that submodules are actually supposed to solve though. Using them for either of those cases would almost definitely be the wrong choice. What they're really for, is vendoring someone else's code into yours. They're still not great even at that, but sometimes they're the best option.
  - ddlsmurf 4y ago
    This î. Also subtree is an interesting relevant tool too.
  - d_watt 4y ago
    Interesting. When you say that's the problem they're intended to solve, do you have a link to that as their intended use case. IE is that "a" usecase or "the" usecase? I've never seen submodules used for that, only for internal dep management, so if there's content about "what they're really for," I'd love to read more.
jayd16 4y ago
The security model seems to be "terrible UX defaults that you turn to unsafe defaults instead." You end up with a lot of gotchas instead of them just working. The mental model of juggling multiple repos in a non-atomic way also violates the rule of least astonishment. Working with read only submodules smooths this part out at least. GUI support is slowly getting better at least.
- pengaru 4y ago
  > The security model seems to be "terrible UX defaults that you turn to unsafe defaults instead." Except practically nothing in git is unsafe, it's like Plan9's Venti in the sense that you can only add content to it. You never lose data, but you can easily lose your bearings. I presume a major reason why there's not much effort put into UX guard rails preventing losing one's bearings in the day-to-day work is a product of the fundamental fact that the underlying committed data is always there. People just need to at least familiarize themselves with `git reflog` to help lose the anxiety.
  - layer8 4y ago
    > the underlying committed data is always there That’s not always true, due to reflog expiration and GC. This is an important difference to e.g. Subversion where you can delete branches/tags just to hide them, but the history always remains 100% there and nothing is ever really deleted.
    - SahAssar 4y ago
      In git if a commit is within the history of a branch or tag it won't be GC:ed, right?
      - layer8 4y ago
        That’s correct, but commits from that history can also be removed, for example by squashing a branch. The Git history is quite mutable in that sense and doesn’t necessarily reflect the actual commit history.
TillE 4y ago
It's really annoying that submodules give you a detached head by default, so working on a submodule within a project is prone to mistakes. Otherwise they've been fine for me.
FrenchyJiby 4y ago
Beyond how hard to use they may or may not be, my personal hatred of git submodules is about bypassing your normal dependency management system. See 12 Factors on Dependencies[1]. I've not seen many uses of submodules that weren't better served by adding the package from pypi/npm/crates/... [1]: https://12factor.net/dependencies https://12factor.net/dependencies
- withinboredom 4y ago
  I've been reading a lot of research papers lately, sometimes with POC repos. When compiling the code, it would often fail because dependencies have changed over the years (we're talking about C/C++ code which doesn't have a package manager like you're probably used to). Most of these repos would fail to compile because it expected libraries installed on the system to not have changed in the past 10 years or so. In exactly one of these repos did the author use a git submodule so I was linking directly to the correct version of the library. Granted, those libraries _did not_ do this, so it still failed to compile 7 years later... If everyone used git submodules for deps, you'd end up with a (block) chain (if everyone's commits were signed) for deps.
  - FrenchyJiby 4y ago
    That's pretty cool reproducibility indeed, love it. I once worked in C++, and the lack of a (standard?) package manager was killing me for pretty much that reason. "This says it's using curl, but did they compile it with a weird flag, or was their version just old?", and stuff like that. For the case you describe, a submodule is the next best thing to proper package management, and I'm glad those submodules are there to save you! But in themselves, the submodules don't guarantee build instructions, and the same guarantees of exactness can be had by locking dependencies. I feel like your message mostly reinforces my belief in the NEED for package manager for every such language combo, having seen the immense value in other systems, and the extremes that we have to go to when they're missing. I hear the Nix (and Guix) community has done great steps to manage these stray systems into reproducibility, using submodule chains of the sort you describe, but augmented with standardized build info to make it system-wide pkg management.
sto_hristo 4y ago
They have their uses and misuses. People misuse them a lot, get burned, they blame them, and then they hate them. If you lack a dependency mgmt system, work mostly solo or in very small and tightly coordinated team and just need some githubbed project to make yours work, submodules may be the right tool for you.
- throwawaylala1 4y ago
  Exactly! Submodules are a tool within a toolkit. Learn the toolkit, learn the tool, and then decide whether or not it'll solve your problem.
  - Dylan16807 4y ago
    But this is such a non-answer to what's good or bad about them.
    - throwawaylala1 4y ago
      It was a response to a comment, not the OP. There is nothing "bad" about submodules. They do what they were intended to do very well. They are often misused because people don't take the time to understand how things work (event at a conceptual level) before diving into it. I've found submodules most useful when I'm tracking an external dependency that doesn't change often -and- I don't have a dependency management system to lean on. For example, I have a repository of 100+ various bash utility scripts. One of the scripts depends on another repository with 500MB of data. I don't use it often so I don't need that large repo cloned alongside my bash scripts every time. In this case: 1) The dependency doesn't change often and I like being deliberate when using a new version of that dependency. 2) The dependency is optional and can simply be ignored when using Git's default clone behavior for my repo. 3) Since it's just a bunch of bash scripts, I don't have a package management system to lean on. I use these scripts on multiple *nix distributions. Could I make it work without Git submodules? Yes. I would have added the dependency to .gitignore and wrote a script to `git clone` the dependency and set it to a specific revision. Submodules just makes that process a lot easier.
arjvik 4y ago
What's the mental model for the use of a git submodule? I've always thought of them as a way to "vendor" a git repository, i.e. declare a dependency on a specific version of another project. I thought they made sense to use only when you're not actively developing the other project (at least within the same mental context). If you did want to develop the other project independently, I thought it best to clone it as a non-submodule somewhere else, push any commits, then pull them down into the submodule.
- ghoward 4y ago
  That's how I have thought of them too, so I've never struggled with them. Hence why I asked the question. I'm glad to know that I'm not the only one to use them as such.
- oivey 4y ago
  I think submodules end up highlighting failures of the (implementation of the) many small repo model. People want to develop in both the submodules and main repo simultaneously, and that’s relatively painful. If you find yourself doing that often, that’s a sign that your repos are highly coupled and failing the intent of the many small repo model anyway.
- preseinger 4y ago
  The main thing that vendoring is supposed to do is to make it so you can build your code even if all your deps disappear. Submodules don't get you that property.
- mewse 4y ago
  I have a custom game engine, and I make games that use that engine. I have about ten different games, all using the same engine, but they were written over the course of many years and so are written against different commits within that engine repo, and git submodules capture that idea perfectly. If I didn’t have something like that where I effectively had a long-lived library which I wanted to pull into multiple projects, I probably wouldn’t bother with the submodules. But it’s *so* much more convenient to have them in submodules than to copy the engine code separately into each project’s separate repos and then manage applying every change to the engine in all the different game repos individually. Really, git submodules are exactly the same thing as subversion ‘externals’, but always pinned to a specific commit (which is an available-but-not-enabled-by-default option under subversion), and with a substantially easier interface so maybe folks who don’t need them are more likely to notice them and grumble about how they don’t solve an issue they have? IMHO git submodules are a huge quality of life improvement over the same system from subversion (as was available in subversion back in the 1.8.x era; I haven’t really used svn in anger in a long time). I definitely wouuldn’t want to go back, or to not have them available.
zbuf 4y ago
If submodules were really slick, they'd become the primary way to manage build-time dependencies.
mkl95 4y ago
Git submodules aren't so bad, but they are a pretty leaky abstraction.
Groxx 4y ago
Submodules are just complicated because Git makes no decisions at all about how they should behave, beyond "never make a decision that could lose data". So you have to understand the tradeoffs and make every decision at every step. It's the safe option. Like, what happens if you remove a submodule between revisions? Git won't remove the files, you could have stuff in there. So it just dangles there, as a pile of changed files that you now have to separately, manually remove or commit, because it's no longer tracked as a submodule. And then repeat this same kind of "X could sometimes be risky, so don't do it" behavior for dozens of scenarios. All of which is in some ways reasonable, and is very much "Git-like behavior". But it's annoying, and if you don't really understand it all it seems like it's just getting in your way all the time for no good reason. Git has been very very slowly improving this behavior in general, but it's still taking an extremely conservative stance on everything, so it'll probably never be streamlined or automagic - making one set of decisions implicitly would get in the way of someone who wants different behavior.
WanderPanda 4y ago
Whats up with git submodules refusing to clone/update/checkout submodules while having all their files showing up as deleted? I encounter this quite a lot and the solution seems to be a git submodule sync --recursive (or something like that) but I don't get why I run into this in the first place? Probably related to forgetting --recurse-submodules when cloning but what do I know?
sascha_sl 4y ago
Submodules aren't bad. But in a world where I have to explain that running revert on the 250 megs of jar files someone comitted isn't a fix, and where people often just delete and re-clone entire repos because they don't know what's going on, they incur a heavy support burden on the few people that know how to use them. You know, the people that already carry everyone else through their job.
- seabrookmx 4y ago
  I feel your pain. I don't expect everyone to be an expert on everything but given how important version control is in this profession, the lack of understanding (or desire to understand!) it's fundamentals by many in our industry is downright sad.
TrianguloY 4y ago
I don't use submodules, but I do use git repositories inside other git repositories, and let IntelliJ manage them both simultaneously as if they were two different projects. Works fine. I once enabled it as submodule to test. It adds nothing and from that moment any change in the child creates a change in the parent, which for my use case is totally unnecessary (I want both of them to be independent, even if hierarchically one is inside the other). Submodules are probably a good option to have libraries that you rarely touch, so you can update/modify them as with a maven/gradle project. For most other user cases submodules make more problems that advantages.
- djmips 4y ago
  On a project where you have a lot of people working on the same code, an advantage (of submodules) is that you can check out someone's branch and it contains the URLs to the specific commits in the submodules that work with their branch.
howinteresting 4y ago
The problem with submodules is that they're read-write. Read-only submodules would be completely fine.
- juped 4y ago
  Yeah this is basically it in the course of actually using them; people should probably only submodule in tags as well. Some of people's problems also come from using them where they're not necessary.
dathinab 4y ago
They are unergonomic! Annoying to setup and keep in sync "correctly" (for given project, EDIT: especially edits). Sure, this depends a bit on the other tooling and for what reason you use. This doesn't meant they are hard or complicated, but the moment the defaults do not do what most times is needed and you can't change the default (in the project, instead of user settings) i.e. need to do additional manual steps all the time some people will hate it many will dislike it.
withinboredom 4y ago
If you think git submodules are bad/confusing, wait until you see subtrees.
anonymoushn 4y ago
If you work on blorp which contains openresty which contains luajit, and you have a patch to luajit that you are making because it enables some work in the end product, you need to make a commit to luajit, make a commit to openresty with your changes, make another commit openresty to change the luajit version, make a commit to blorp with your changes, and make another commit to blorp to change the openresty version. You will create 3 code reviews none of which actually contain all of your changes together. Your coworkers have decent odds of not being able to build their software because they don't have a keybind for `git submodule update --init --recursive` yet.
tommyjl 4y ago
I recently started using git subtree[0] instead of dealing with all the problems with git submodules, and have been very happy with the experience so far. It does copy every file into your repository, though. [0]: https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt https://github.com/git/git/blob/master/contrib/subtree/git-s...
- matthewmacleod 4y ago
  This is funny - I was going to chip in and say that you really start to appreciate submodules once you’ve experienced the fractal hell of subtrees! If I ever have to untangle a messed-up repository containing a subtree again in my life I’m quitting development and moving to a cabin in the woods.
  - tommyjl 4y ago
    Haha, my usecase is really simple. I have a couple of applications that need to run at the same time on the development machines and they only need to be updated once or twice a year. With subtrees, only the one person who maintains the subtrees actually needs to care about the other applications, and the other developers can mostly ignore that the other applications exist. If we used containers, a Docker compose file would make things even simpler for us, but alas.
  - preseinger 4y ago
    I have exactly the opposite experience. Submodules are hell, subtrees are simple.
    - silon42 4y ago
      Agreed.
dustingetz 4y ago
mutable pointers in the middle of your immutable lineage. completely broken model
JoshTriplett 4y ago
The biggest reason I find git submodules painful: a "commitlink" object in a git tree does not count as a reference to that commit or anything that commit references, for the purposes of garbage collecting the repository or pushing and pulling changes. You can't have the only reference in your repository to a given commit be a commitlink within another tree. I'd like to jettison the entire model of "reference another repository that you may or may not have", along with the `.gitmodules` file as anything other than optional metadata, and instead have them be a fully integrated thing that necessarily comes along with any pull/push/clone.
Pathogen-David 4y ago
As many others in this thread have stated, the main issue is they have fairly poor UX and if you aren't used to them they can be pretty annoying. They especially have quirks when they're removed from (or moved within) an existing Git repository. One thing I haven't seen mentioned in this thread though is that they force an opinion of HTTPS vs SSH for the remote repository. If a developer usually uses SSH, their credential manager might not be authenticated over HTTPS (if they even have one configured at all!) If they usually use HTTPS, they might not even have an SSH keypair associated with their identity. If they're on Windows setting up SSH is generally even higher friction than it is on Linux. For someone just casually cloning a repository this is a lot of friction right out of the gate, and they haven't even had to start dealing with deciphering your build instructions yet! ------- Personally I still use Git submodules despite their flaws because they mesh well with my workflow. (This is partially due to the fact that when I transitioned from Hg to Git it was at a job that used them heavily.) The reality is every solution to bringing external code into your project (whether it's using submodules, subtrees, tools like Josh, scripts to clone separately, IDE features for multi-repo work, ensuring dependencies are always packages, or just plain ol' copy+pasting) all have different pros and cons. None of them are objectively bad, none are objectively best for all situations. You need to determine which workflow makes the most sense for you, your project, and your collaborators.
truffdog 4y ago
Pointers are hard.
normaltool 4y ago
I don't think most people have a complete idea about git submodules. but I think it's easier to say bad directly because everyone around him is vilifying it.
Izkata 4y ago
They act almost the same as pinned-revision svn externals, which people don't really seem to have a problem with. The biggest difference I can think of is needing a special command to pull in the submodules, where svn pulls its externals automatically.
tkuraku 4y ago
Git submodules aren't perfect, but they can definitely be useful. I use them all the time and for my use case they are a good solution. They do take some getting used to though.
AceJohnny2 4y ago
Git submodules also don't interact well with worktrees [1]. Do not try to change the branch of a worktree that has a submodule. [1] https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree
pornel 4y ago
The SVN implementation worked pretty seamlessly, almost like a regular subdirectory. There was no gotcha of a non-recursive clone/checkout. If you've used this feature, your users wouldn't keep getting "broken" checkouts. There was no gotcha of state split between .gitmodules, top-level .git state, and submodule's .git, and the issues caused by them being out of sync. There was no gotcha of referencing an unpushed commit. Submodules are weirdly special and let all of their primitive implementation details leak out to the interface. You can't just clone the repo any more, you have to clone recursively. You can't just checkout branches any more, you have to init and update the submodules too, with extra complications if the same submodules don't exist in all branches. You can't just commit/push, you have to commit/push the submodules first. With submodules every basic git operation gets extra steps and novel failure modes. Some operations feel outright buggy, e.g. rebase gets confused and fails when an in-tree directory has been changed to a submodule. Functionality provided by submodules is great, but the implementation feels like it's intentionally trying to make less-than-expert users feel stupid.
jlokier 4y ago
I would love to hear from people who've been using the "git subtree" command instead. Any good experiences? --- My colleagues and myself lost work a few times when working on a project whose top-level Makefile / build script ran some "git submodule" commands from time to time, when they detected that a submodule appeared to be out of date. Those commands wiped work in progress, because of submodules' tendancy to leave junk around when switching to-level branches, or the set of submodules changed, or the same with recursion over vendor submodules. That junk caused a few end-users to end up building and running incorrect code, or to get unexpected build errors, so the policy was for the build scripts to wipe it. In other words, policy was to prioritise the experience for end-users who cloned the repo and occasionally updated or switched branches. Unfortunately that meant it clobbered work in progress by devs occasionally. If you were careful it wouldn't happen, but if you left a trail of small changes and forgot about them, and did a git pull or such to get on with other work, intending to still find those work in progress changes later, they'd sometimes be gone. Such changes were things like those which needed to edit inside and outside a module (e.g. for API changes), or improvements to a submodule's code that there was no hurry to finish or commit, changes to add diagnostics, spotted improvements to tests, etc. Often when those changes were wiped, the dev wouldn't notice for a while. Then later, "eh, I thought I had already done this before...". My solution was to stop using the standard build target, and remember to use "git submodule" myself at appropriate times such as when switching branches and updating. That way I never lost any work, but it was not how I was "supposed" to work. The team discussed improvements to the auto-update-clean-and-hard-reset commands, to make it check more carefully so it wouldn't run as often. But the problem remained, and that refinement made the build options rather ugly, a kind of "make REALLY_REALLY_DONT_UPDATE=1" sort of thing. Sensible defaults that Just Work™ for everyone were never found. I also found submodules annoyingly time-consuming when making many changes that spanned inside and outside modules or across modules. The dance backward and forward to make a PR in the module to something that's compatible with before-and-after, then a PR in what uses it, then a PR clean up the module, perhaps multiple iterations needed, with each step having a potentially slow review cycle in between, over GitHub's slow web interface as well. Understandable for stable APIs shared among multiple projects, but pointless busywork (and worse commit logs) for small changes that could be a single PR in a monorepo. (ps. Please see first line of this comment.)
deleted 4y ago
[deleted]
cppforlife 4y ago
i never found myself struggling with submodules, but at times i found myself just slightly annoyed (especially when having to remove/replace submodules), especially when they are used for simpler use cases. i actually ended up creating https://carvel.dev/vendir/ https://carvel.dev/vendir/ for some of the overlapping use cases. aside from not being git specific (for source content or destination), its entirely transparent to consumers of the repo as they do not need to know how some subset of content is being managed. (i am of course a fan of committing vendored content into repos and ignore small price of increasing repo size).
neallindsay 4y ago
I think Git submodules are good, but they have a very narrow set of use-cases. Sometimes people use submodules when what they really want is a sub-tree merge.
bjt2n3904 4y ago
We've just transitioned from using submodules at work to subtrees. I hate them slightly more than submodules. Git describe? Totally broken. Everything else is a tangled mess of junk. But what armchairhacker says is SPOT ON.
lobocinza 4y ago
Git submodules are cool but can be confusing because people are already used to their language package manager. They also add overhead as changes frequently have to be pulled/pushed downstream/upstream. But in cases where it makes sense to use it, it's a great tool. Eg: theme that is reused in 3 sites is in it's own repo and is a submodule in each site.
polski-g 4y ago
I recommend git subrepo
senorsmile 4y ago
They're very useful. git submodule update --init --recursive will cover 99% of the time. There are weird ways of working with updates to submodules, but for the most part everything just works.
kazinator 4y ago
Instead of submodules, you can just fech whatever repo you want into your repo. Create a tracking branch for it which moves the stuff into a subdirectory, and then merge that to your master. Moreover, when you do the initial fetch, you can limit the depth. That will save space if you don't care for the full history of that repo.
Too 4y ago
A lot of people here complain about the complex UX. This a big problem but is something you get used to and can live with. An even bigger problem is when you start substituting a dependency manager with submodules. It has no way to deal with transitive dependencies or diamond-dependencies. What are you going to do when lib A->B->D and A->C->D? Your workspace will now have two duplicate checkouts of D and any update to D requires commiting 3 repos in sequence to update the hashes. If you are really unlucky there can only be one instance of D running on the system but the checkouts differ. The correct way to deal with this is to only have one top level superproject where all repos, even transitive ones are vendored and pinned. The difficulty is knowing if your repo really is the top level, or if someone else will include it in an even bigger context. Rule of thumb would be that your superproject shouldn’t itself contain any code, only the submodules.
dekhn 4y ago
Because git is a version control system that is so in love with its data structure, it can't find time to make the rest of the system coherent or useful.

Falkon1313 4y ago

Odd, I found branching and merging so much easier in SVN. Git doesn't even technically have branches, just pointers to commits which can easily get mixed up, go headless and fail out in ways that just never happened in SVN. And rebasing a branch with several commits can be a nightmare since you have to re-merge almost (but not exactly) the same code over and over again at whatever state it was in at some time in the past when some previous commit happened - unless you abort out and squash first. In SVN, you just merged the whole branch once and when both were in their final current state. Of course, git's a lot more powerful, but with that comes complexity. SVN branching and merging was a snap comparatively.

kevin_thibedeau 4y ago
SVN will frequently insist that merge conflicts exist where there shouldn't be any if trees have been modified by deleting or moving directories. This is so pervasive that organizations will avoid doing merges because of the manual fixups you have to do on long lived branches. There's metadata now for tracking branch history but older versions couldn't figure out that two branches in a merge had a common ancestor. A puzzling thing to omit from a VCS.
- COMMENT___ 4y ago
  AFAIK, conflict resolution in Subversion is being improved: * https://subversion.apache.org/docs/release-notes/1.10.html#conflict-resolver https://subversion.apache.org/docs/release-notes/1.10.html#c... * https://subversion.apache.org/docs/release-notes/1.11.html#conflict-resolver https://subversion.apache.org/docs/release-notes/1.11.html#c...
Izkata 4y ago
> Git doesn't even technically have branches, just pointers to commits which can easily get mixed up, go headless and fail out in ways that just never happened in SVN. I mean, if that's a reason to say git doesn't technically have branches, then neither does svn. It has subdirectories that you can make copies of at any level, and copy commits between them at any level, such that you can make an amazing repo-within-a-repo mess not possible in git.

abirch 4y ago

Git makes sense for the Linux kernel and merging patches at scale. 99% of software development would be fine with subversion

lambda_lord 4y ago
Git completely replaced Subversion so quickly because the benefits were apparent even at a small scale. Subversion was centralized and slow whereas Git branches were cheap and fast. It turns out the distributed model is just a lot better. Even my college project teams benefited from the superior experience of Git.
- bsder 4y ago
  Except that everybody changed to Github not git. And effectively recreated Subversion with caching.
  - dopidopHN 4y ago
    No, I worked with gitlab, bitbucket, custom git server installs in the last 3 years alone.
  - PaulDavisThe1st 4y ago
    "Subversion with caching" is not subversion. I used subversion for a long time and was resistant to moving ardour.org to use git instead. 24hours after we switched (we never use 3rd party git hosting as our canonical repo), I was already convinced it was not the right choice, but an excellent choice.
  - astrange 4y ago
    It's also Subversion where you can commit your changes and write the message before pushing, instead of at the same time, so you get the chance to review it. That's enough to make it better.
- pizza234 4y ago
  Interestingly, there was a distributed SCM build on top of SVN, called SVK (https://wiki.c2.com/?SvkVersionControl https://wiki.c2.com/?SvkVersionControl). Being distributed, it solved the main gripes with SVN; it also added a better merging algorithm (https://foswiki.org/pub/Development/SVK/svk-visual-guide.pdf https://foswiki.org/pub/Development/SVK/svk-visual-guide.pdf), solving another big gripe. I was actually satisfied of it, and surprised that it never got attention, in particular, because there were no requirements in order to use it with existing SVN repositories. I'm actually baffled, because SVN is still active, so SVK would still be useful nowadays.
  - kevin_thibedeau 4y ago
    May as well use Git SVN integration for this.
dopidopHN 4y ago
Have you use subversion ? If yes, do you remember how slow it was? How cutting a branch and then merge it was seen as something for senior dev to handle ?
- abirch 4y ago
  I did use subversion in college. Never professionally, so creating our own branches wasn't a pain. I think there is a reason why this XKCD was made: https://xkcd.com/1597/ https://xkcd.com/1597/
  - avgcorrection 4y ago
    “Delete your work and clone again” is ridiculous. Just goes to show that you don’t know what you’re talking about (or the xkcd guy for that matter).
    - abirch 4y ago
      I'm talking about my needs, ditto with the XKCD guy. Basically I just need to be able to create a PR from my local changes. Not much more.
pantalaimon 4y ago
Subversion requires a server, so it’s not suitable for the small local one-off repo
- rad_gruchalski 4y ago
  No. It doesn’t. svn checkout file://path/to/repo works absolutely fine.
  - oneplane 4y ago
    But it does need a server to collaborate, doesn't it?
    - rad_gruchalski 4y ago
      Define a server. NFS will be enough for more than one person to use a repo over the file protocol. It will work fine over a Windows share, Samba, NFS, and so on. It doesn’t need svn or http protocol to operate.
      - oneplane 4y ago
        I could define a server but I'm not a dictionary so I won't. SVN doesn't work peer-to-peer or over email. And that's fine. It's just not ready to go with only local tools. Of course, you can use GitHub with Subversion just fine, but that wasn't the point. The point was that Subversion alone is never enough if you want to collaborate.
        rad_gruchalski 4y ago
        > I could define a server but I'm not a dictionary so I won't. That’s a bummer. It would help us point the discussion in the right direction. > SVN doesn't work peer-to-peer or over email. Why not? What is so different about people having their own subversion repositories over file protocol vs people having their git repositories? Why would you not be able to send a subversion patch in an email? As someone who uses git for 10 years, I understand it may not be as ergonomic as with git. But why not? > The point was that Subversion alone is never enough if you want to collaborate. Why not? Is git enough?
avgcorrection 4y ago
Anything can be made to work. But I don’t see why I would want to handicap myself with a truly centralized SCM system. Sure, we use one canonical repository. All our “pull requests” are really merge requests, mostly to the main branch. So that’s pretty centralized, right? So why use a distributed VCS? Well, why use a local editor or IDE for code that is ultimately going to end up in the cloud somewhere? Sure, you might want to out of preference, but why should you be forced to? The fact is that wherever the code will end up is besides the point when it comes to how to develop it. The truly important thing about distributed VCS is that it forces almost all of the operations on the repository to be usable locally. And why should it not be? What’s “git log”, “git blame”, or “git merge” got to do with whether there is one canonical repo or a hierarchy of upstreams? I think that this idea that non-distributed VCS is somehow the default—as in the obvious, simple thing to implement—is just backwards. Of course the default assumption for any VCS operation—unless it has a name like “send-email”—should be that it operates on your own local copy. Sure, we use a centralized repo structure. And the only call-central-command operation I use is “git push”. All the other fiddling and querying—and all the things that make version-control-as-history useful—is local.

akavel 4y ago

Mercurial covers generally the same concepts as git and is thus also not trivial to learn for someone uninitiated; yet its interface was like day vs. night when compared to git since their very early days. It proves one can design a decent interface if one actually tries to care about usability and friendliness. How I remember the past days, git won the rivalry squarely due to GitHub becoming popular (though I assume there were some reasons why GH chose git over hg).

mumblemumble 4y ago
Back in the day there were actually 2 or 3 different cloud SCM hosting providers that chose Mercurial. As I recall, a couple of them, BitBucket and Kiln, also had better Web UIs than GitHub did. Versus just GitHub offering Git, and it was kind of a duffer in my opinion. GitHub has come a long way. But I would guess the main reason it managed to become dominant is not because it had a better product. (At the time, it didn't.) It's because Git benefited from the celebrity of its author, Linus Torvalds.
- theamk 4y ago
  It's crazy how people forget the past. Back in the day, git was "rewrite, rebases modify past to make beautiful commit" vs hg "rewriting past is bad, beautiful commits are lies about history". Turns out people don't care about truthful history. (Nowadays mercurial can do rebase/amend just fine.. But it is too late)
  - mumblemumble 4y ago
    I think that the truth is probably somewhere in between. I do like to squash and rebase before moving changes upstream. But, to me, that isn't really history quite yet. Or at least, it's not history that's worth recording. All those micro-commits from the work in progress are, in some ways, more akin to my editor's undo history. Which is also something I don't save. It's also clear, in hindsight, that Mercurial's original position on this subject failed to anticipate AWS credentials accidentally being committed to source control. But I have also seen (and done) some amount of history rewriting in long-lived branches that I don't think would have been necessary if Git had had some of Mercurial's ergonomics. Workflows for merging two different repositories while retaining the commit history from each, for example.
    - capitainenemo 4y ago
      FWIW, Mercurial has had "censor" command for blowing away the contents of those revisions with AWS keys since 2015. Although once stuff is pushed to the public repo you're probably going to want to change those keys regardless. And if it's on the local one, there's plenty of options for removing the commit.
  - astrange 4y ago
    Rewriting history is actively good because it means you can view it as a series of logical patches. I have worked on large projects using earlier versions of hg and they were absolutely full of merge commits just labeled "Merge" - some of them were safe, some of them had random changes in them, and some of them had automatic merged changes that actually caused problems. It was also much slower than git. But I knew someone working on Google Code at the time who liked it better because it was "clean" and in Python.
    - capitainenemo 4y ago
      It was much much faster than git at HTTP at the time though. That's why Google Code selected it. Also it was faster at imports which was why Mozilla selected it for their transition. Some other things it was slower, that's changed over time ofc. Also, in terms of clean history, mercurial has best of both worlds with phases and hidden by default commits to keep track of such cleanup.
      - astrange 4y ago
        > Also, in terms of clean history, mercurial has best of both worlds with phases and hidden by default commits to keep track of such cleanup. Yeah, it has more features now but at the time it didn’t. There was something called patch queue you could use for early stage work but that was all.
  - tessierashpool 4y ago
    in other words, the comment you’re replying to rewrote the past of rewriting the past.
  - thayne 4y ago
    The fact that mercurial is substantially slower than git was probably also a big factor.
- Izkata 4y ago
  Around 2013, having only knowledge of svn at the time, I tried both git and mercurial to see which I liked more and found git to be a lot more intuitive than mercurial. It's been long enough I don't remember the details about why I didn't like mercurial, but fame power had nothing to do with it, nor did website integration - I was only using it locally. How it worked just didn't fit with how I thought about version control.
cycomanic 4y ago
That's not what I remember. Back in the days when people started implementing dvcs git was just much faster than everyone else (that was in fact the reason why Linus wrote it). Once the kernel was using it, its mindshare just grew much faster because of the publicity this implied. In other words it was largely a case of "if the kernel devs are using it it must be good". When GitHub and all the other hosting services started many still had mercurial or other dvcs (launchpad was bzr for example), but by that time the ship had sailed already I would argue.

Dylan16807 4y ago

> It was a response to a comment, not the OP. You were elaborating on the comment, weren't you? And that comment was quite vague too. > There is nothing "bad" about submodules. The difference between "what are uses and misuses" and "what is good and bad" is not enormous. I think you're nitpicking. Plus, there are things that are flat-out bad. It's a human-made program under constant development, after all.

throwawaylala1 4y ago
> You were elaborating on the comment, weren't you? And that comment was quite vague too. What was vague about that comment? > The difference between "what are uses and misuses" and "what is good and bad" is not enormous. This is incorrect. The definition of "bad" is something of low quality. Misusing a tool and then calling it low quality is dumb. > Plus, there are things that are flat-out bad. It's a human-made program under constant development, after all. What about git submodules is "flat-out bad"? It's ironic that you're criticizing others' comments on being vague and providing non-answers while you've contributed nothing to the discussion. Maybe educate yourself on technical things and you won't need to waste your time having meta discussions on internet forums. Looks to me like you want to contribute but have nothing of value to say.
- Dylan16807 4y ago
  > What about git submodules is "flat-out bad"? Do you want me to grep the changelog for fixed mistakes? Do you think they fixed all of them, just now, june 2022? There are mistakes, which are flat-out bad. Trying to fix them is a continuous process that is not complete. And there are probably bad decisions that will be baked in forever, but that's not necessary to prove my point. > It's ironic that you're criticizing others' comments on being vague and providing non-answers while you've contributed nothing to the discussion. It's not ironic. One person asks for input. Another person gives a vague answer. I call the answer too vague to be useless. That doesn't make it my job to be specific now. I'm not answering OP. I'm giving feedback on a different answer. Which is what you were originally doing too, so don't complain that it's not "of value". And I'm not criticizing submodules here either. I'll make my argument more clear. The idea that any software has "nothing bad" about it is ridiculous. That's a generic argument for all software. If you really want examples, look at the changelog. But I think it should be obvious. If you think it would even be useful for me to provide an example, then there's probably been a terrible miscommunication.
  - throwawaylala1 4y ago
    The miscommunication is in your mind. Every software out there has defects. Bugs, performance issues, UX issues, etc. To assume that anyone in this thread actually means that submodules have no defects whatsoever... well... that's just creating a straw man to tear down because you're grumpy or something. The issue appears to be with your understanding of what "bad" means in this context. When the OP (and commenter I was responding to) says "bad", they're referring to the overall experience of using git submodules. That specifically does not include the minor fixes/improvements you'd find in a changelog. By your definition, all software is "bad" because all software has defects in varying degrees. That of course makes the entire conversation pointless and pedantic. Git submodules work just fine when they're used properly. I've been using them for years without issue and so have many other commenters here. Are there things we (the users of git submodules) would like to see improved? Of course! But that doesn't violate the statement that there is "nothing bad" about submodules in the context of this discussion. I really hope this helps you navigate conversations with a little more nuance. If not, that's okay too. Have a nice weekend.
    - Dylan16807 4y ago
      > When the OP (and commenter I was responding to) says "bad", they're referring to the overall experience of using git submodules. Ah, here's the part where things went wrong. I said "what's good or bad", which is not a judgement of the overall experience. It's the same thing as "uses and misuses" from a different angle. Talking about what software is better at and worse at is a good response to "why is it bad?". "No it isn't" is mostly a waste of time, especially when a lot of people have bad experiences. And don't say that the miscommunication is in my mind when you glossed over the word "good" entirely. > By your definition, all software is "bad" By my definition, all software has bad elements and bad contexts. And good ones, too. Asking where those lie is not pedantic or pointless at all. > that doesn't violate the statement that there is "nothing bad" about submodules in the context of this discussion. If an "overall experience" being good enough means that there is "nothing bad", I'm not the one lacking nuance. You've just obliterated nuance.