4 ms·

I'd be interested to know why projects like this don't make use of git submodules. They take some getting used to but once understood seem to do a reasonable jo

by zmmmmm 4y ago

I'd be interested to know why projects like this don't make use of git submodules. They take some getting used to but once understood seem to do a reasonable job of multi-repository projects to me.

theshrike79 4y ago
Submodules are an utter clusterfuck. I consider myself a pretty experienced user and every time I try to use submodules, they bite me in the ass eventually. The way they are operated just doesn't fit the way any human thinks.
- cflewis 4y ago
  Gosh, I’m so pleased for all the replies here. I felt the same way. Submodules made me feel stupid, like my mental model of Git was just entirely broken. What I wanted to do was be able to have people work against a pinned version of a different Git repo, then update the sub module whenever we felt the need and handle the build breaks. This task seemed impossible to do correctly over time, which I just could not understand. How was the submodule getting updated when I didn’t call anything? Why are submodule changes appearing in other commits? I just couldn’t figure it out. I am joining a new project and they started talking about submodules and I what I said was “yeah uhhuh cool” but inside I was pretty nervous . But I couldn’t be sure it wasn’t because I was a dummy and they knew exactly what they were doing, so I kept quiet.
tssva 4y ago
The first section of the article discusses submodules and some of the pain points they suffered while moving to them.
aseipp 4y ago
Because submodules absolutely suck, and if you have the option of merging the two repos, they're 1000x more terrible. This terribleness increases nearly linearly with the number of active project participants. Why? Because they completely violate Git's operating model, which is to track content, not pointers to content. This is the fundamental problem, regardless of project dynamics, Kubernetes, whatever. In fact, that they make "git commit", one of the most fundamental operations you can perform, significantly riskier, because you can easily introduce submodule pointer changes from your dirty working copy. And people do this all the time. One of the most common cases is committing a submodule "update" while accidentally in a dirty working tree, so when you push the commit pointer simply doesn't exist anywhere. They also fail immediately once you try to `git merge` (or less commonly, git cherry-pick), because their merge conflicts, by definition, cannot be resolved automatically. Because they are pointers, not content. But git works on content. So just to be clear, two of the most fundamental day-to-day operations you can perform are turned into massive liabilities from this feature, ones that are likely to either break your build and/or just make your life harder. As someone who had to maintain stable and development branches of a project, cherry pick between the two, cut releases, etc, submodules are simply hell, because they make an already difficult job worse. This is a good sign that they are a liability. In a past life we actually had so many people push invalid submodule updates over time that we eventually wrote a git hook on our server to reject all commits with submodule updates that didn't exist in the corresponding repository, and that were not specifically tagged in the commit message as updating a submodule (through a magic set of keywords.) The fact we even had to do this is its own pain. I have maintained projects that have long-standing histories with dozens of submodules. And every single time we eliminated one of those submodules (often by merging into the parent repository, or simply dropping the dependency entirely), we all breathed a sigh of relief, and our lives all got significantly better from that point forward. As you can tell, this experience has made me very prepared to fight against submodules everywhere I might see or encounter them. But trust me: it's for your sake, not mine; 'cause there ain't a chance in hell anyone is adding any to my repositories.
- tedmiston 4y ago
  It sounds like you are arguing in favor of vendored dependencies over submodules?
  - aseipp 4y ago
    If I had to choose between the two? Probably, yeah. But it depends on what's being vendored, I guess. Like, I might not want to vendor the entire transitive dependency tree of some app if it's not security critical. But I would probably vendor the tree of an OSS repository I patched extensively to meet my needs, yes. Depending on the environment some other solutions might be possible. It's a complex topic. I will admit that if your dependency in this case is something that changes extremely rarely, and will only see updates maybe like, bi-annually, submodules are "OK." Not great, but they'll work, and presumably won't inflict massive psychic damage on your team members. But it's once they receive any foot traffic by more-than-one-person that all the real pain begins.
olvy0 4y ago
That's strange this just trended on HN. My team is about to decide on the design of a new feature for our internal product, which can essentially be done by either using sub-modules or re-implementing them. Our product has a somewhat simplistic git interface (behind the scenes it's anything but) and I've tried to keep it so, however lately customers have started demanding we also support submodules. The problem is that we use git to mirror a hierarchical database, so using submodules means mirroring another hierarchy inside our hierarchy. This would mean changing the current assumptions in the code to ignore things in the sub-hierarchy except for the sub-sub-hierarchy we care about. Yeah this is hand-wavy but the design constraints I've had are kinda hard to explain. Also, sub-modules would require changing all our git calls to take submodules into account, including cloning, reset, branches etc. And I've read many people's bad experience with sub-modules, including the ones in this sub-thread and so now I'm afraid they might hurt our maintainability in the long run. I've thought of a couple of ways to do this without sub-modules, including using git subtree, but all of them have drawbacks. I've actually found a neat way to merge a sub-directory of another remote repo to the current repo - which means we wouldn't have to change any of the existing code. It involves only "standard" git commands, basically only checkout, reset, and merge (without "exotic" commands like subtree, read-tree, and whatnot). And using "exclude" to keep only the content of a single subdirectory of the remote. But it does require us to maintain a file that's exactly like .gitsubmodules to keep track of remotes. And that's the thing which git submodules does for us "for free". Also I've developed a bad state for bespoke solutions and NIH. I already fear I have contributed more than enough NIH to my company by developing the existing solution, but given the conditions I think it was the only logical solution (a previous bespoke solution failed and was cancelled). But the longer I read and experiment with submodules it looks like they are also a kind of bespoke solution around basic git, and essentially require changing the way you handle operations such as reset, checkout, etc. Training all our users to fix errors due to out-of-sync submodules looks like a nightmare, when they already have problems with the current solution and with git in general. So I'm really conflicted on what our current path should be.
- tedmiston 4y ago
  I seem to be one of the few in the comments that actually likes submodules. But the description of this use case / requirement sounds complicated / confusing to me ("we use git to mirror a hierarchical database, so using submodules means mirroring another hierarchy inside our hierarchy"). I don't know what your product is but I hope for your sanity a PoC is possible.

by zmmmmm 4y ago

I'd be interested to know why projects like this don't make use of git submodules. They take some getting used to but once understood seem to do a reasonable job of multi-repository projects to me.

theshrike79 4y ago

Submodules are an utter clusterfuck. I consider myself a pretty experienced user and every time I try to use submodules, they bite me in the ass eventually. The way they are operated just doesn't fit the way any human thinks.

cflewis 4y ago
Gosh, I’m so pleased for all the replies here. I felt the same way. Submodules made me feel stupid, like my mental model of Git was just entirely broken. What I wanted to do was be able to have people work against a pinned version of a different Git repo, then update the sub module whenever we felt the need and handle the build breaks. This task seemed impossible to do correctly over time, which I just could not understand. How was the submodule getting updated when I didn’t call anything? Why are submodule changes appearing in other commits? I just couldn’t figure it out. I am joining a new project and they started talking about submodules and I what I said was “yeah uhhuh cool” but inside I was pretty nervous . But I couldn’t be sure it wasn’t because I was a dummy and they knew exactly what they were doing, so I kept quiet.

tssva 4y ago

The first section of the article discusses submodules and some of the pain points they suffered while moving to them.

aseipp 4y ago

Because submodules absolutely suck, and if you have the option of merging the two repos, they're 1000x more terrible. This terribleness increases nearly linearly with the number of active project participants. Why? Because they completely violate Git's operating model, which is to track content, not pointers to content. This is the fundamental problem, regardless of project dynamics, Kubernetes, whatever. In fact, that they make "git commit", one of the most fundamental operations you can perform, significantly riskier, because you can easily introduce submodule pointer changes from your dirty working copy. And people do this all the time. One of the most common cases is committing a submodule "update" while accidentally in a dirty working tree, so when you push the commit pointer simply doesn't exist anywhere. They also fail immediately once you try to `git merge` (or less commonly, git cherry-pick), because their merge conflicts, by definition, cannot be resolved automatically. Because they are pointers, not content. But git works on content. So just to be clear, two of the most fundamental day-to-day operations you can perform are turned into massive liabilities from this feature, ones that are likely to either break your build and/or just make your life harder. As someone who had to maintain stable and development branches of a project, cherry pick between the two, cut releases, etc, submodules are simply hell, because they make an already difficult job worse. This is a good sign that they are a liability. In a past life we actually had so many people push invalid submodule updates over time that we eventually wrote a git hook on our server to reject all commits with submodule updates that didn't exist in the corresponding repository, and that were not specifically tagged in the commit message as updating a submodule (through a magic set of keywords.) The fact we even had to do this is its own pain. I have maintained projects that have long-standing histories with dozens of submodules. And every single time we eliminated one of those submodules (often by merging into the parent repository, or simply dropping the dependency entirely), we all breathed a sigh of relief, and our lives all got significantly better from that point forward. As you can tell, this experience has made me very prepared to fight against submodules everywhere I might see or encounter them. But trust me: it's for your sake, not mine; 'cause there ain't a chance in hell anyone is adding any to my repositories.

tedmiston 4y ago
It sounds like you are arguing in favor of vendored dependencies over submodules?
- aseipp 4y ago
  If I had to choose between the two? Probably, yeah. But it depends on what's being vendored, I guess. Like, I might not want to vendor the entire transitive dependency tree of some app if it's not security critical. But I would probably vendor the tree of an OSS repository I patched extensively to meet my needs, yes. Depending on the environment some other solutions might be possible. It's a complex topic. I will admit that if your dependency in this case is something that changes extremely rarely, and will only see updates maybe like, bi-annually, submodules are "OK." Not great, but they'll work, and presumably won't inflict massive psychic damage on your team members. But it's once they receive any foot traffic by more-than-one-person that all the real pain begins.

olvy0 4y ago

That's strange this just trended on HN. My team is about to decide on the design of a new feature for our internal product, which can essentially be done by either using sub-modules or re-implementing them. Our product has a somewhat simplistic git interface (behind the scenes it's anything but) and I've tried to keep it so, however lately customers have started demanding we also support submodules. The problem is that we use git to mirror a hierarchical database, so using submodules means mirroring another hierarchy inside our hierarchy. This would mean changing the current assumptions in the code to ignore things in the sub-hierarchy except for the sub-sub-hierarchy we care about. Yeah this is hand-wavy but the design constraints I've had are kinda hard to explain. Also, sub-modules would require changing all our git calls to take submodules into account, including cloning, reset, branches etc. And I've read many people's bad experience with sub-modules, including the ones in this sub-thread and so now I'm afraid they might hurt our maintainability in the long run. I've thought of a couple of ways to do this without sub-modules, including using git subtree, but all of them have drawbacks. I've actually found a neat way to merge a sub-directory of another remote repo to the current repo - which means we wouldn't have to change any of the existing code. It involves only "standard" git commands, basically only checkout, reset, and merge (without "exotic" commands like subtree, read-tree, and whatnot). And using "exclude" to keep only the content of a single subdirectory of the remote. But it does require us to maintain a file that's exactly like .gitsubmodules to keep track of remotes. And that's the thing which git submodules does for us "for free". Also I've developed a bad state for bespoke solutions and NIH. I already fear I have contributed more than enough NIH to my company by developing the existing solution, but given the conditions I think it was the only logical solution (a previous bespoke solution failed and was cancelled). But the longer I read and experiment with submodules it looks like they are also a kind of bespoke solution around basic git, and essentially require changing the way you handle operations such as reset, checkout, etc. Training all our users to fix errors due to out-of-sync submodules looks like a nightmare, when they already have problems with the current solution and with git in general. So I'm really conflicted on what our current path should be.

tedmiston 4y ago
I seem to be one of the few in the comments that actually likes submodules. But the description of this use case / requirement sounds complicated / confusing to me ("we use git to mirror a hierarchical database, so using submodules means mirroring another hierarchy inside our hierarchy"). I don't know what your product is but I hope for your sanity a PoC is possible.