4 ms·

Beyond how hard to use they may or may not be, my personal hatred of git submodules is about bypassing your normal dependency management system. See 12 Factors

Beyond how hard to use they may or may not be, my personal hatred of git submodules is about bypassing your normal dependency management system. See 12 Factors on Dependencies[1]. I've not seen many uses of submodules that weren't better served by adding the package from pypi/npm/crates/... [1]: https://12factor.net/dependencies https://12factor.net/dependencies

withinboredom 4y ago
I've been reading a lot of research papers lately, sometimes with POC repos. When compiling the code, it would often fail because dependencies have changed over the years (we're talking about C/C++ code which doesn't have a package manager like you're probably used to). Most of these repos would fail to compile because it expected libraries installed on the system to not have changed in the past 10 years or so. In exactly one of these repos did the author use a git submodule so I was linking directly to the correct version of the library. Granted, those libraries _did not_ do this, so it still failed to compile 7 years later... If everyone used git submodules for deps, you'd end up with a (block) chain (if everyone's commits were signed) for deps.
- FrenchyJiby 4y ago
  That's pretty cool reproducibility indeed, love it. I once worked in C++, and the lack of a (standard?) package manager was killing me for pretty much that reason. "This says it's using curl, but did they compile it with a weird flag, or was their version just old?", and stuff like that. For the case you describe, a submodule is the next best thing to proper package management, and I'm glad those submodules are there to save you! But in themselves, the submodules don't guarantee build instructions, and the same guarantees of exactness can be had by locking dependencies. I feel like your message mostly reinforces my belief in the NEED for package manager for every such language combo, having seen the immense value in other systems, and the extremes that we have to go to when they're missing. I hear the Nix (and Guix) community has done great steps to manage these stray systems into reproducibility, using submodule chains of the sort you describe, but augmented with standardized build info to make it system-wide pkg management.
  - withinboredom 4y ago
    That’s a good point about build instructions, but hopefully they are encoded via a make file. In reality, I only had to make very minor changes to the experimental code to get it to compile, so I really appreciated maintainers keeping backwards compatibility for the most part. Some of this code was over 10 years old, and it still worked… that was pretty impressive since I mostly work with higher level languages where dependencies can change very significantly between versions, or even be abandoned within a decade.
- humanrebar 4y ago
  > we're talking about C/C++ code which doesn't have a package manager like you're probably used to No, there are 3-4 suitable package managers [1] depending on what your bar for an acceptable package manager is. It's just C and C++ engineers have gone so long without one that they don't even think about it. Honestly, we just need to start expecting more (and contributing more) with respect to unpackaged C and C++. [1] vcpkg, Conan, nix, and spack off the top of my head
SahAssar 4y ago
Why would using pypi/npm/crates be better? If I'm already using git (which I will be) then using something else for package management needs to be a huge step up, especially if the other systems you mentioned are language specific and a git solution would probably be language agnostic.
- FrenchyJiby 4y ago
  My assumption here is that your language has already got a dependency management system beyond git, because there's just SO much value in specifying and isolating dependencies and associated build tools, things git doesn't care about (off scope). Just a few off the top of my head: identifying/resolving transitive dependencies and conflicts, separating development-only deps, optional dependencies for extra features, dependency ranges for pinning and exact locking, proxying/caching for offline-ish rebuild, auto-upgrades, license checking (GPL...), CVE flagging... All these are ~standardized per language. If you've got (say) a Python script in git, and submodules seem like a good idea for specifying all your dependencies, instead of going towards a package manager, it feels to me like we're a long way away from sound, practical engineering. For language agnostic, I'm sure the Nix/Guix folks can hook you up with a variant of this, even while using git (maybe even submodules!), but all the value I specified above is still there. I guess I don't mind git, maybe even using submodules, but I hate the substitution of good package management hygiene over "just import the next folder down, which I submoduled now".
gitgud 4y ago
> bypassing your normal dependency management system You can use release tags when using Git submodules, which makes it closer to a "normal dependency management system". It's better than using the commit hash or branch, but still not that great without flexible semver dependencies...
- FrenchyJiby 4y ago
  Indeed! I've enjoyed the middle ground that some package managers provide, where one can point to a git repo's tag/branch/commit hash instead of the upstream package repository of choice. This means if you REALLY wanted to do some vendoring approach with custom code, you don't have to do the git submodule, just use the dependency as usual, it's just fetched from git, and built the normal way. It's not for everybody (I still recommend to point to known packages first), but it's cool to acknowledge the usecase of "I need to get this from git repo" without being savages about it and throwing away our tools.
quickthrower2 4y ago
I interpreted the link as saying don’t make me manually install dependencies. Using git submodules is no worse or better than say npm in that regard.

withinboredom 4y ago

I've been reading a lot of research papers lately, sometimes with POC repos. When compiling the code, it would often fail because dependencies have changed over the years (we're talking about C/C++ code which doesn't have a package manager like you're probably used to). Most of these repos would fail to compile because it expected libraries installed on the system to not have changed in the past 10 years or so. In exactly one of these repos did the author use a git submodule so I was linking directly to the correct version of the library. Granted, those libraries _did not_ do this, so it still failed to compile 7 years later... If everyone used git submodules for deps, you'd end up with a (block) chain (if everyone's commits were signed) for deps.

FrenchyJiby 4y ago
That's pretty cool reproducibility indeed, love it. I once worked in C++, and the lack of a (standard?) package manager was killing me for pretty much that reason. "This says it's using curl, but did they compile it with a weird flag, or was their version just old?", and stuff like that. For the case you describe, a submodule is the next best thing to proper package management, and I'm glad those submodules are there to save you! But in themselves, the submodules don't guarantee build instructions, and the same guarantees of exactness can be had by locking dependencies. I feel like your message mostly reinforces my belief in the NEED for package manager for every such language combo, having seen the immense value in other systems, and the extremes that we have to go to when they're missing. I hear the Nix (and Guix) community has done great steps to manage these stray systems into reproducibility, using submodule chains of the sort you describe, but augmented with standardized build info to make it system-wide pkg management.
- withinboredom 4y ago
  That’s a good point about build instructions, but hopefully they are encoded via a make file. In reality, I only had to make very minor changes to the experimental code to get it to compile, so I really appreciated maintainers keeping backwards compatibility for the most part. Some of this code was over 10 years old, and it still worked… that was pretty impressive since I mostly work with higher level languages where dependencies can change very significantly between versions, or even be abandoned within a decade.
humanrebar 4y ago
> we're talking about C/C++ code which doesn't have a package manager like you're probably used to No, there are 3-4 suitable package managers [1] depending on what your bar for an acceptable package manager is. It's just C and C++ engineers have gone so long without one that they don't even think about it. Honestly, we just need to start expecting more (and contributing more) with respect to unpackaged C and C++. [1] vcpkg, Conan, nix, and spack off the top of my head

SahAssar 4y ago

Why would using pypi/npm/crates be better? If I'm already using git (which I will be) then using something else for package management needs to be a huge step up, especially if the other systems you mentioned are language specific and a git solution would probably be language agnostic.

FrenchyJiby 4y ago
My assumption here is that your language has already got a dependency management system beyond git, because there's just SO much value in specifying and isolating dependencies and associated build tools, things git doesn't care about (off scope). Just a few off the top of my head: identifying/resolving transitive dependencies and conflicts, separating development-only deps, optional dependencies for extra features, dependency ranges for pinning and exact locking, proxying/caching for offline-ish rebuild, auto-upgrades, license checking (GPL...), CVE flagging... All these are ~standardized per language. If you've got (say) a Python script in git, and submodules seem like a good idea for specifying all your dependencies, instead of going towards a package manager, it feels to me like we're a long way away from sound, practical engineering. For language agnostic, I'm sure the Nix/Guix folks can hook you up with a variant of this, even while using git (maybe even submodules!), but all the value I specified above is still there. I guess I don't mind git, maybe even using submodules, but I hate the substitution of good package management hygiene over "just import the next folder down, which I submoduled now".

gitgud 4y ago

> bypassing your normal dependency management system You can use release tags when using Git submodules, which makes it closer to a "normal dependency management system". It's better than using the commit hash or branch, but still not that great without flexible semver dependencies...

FrenchyJiby 4y ago
Indeed! I've enjoyed the middle ground that some package managers provide, where one can point to a git repo's tag/branch/commit hash instead of the upstream package repository of choice. This means if you REALLY wanted to do some vendoring approach with custom code, you don't have to do the git submodule, just use the dependency as usual, it's just fetched from git, and built the normal way. It's not for everybody (I still recommend to point to known packages first), but it's cool to acknowledge the usecase of "I need to get this from git repo" without being savages about it and throwing away our tools.

quickthrower2 4y ago

I interpreted the link as saying don’t make me manually install dependencies. Using git submodules is no worse or better than say npm in that regard.