8 ms·

Rework git core for native submodules

103 points by sdqali 13y ago

niggler 13y ago
"This is going nowhere. You're stuck at making the current submodule system work, not answering my questions, diverting conversation, repeatedly asking the same stupid questions, labelling everything that I say "subjective", and refusing to look at the objective counterpart (aka, the code). It's clear to me that no matter how many more emails I write, you're not going to concede. I'm not interested in wasting any more of my time with this nonsense. I give up." http://thread.gmane.org/gmane.comp.version-control.git/220514/ http://thread.gmane.org/gmane.comp.version-control.git/22051...
- Tobu 13y ago
  Hah, I thought you quoted a maintainer, but this is from the original submitter. Cheeky.
  - tinco 13y ago
    It's not cheeky, it's desparate. He proposes an honest and well thought through idea that he spent a lot of time on, and someone he looks up to just behaves like a complete ass. Junio et al. do nothing but thinking against him, instead of with him. They'd do better just not responding at all.
    - drtse4 13y ago
      If he can't defend his idea among project maintainers it's not worth implementing imho. While the first implementation could have been made in a rush, if this needs to be fixed let's give it some thought this time. While they were clearly not supportive, also clearly this guy is not the right one for this job.
    - jedbrown 13y ago
      I followed the discussion and read it exactly opposite. Ram was putting the cart in front of the horse ("I really need you to start reviewing the code now." see replies [1,2]) and everyone else involved in the discussion wanted to understand the benefits first. Junio was never dismissive of the idea, he just requested a coherent argument of the benefits so that real issues could be discussed. It is understood that submodules are not a smooth workflow in many cases, but Ram's proposed change would be very disruptive and most stated "benefits" of his design are red herrings. [1] http://permalink.gmane.org/gmane.comp.version-control.git/220275 http://permalink.gmane.org/gmane.comp.version-control.git/22... [2] http://permalink.gmane.org/gmane.comp.version-control.git/220299 http://permalink.gmane.org/gmane.comp.version-control.git/22...
      - kelnos 13y ago
        I agree with you on the tone of the conversation, but from my -- admittedly biased -- view, submodules are an abomination, and any serious proposal to come up with an alternative should be welcomed with open arms. Ramkumar may have taken the questions directed at him the wrong way, but IMO the questioner shares equal fault for that. Know (or learn) your audience, and tailor your responses so you you achieve a good outcome. "I'm super frustrated and feel like I've wasted my time so I give up" is not a good outcome, for any of the parties concerned.
- artagnon 13y ago
  To clarify: "I give up" was referring to giving up on the argument in that subthread, not on the idea in general. It requires a lot of hard work and perseverance to get something this disruptive merged; I'm merely taking a break to do more groundwork before coming back with a v2 of the approach. I have utmost respect for Junio, Linus and the others, but realize that they have some negative attributes like all human beings do. Junio can be especially defensive when it comes to something new, although it's not completely without reason. After all, we do have an ultra-stable and well-maintained piece of software because of him.
drtse4 13y ago
This thread is a mess... and i'm not sure statements like this one "'git add' should not go past submodule boundaries. I should not be able to 'git add clayoven/' or 'git add clayoven/LICENSE'" are a good start. Gives a simplified description of what he want to do without going too in-depth about why that path was chosen and starts coding right away.
- tinco 13y ago
  Why would he need to go in-depth about why that path was chosen, isn't it obvious? The workflow he proposes is miles better than how gitmodules is working now.
  - drtse4 13y ago
    Why? Simply to discuss it and evaluate alternatives that could be better. I'm referring to the solution he proposed not to the fact that git modules have a lot of space for improvement. "miles better" considering that we are talking about git modules it's not really that hard to devise.
Tobu 13y ago
I've started the thread on Linus's first reply, and the guy is completely unconvincing. He was after a quick feature improvement (I don't really know what but Linus seemed to) and implemented it, but he gave little thought to the overall design (either his or Git's). Big meh, and I'm normally interested in the evolution of Git.
- kzrdude 13y ago
  Linus' first reply just laid bare that the object format matters much more than the implementation around it. That's how git got as far as it did until now, by using a sound data model.
- tinco 13y ago
  I don't know man, anyone who uses gitmodules knows that they are a pain and really unlike anything else in git. This guy had an idea on how to improve it, introduce a cool new basic object type to git and he even wrote a PoC, I'd say hats off to that. Linus makes a rather unconvincing argument against the system, saying the current system allows for submodules be different for local sites. As if the proposed system would not support that, and as if the current 'dirty submodule' system is a better solution. He's being an absolute moron. And Junio is just being very unproductive, he seems fully incapable of inducing anything from the design Ramkumar proposes and fails to see implications that anyone could see, even though he is a core git guy. And frankly he's being an ass too. What I see is someone enthousiastically trying to fix a core problem of git in an ambitious but well constructed way, and a bunch of old guys just bashing the life out of him. I think he's better off just not asking Juno or Linus for advice and just keep on hacking on his fork. I know I would use it.
  - Tobu 13y ago
    He's not the only one working on this. But he doesn't have the skills to defend his ideas (it might be just communication skills, it might not). As it is he won't be able to make the big revolutionary step the patch was promising. If he had been making his own VCS he wouldn't need this kind of review, but Git is an agreed-upon format and protocol; it is absolutely necessary to start by considering the downsides when core changes will affect a large user base.
qznc 13y ago
He wants to unify submodules and subtrees? Sounds fishy to me, since these are for very completely different use cases. Submodules are for tying project parts together, where you have control over all of them. For example, the clang compiler frontent could submodule the LLVM backend. Both are under the LLVM project, so people usually work on both of them at the same time. They should not be in the same repo, since LLVM also has other users unrelated to clang. Subtrees are for integrating external projects, which are not really under your control, but you probably want to follow upstream developments. Since a subtree includes all the repo data, you can cleanly check out, even if the external origin repository vanishes.
- scribu 13y ago
  `git subtree` seems like the perfect tool to complement `git submodule`. Too bad it's not enabled by default: http://engineeredweb.com/blog/how-to-install-git-subtree/ http://engineeredweb.com/blog/how-to-install-git-subtree/
  - jedbrown 13y ago
    Subtree only needs to be installed by the maintainer that interacts with the submodule's upstream. Everyone else just makes normal commits in the parent repo. They don't even need to know that the subtree has its own upstream (but they likely write better commit messages if they know).
  - dustingetz 13y ago
    parent comment is out of date; git subtree is part of git since roughly git 1.8.
    - Tobu 13y ago
      It's part of git/contrib. Depending on the packaging you still need to enable it manually.
- jedbrown 13y ago
  This is backwards. Subtree import all the data from the sub-project. (There is no way to clone without getting the subtrees because they are a native part of the repository.) You interact with subtree as if you had one project, committing without needing to know that the subtree has its own upstream. You can split out the subtree history and send it upstream. Splitting it out changes the SHA1. You can merge from upstream back into the subtree. Subtree makes the most sense when you have a component that is completely dominated by its parent, but which you want to also release stand-alone. Submodules provide weaker coupling and make the most sense when the submodule has its own healthy upstream and you want to track those versions. It's awkward if all submodule development is happening from within the parent.
alexchamberlain 13y ago
I would like to applaud this guy; he has got insightful and polite answers from Linus.
- k3n 13y ago
  I noticed that too; after getting my popcorn ready, I could find only mild technical disagreements. I'd be honored to have an idea shot down so mercifully by Torvalds.
  - alexchamberlain 13y ago
    As would I...
    - akkartik 13y ago
      I think both of you are being unfair. Find me an example when some newcomer submits a patch and gets flamed by Linus. His flames tend to steer clear of actual code (at least at the start), and of outsiders.
drewcrawford 13y ago
I am not a git maintainer, but as someone interested in improving submodules I can try to summarize the thread. Submodules are difficult to use in practice for a wide variety of reasons. There are serious, complex proposals that have made it into git-contrib to build a "better" submodule, but for various reasons these have produced systems that merely make the tradeoffs in a different way that some people prefer. This is not like any of those proposals. His problem is that "git add" "git diff", etc., don't "understand" submodules. It would be as if ls, cd etc. don't "follow" symlinks, so that you had to navigate to the correct directory yourself before you can use standard unix tools. This is a serious problem, but his solution is essentially "we should use hardlinks instead of symlinks". That is, he wants to take the code that understands submodules out of the individual tools, and pop them in the filesystem somewhere where they are "shared" among more of the tools and don't have to exist in any of them. There are many objections to this proposal. The chief one seems to be that this does not seem to directly address any particular problem. I think Ramkumar perceives that the reason git add/diff/rm don't support submodules is as a metaproblem "it is too hard to add submodule support to arbitrary tool". Whereas the git maintainers are saying "It is possible to add submodule support to arbitrary tool." So that's the initial standoff. Another problem is that this requires a filesystem change, and that is essentially the most stable part of git that breaks incompatibility with other versions. If you read Linus's rants, you know that he generally applies an enormous amount of scrutiny to breaking compatibility. And so from his desk, you would need not just one clear benefit, but an overwhelming number of them, to break the contract like this. But what I suspect is the True Rejection here is that this will pan out like all the proposals before it: to be different, but not strictly better, than the current implementation. To return to the POSIX analogy: we have both symlinks and hardlinks, and which one is better depends on what you are doing, there is no "one true link". If you replace all the symlinks with hardlinks, I think you will run into trouble with the hardlinks too. Finally, it is unfortunate that the flamewar is about the monolithic patch rather than about some of the principles that led to the patch. I think Ramkumar has had (at least) two very good insights: that "git add" and friends should understand submodules a lot better than they do, and also that they should have this understanding by way of consuming some API that understands them rather than incorporating separate code for submodules into every tool. These strike me as a concrete improvement over the existing system, and I wish that the energy that leads to huge unusable patches like this could be redirected into usable ones.
richardwhiuk 13y ago
I'd really like something like this to happen, but I agree that this set of patches isn't likely to get included. Submodules are my biggest gripe with git usage, and what persuades me not to suggest people roll git out more widely. I've seen various strategies to avoid submodules (build scripts that clone sub repos instead is one example alternative) but it'd be much nicer if it there was a One True Way which worked properly.
plorkyeran 13y ago
Mostly unrelated to the topic, but I'm always amused by things like "teach ce_compare_gitlink() about OBJ_LINK". I've never seen any other project that anthropomorphizes the code like that, and I sort of like how it makes the resulting changelog read.
- davvid 13y ago
  I've never seen any other project that anthropomorphizes the code like that Git's SubmittingPatches document says to use an imperative tone in commit messages. That's why it reads the way it does.
lnanek2 13y ago
Sure would be nice. Sometimes I'm working on projects and get sent repos to work on with all the deps missing, because people just cloned the deps into subdirectories and git ignored them or something. Would be much better if they had a .git in every folder like Subversion does nowadays instead of trying to have a special root that includes and ignores certain children.
stormbrew 13y ago
So, I'm curious. In response to Linus' comment that "... .gitmodules was always a bit of a hack, but it's a working hack ...", does anyone who's actually used them really feel that they are indeed a 'working' hack? I find that whenever I interact with a git repo with submodules I spend an inordinate amount of time wrangling them to do things they clearly weren't meant to do. I find that most people I talk to about them have experienced the same. And then I go and do something like try to use bisect in concert with them and I basically want to shoot my computer. Am I missing something?
artagnon 13y ago
Who's who, for those of you just joining in: - Linus is the original author of Git, and he wrote it in April 2005. He doesn't contribute anymore, and is rarely seen on the Git mailing list these days (except when something like this happens). In number of patches, he's #4, after Junio, Jeff, and Shawn. - Junio is the maintainer of the Git project. He took over maintainership of Git a few months after it was originally built, in July 2005. - Jonathan is a very big contributor at #6. He doesn't focus on any one part of the codebase, and contributes to a wide spectrum. - Jens primarily contributes to submodule.c/ git-submodule.sh, the current submodule implementation. Along with Heiko, he's one of the authorities on the current submodule system. - Ram is a small contributor. He started out in Jan 2010 with two GSoC projects: one in 2010, and another in 2011 (neither were in submodules).
comex 13y ago
Just to comment on one of the issues in the thread: not everyone uses a command line editor or even an editor which can be easily invoked from the command line (though I do), so requiring a special command, "git edit-link", to edit some inherently textual data that seems to work perfectly well being stored as a normal text file in the repository, is a little gross.

103 points by sdqali 13y ago

"This is going nowhere. You're stuck at making the current submodule system work, not answering my questions, diverting conversation, repeatedly asking the same stupid questions, labelling everything that I say "subjective", and refusing to look at the objective counterpart (aka, the code). It's clear to me that no matter how many more emails I write, you're not going to concede. I'm not interested in wasting any more of my time with this nonsense. I give up." http://thread.gmane.org/gmane.comp.version-control.git/220514/ http://thread.gmane.org/gmane.comp.version-control.git/22051...

Tobu 13y ago
Hah, I thought you quoted a maintainer, but this is from the original submitter. Cheeky.
- tinco 13y ago
  It's not cheeky, it's desparate. He proposes an honest and well thought through idea that he spent a lot of time on, and someone he looks up to just behaves like a complete ass. Junio et al. do nothing but thinking against him, instead of with him. They'd do better just not responding at all.
  - drtse4 13y ago
    If he can't defend his idea among project maintainers it's not worth implementing imho. While the first implementation could have been made in a rush, if this needs to be fixed let's give it some thought this time. While they were clearly not supportive, also clearly this guy is not the right one for this job.
  - jedbrown 13y ago
    I followed the discussion and read it exactly opposite. Ram was putting the cart in front of the horse ("I really need you to start reviewing the code now." see replies [1,2]) and everyone else involved in the discussion wanted to understand the benefits first. Junio was never dismissive of the idea, he just requested a coherent argument of the benefits so that real issues could be discussed. It is understood that submodules are not a smooth workflow in many cases, but Ram's proposed change would be very disruptive and most stated "benefits" of his design are red herrings. [1] http://permalink.gmane.org/gmane.comp.version-control.git/220275 http://permalink.gmane.org/gmane.comp.version-control.git/22... [2] http://permalink.gmane.org/gmane.comp.version-control.git/220299 http://permalink.gmane.org/gmane.comp.version-control.git/22...
    - kelnos 13y ago
      I agree with you on the tone of the conversation, but from my -- admittedly biased -- view, submodules are an abomination, and any serious proposal to come up with an alternative should be welcomed with open arms. Ramkumar may have taken the questions directed at him the wrong way, but IMO the questioner shares equal fault for that. Know (or learn) your audience, and tailor your responses so you you achieve a good outcome. "I'm super frustrated and feel like I've wasted my time so I give up" is not a good outcome, for any of the parties concerned.
artagnon 13y ago
To clarify: "I give up" was referring to giving up on the argument in that subthread, not on the idea in general. It requires a lot of hard work and perseverance to get something this disruptive merged; I'm merely taking a break to do more groundwork before coming back with a v2 of the approach. I have utmost respect for Junio, Linus and the others, but realize that they have some negative attributes like all human beings do. Junio can be especially defensive when it comes to something new, although it's not completely without reason. After all, we do have an ultra-stable and well-maintained piece of software because of him.

drtse4 13y ago

This thread is a mess... and i'm not sure statements like this one "'git add' should not go past submodule boundaries. I should not be able to 'git add clayoven/' or 'git add clayoven/LICENSE'" are a good start. Gives a simplified description of what he want to do without going too in-depth about why that path was chosen and starts coding right away.

tinco 13y ago
Why would he need to go in-depth about why that path was chosen, isn't it obvious? The workflow he proposes is miles better than how gitmodules is working now.
- drtse4 13y ago
  Why? Simply to discuss it and evaluate alternatives that could be better. I'm referring to the solution he proposed not to the fact that git modules have a lot of space for improvement. "miles better" considering that we are talking about git modules it's not really that hard to devise.

Tobu 13y ago

I've started the thread on Linus's first reply, and the guy is completely unconvincing. He was after a quick feature improvement (I don't really know what but Linus seemed to) and implemented it, but he gave little thought to the overall design (either his or Git's). Big meh, and I'm normally interested in the evolution of Git.

kzrdude 13y ago
Linus' first reply just laid bare that the object format matters much more than the implementation around it. That's how git got as far as it did until now, by using a sound data model.
tinco 13y ago
I don't know man, anyone who uses gitmodules knows that they are a pain and really unlike anything else in git. This guy had an idea on how to improve it, introduce a cool new basic object type to git and he even wrote a PoC, I'd say hats off to that. Linus makes a rather unconvincing argument against the system, saying the current system allows for submodules be different for local sites. As if the proposed system would not support that, and as if the current 'dirty submodule' system is a better solution. He's being an absolute moron. And Junio is just being very unproductive, he seems fully incapable of inducing anything from the design Ramkumar proposes and fails to see implications that anyone could see, even though he is a core git guy. And frankly he's being an ass too. What I see is someone enthousiastically trying to fix a core problem of git in an ambitious but well constructed way, and a bunch of old guys just bashing the life out of him. I think he's better off just not asking Juno or Linus for advice and just keep on hacking on his fork. I know I would use it.
- Tobu 13y ago
  He's not the only one working on this. But he doesn't have the skills to defend his ideas (it might be just communication skills, it might not). As it is he won't be able to make the big revolutionary step the patch was promising. If he had been making his own VCS he wouldn't need this kind of review, but Git is an agreed-upon format and protocol; it is absolutely necessary to start by considering the downsides when core changes will affect a large user base.

qznc 13y ago

He wants to unify submodules and subtrees? Sounds fishy to me, since these are for very completely different use cases. Submodules are for tying project parts together, where you have control over all of them. For example, the clang compiler frontent could submodule the LLVM backend. Both are under the LLVM project, so people usually work on both of them at the same time. They should not be in the same repo, since LLVM also has other users unrelated to clang. Subtrees are for integrating external projects, which are not really under your control, but you probably want to follow upstream developments. Since a subtree includes all the repo data, you can cleanly check out, even if the external origin repository vanishes.

scribu 13y ago
`git subtree` seems like the perfect tool to complement `git submodule`. Too bad it's not enabled by default: http://engineeredweb.com/blog/how-to-install-git-subtree/ http://engineeredweb.com/blog/how-to-install-git-subtree/
- jedbrown 13y ago
  Subtree only needs to be installed by the maintainer that interacts with the submodule's upstream. Everyone else just makes normal commits in the parent repo. They don't even need to know that the subtree has its own upstream (but they likely write better commit messages if they know).
- dustingetz 13y ago
  parent comment is out of date; git subtree is part of git since roughly git 1.8.
  - Tobu 13y ago
    It's part of git/contrib. Depending on the packaging you still need to enable it manually.
jedbrown 13y ago
This is backwards. Subtree import all the data from the sub-project. (There is no way to clone without getting the subtrees because they are a native part of the repository.) You interact with subtree as if you had one project, committing without needing to know that the subtree has its own upstream. You can split out the subtree history and send it upstream. Splitting it out changes the SHA1. You can merge from upstream back into the subtree. Subtree makes the most sense when you have a component that is completely dominated by its parent, but which you want to also release stand-alone. Submodules provide weaker coupling and make the most sense when the submodule has its own healthy upstream and you want to track those versions. It's awkward if all submodule development is happening from within the parent.

alexchamberlain 13y ago

I would like to applaud this guy; he has got insightful and polite answers from Linus.

k3n 13y ago
I noticed that too; after getting my popcorn ready, I could find only mild technical disagreements. I'd be honored to have an idea shot down so mercifully by Torvalds.
- alexchamberlain 13y ago
  As would I...
  - akkartik 13y ago
    I think both of you are being unfair. Find me an example when some newcomer submits a patch and gets flamed by Linus. His flames tend to steer clear of actual code (at least at the start), and of outsiders.

drewcrawford 13y ago

I am not a git maintainer, but as someone interested in improving submodules I can try to summarize the thread. Submodules are difficult to use in practice for a wide variety of reasons. There are serious, complex proposals that have made it into git-contrib to build a "better" submodule, but for various reasons these have produced systems that merely make the tradeoffs in a different way that some people prefer. This is not like any of those proposals. His problem is that "git add" "git diff", etc., don't "understand" submodules. It would be as if ls, cd etc. don't "follow" symlinks, so that you had to navigate to the correct directory yourself before you can use standard unix tools. This is a serious problem, but his solution is essentially "we should use hardlinks instead of symlinks". That is, he wants to take the code that understands submodules out of the individual tools, and pop them in the filesystem somewhere where they are "shared" among more of the tools and don't have to exist in any of them. There are many objections to this proposal. The chief one seems to be that this does not seem to directly address any particular problem. I think Ramkumar perceives that the reason git add/diff/rm don't support submodules is as a metaproblem "it is too hard to add submodule support to arbitrary tool". Whereas the git maintainers are saying "It is possible to add submodule support to arbitrary tool." So that's the initial standoff. Another problem is that this requires a filesystem change, and that is essentially the most stable part of git that breaks incompatibility with other versions. If you read Linus's rants, you know that he generally applies an enormous amount of scrutiny to breaking compatibility. And so from his desk, you would need not just one clear benefit, but an overwhelming number of them, to break the contract like this. But what I suspect is the True Rejection here is that this will pan out like all the proposals before it: to be different, but not strictly better, than the current implementation. To return to the POSIX analogy: we have both symlinks and hardlinks, and which one is better depends on what you are doing, there is no "one true link". If you replace all the symlinks with hardlinks, I think you will run into trouble with the hardlinks too. Finally, it is unfortunate that the flamewar is about the monolithic patch rather than about some of the principles that led to the patch. I think Ramkumar has had (at least) two very good insights: that "git add" and friends should understand submodules a lot better than they do, and also that they should have this understanding by way of consuming some API that understands them rather than incorporating separate code for submodules into every tool. These strike me as a concrete improvement over the existing system, and I wish that the energy that leads to huge unusable patches like this could be redirected into usable ones.

richardwhiuk 13y ago

I'd really like something like this to happen, but I agree that this set of patches isn't likely to get included. Submodules are my biggest gripe with git usage, and what persuades me not to suggest people roll git out more widely. I've seen various strategies to avoid submodules (build scripts that clone sub repos instead is one example alternative) but it'd be much nicer if it there was a One True Way which worked properly.

plorkyeran 13y ago

Mostly unrelated to the topic, but I'm always amused by things like "teach ce_compare_gitlink() about OBJ_LINK". I've never seen any other project that anthropomorphizes the code like that, and I sort of like how it makes the resulting changelog read.

davvid 13y ago
I've never seen any other project that anthropomorphizes the code like that Git's SubmittingPatches document says to use an imperative tone in commit messages. That's why it reads the way it does.

lnanek2 13y ago

Sure would be nice. Sometimes I'm working on projects and get sent repos to work on with all the deps missing, because people just cloned the deps into subdirectories and git ignored them or something. Would be much better if they had a .git in every folder like Subversion does nowadays instead of trying to have a special root that includes and ignores certain children.

stormbrew 13y ago

So, I'm curious. In response to Linus' comment that "... .gitmodules was always a bit of a hack, but it's a working hack ...", does anyone who's actually used them really feel that they are indeed a 'working' hack? I find that whenever I interact with a git repo with submodules I spend an inordinate amount of time wrangling them to do things they clearly weren't meant to do. I find that most people I talk to about them have experienced the same. And then I go and do something like try to use bisect in concert with them and I basically want to shoot my computer. Am I missing something?

artagnon 13y ago

Who's who, for those of you just joining in: - Linus is the original author of Git, and he wrote it in April 2005. He doesn't contribute anymore, and is rarely seen on the Git mailing list these days (except when something like this happens). In number of patches, he's #4, after Junio, Jeff, and Shawn. - Junio is the maintainer of the Git project. He took over maintainership of Git a few months after it was originally built, in July 2005. - Jonathan is a very big contributor at #6. He doesn't focus on any one part of the codebase, and contributes to a wide spectrum. - Jens primarily contributes to submodule.c/ git-submodule.sh, the current submodule implementation. Along with Heiko, he's one of the authorities on the current submodule system. - Ram is a small contributor. He started out in Jan 2010 with two GSoC projects: one in 2010, and another in 2011 (neither were in submodules).

comex 13y ago

Just to comment on one of the issues in the thread: not everyone uses a command line editor or even an editor which can be easily invoked from the command line (though I do), so requiring a special command, "git edit-link", to edit some inherently textual data that seems to work perfectly well being stored as a normal text file in the repository, is a little gross.