6 ms·

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from ano

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

mappu 7y ago
> For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.
- monocasa 7y ago
  Sometimes you don't own the other repo.
  - scarejunba 7y ago
    Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.
    - phonebanshee 7y ago
      Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.
      - scarejunba 7y ago
        Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.
    - MayeulC 7y ago
      Well, you are right, but there would still be some advantages to submodules: 1. Check the files hash themselves: while you can definitely put the commit ID in the URL, nothing prevents the remote server (though unlikely if github) to answer with another version of the file (and could even do so selectively for your build server). 2. Simple upgrade path: with submodules, you can just `cd` into them and run `git pull` or `git checkout v11.5.2`, and git itself could inform you that a newer version is available if tracking a branch. I also agree with the contribution aspect, though it is less important in some cases. I take the latest example I have in mind where this could have been useful: For integration into F-Droid, RiotX needed not to include binary artifacts of a library, but the source itself. The source repository is quite big (multiple languages), but the thing of interest is a single java file [1]. They ended up simply copy-pasting the file [2] in their repo, which makes its origin less obvious, and more subject to bit-rot and vulnerabilities. [1]: https://github.com/google/diff-match-patch/blob/master/java/src/name/fraser/neil/plaintext/diff_match_patch.java https://github.com/google/diff-match-patch/blob/master/java/... [2]: https://github.com/vector-im/riotX-android/pull/760 https://github.com/vector-im/riotX-android/pull/760
war1025 7y ago
At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.
- vlovich123 7y ago
  Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?
  - war1025 7y ago
    I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val): twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/ install_linux_gcc54
    - pcl 7y ago
      I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?
krupan 7y ago
I thought one of the big benefits of monorepos was that you didn't mess with submodules anymore?
- potatochup 7y ago
  It still might be needed for external dependencies. the code an organization writes might be in one repo, but if you want to bring in some other library, like libssl (assuming there is no better package manager for your language) submodules are often used

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

mappu 7y ago

> For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.

monocasa 7y ago
Sometimes you don't own the other repo.
- scarejunba 7y ago
  Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.
  - phonebanshee 7y ago
    Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.
    - scarejunba 7y ago
      Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.
  - MayeulC 7y ago
    Well, you are right, but there would still be some advantages to submodules: 1. Check the files hash themselves: while you can definitely put the commit ID in the URL, nothing prevents the remote server (though unlikely if github) to answer with another version of the file (and could even do so selectively for your build server). 2. Simple upgrade path: with submodules, you can just `cd` into them and run `git pull` or `git checkout v11.5.2`, and git itself could inform you that a newer version is available if tracking a branch. I also agree with the contribution aspect, though it is less important in some cases. I take the latest example I have in mind where this could have been useful: For integration into F-Droid, RiotX needed not to include binary artifacts of a library, but the source itself. The source repository is quite big (multiple languages), but the thing of interest is a single java file [1]. They ended up simply copy-pasting the file [2] in their repo, which makes its origin less obvious, and more subject to bit-rot and vulnerabilities. [1]: https://github.com/google/diff-match-patch/blob/master/java/src/name/fraser/neil/plaintext/diff_match_patch.java https://github.com/google/diff-match-patch/blob/master/java/... [2]: https://github.com/vector-im/riotX-android/pull/760 https://github.com/vector-im/riotX-android/pull/760

war1025 7y ago

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.

vlovich123 7y ago
Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?
- war1025 7y ago
  I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val): twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/ install_linux_gcc54
  - pcl 7y ago
    I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?

krupan 7y ago

I thought one of the big benefits of monorepos was that you didn't mess with submodules anymore?

potatochup 7y ago
It still might be needed for external dependencies. the code an organization writes might be in one repo, but if you want to bring in some other library, like libssl (assuming there is no better package manager for your language) submodules are often used