2 ms·

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.

vlovich123 7y ago
Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?
- war1025 7y ago
  I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val): twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/ install_linux_gcc54
  - pcl 7y ago
    I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?
    - war1025 7y ago
      partial clone is different from sparse checkout. We are using sparse checkout. Partial clone is the one that only pulls down objects that are needed by the store.
- stolee 7y ago
  LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand. Git’s partial clone is a more natural way of achieving the same outcome.
- alexhutcheson 7y ago
  Using sparse-checkout by itself will still download the entire repository and its complete history into .git. If you additionally use the "partial clone" feature, then you can restrict what gets downloaded and stored in .git as well - it will download only the objects that are needed for your selected directories (along with their complete history). On big repositories with long history this might still be too much data, so you might also want to use the "shallow clone" feature (via the --depth flag) to restrict how much history you download.

vlovich123 7y ago

Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

war1025 7y ago
I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree. Edit: wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue" twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val): twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val): twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66: def returnValueOccurred(self): wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/ install_linux_gcc54
- pcl 7y ago
  I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?
  - war1025 7y ago
    partial clone is different from sparse checkout. We are using sparse checkout. Partial clone is the one that only pulls down objects that are needed by the store.
stolee 7y ago
LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand. Git’s partial clone is a more natural way of achieving the same outcome.
alexhutcheson 7y ago
Using sparse-checkout by itself will still download the entire repository and its complete history into .git. If you additionally use the "partial clone" feature, then you can restrict what gets downloaded and stored in .git as well - it will download only the objects that are needed for your selected directories (along with their complete history). On big repositories with long history this might still be too much data, so you might also want to use the "shallow clone" feature (via the --depth flag) to restrict how much history you download.