5 ms·

Git submodules just don't seem to work in practice, and that could be a big chance for a competing scm to overtake git. Having dependency management inline with

Git submodules just don't seem to work in practice, and that could be a big chance for a competing scm to overtake git. Having dependency management inline with scm would allow for more simplicity in the overall process.

amelius 6y ago
This is also my biggest issue with Git. And integrated/streamlined support for large files.
- wickedOne 6y ago
  what would be a typical large file you want to have in your codebase? usually this shouldn't be a consideration
  - _ph_ 6y ago
    I recently had to deal with a PowerPoint document which is slightly larger than 50 megabytes, which in todays terms, isn't very much. Before, I had kept it in SVN and that has no issues storing larger files. It is a bit shocking that Git has issues with not so tiny files.
  - diffeomorphism 6y ago
    I want to use it for files not just source code. For instance including graphics, the pdfs generated by latex or just anything. Or storing lots of directories. Currently, I use dropbox for that, but if git could do that...
    - wickedOne 6y ago
      though i agree that there isn't a proper vcs out there for large files (adobe bridge was a nice attempt) git wasn't designed for that and one might wonder whether you want git to be _that_ multi purpose.
      - _ph_ 6y ago
        If you have a significant project, you want to store at least a reasonable amount of media with it. Images, documentation. Git doesn't necessarily have to be the best system for handling high-gigabyte sized binaries, but should at least deal gracefully, with small and medium sized binary files. I am also not sure, why not more effort was spent making Git support large files even well.
        amelius 6y ago
        By the way, if you're storing large files under version control in Git it is often useful to use the "--depth=1" flag when cloning or pulling repositories. That way you only download the stuff you really need and leave the rest of the history on the server until you need it.
    - sofixa 6y ago
      Git LFS can do that.
      - war1025 6y ago
        Git LFS is basically the CVS of binary file version control. It is a pile of garbage, but it's better than nothing.
  - amelius 6y ago
    In addition to what others posted here, sometimes it is nice to put generated files under version control with the source code that generated them. For example, simulation results, deep learning models, graphs that you need to include in your LaTeX documents (which can be considered partly source code, partly generated content). Also, deep learning training data often consists of large image files, and can also be considered "source code", and in any case it can be very useful to put these under version control. And finally it can be useful to put external dependencies as tar-files into your source tree.
    - watwatinthewat 6y ago
      I appreciate you laying this out because it's something I have struggled with and thought I just didn't know the right way to handle it. For writing tests in a deep learning code base, rather than simply including a native data file (image, CSV, whatever), I've taken to writing a fake data creator class. It always feels like overkill when an alternative solution is including a native data file or two that already exists.
  - xiphias2 6y ago
    It doesn't matter actually. Tools should be easy to learn and as free of edge cases as possible. An example happening is constant propagation in Rust: it's the same feature, but with every release it can cover more of the code base.
- sofixa 6y ago
  Git LFS has been a thing for years.
  - amelius 6y ago
    Git LFS is not integrated well enough, imho. As I posted elsewhere, this software uses a separate network channel for transferring large files. You can't use it with git-daemon and git-shell, but you'll have to setup another server, configure access control etc. and your repository will be split into different parts of your file system.
  - deleted 6y ago
    [deleted]
  - aesh2Xa1 6y ago
    Git LFS lacks a "host your own server" setup. That's a big downside. Hosting your own revision history should be simple.
    - overgard 6y ago
      I'm doing that with LFS on a gitlab droplet on digital ocean and its trivial (with like a hundred gb repo). Dunno if it was bad in the past but it's not hard now.
  - war1025 6y ago
    If you want a terrible time, have several members of your team not install git lfs, and have them be the main people that commit binary files. LFS uses some sort of internal filtering and tracking to determine which binary files might have changed. It seems to have trouble deciding if there are actually dirty files that need changed. So you can't just say, "Okay, go find all the binary files that didn't actually get moved to LFS and correct them" Instead you end up with random moments where you want to commit a single file and git instead detects 1000 png files that it absolutely could not go on without doing something about. But then the diff is a disaster so good luck understanding that what it is actually mad about is that it wants to move the files into LFS. The only way I finally figured it out was to manually load the object blob and notice one of them was an LFS pointer file. I personally think git annex handles things more elegantly, but lfs won that battle.
- vadishev 6y ago
  This and also path based access (limiting access to certain directories/files within a branch).
wickedOne 6y ago
been using submodules for years without any issues other than the routine before "deinit" was introduced was somewhat tricky. could you elaborate?
- rwmj 6y ago
  I've been using git submodules for years but there are some real problems with them: * Changing from a subdirectory to a submodule breaks lots of things like git reset and git bisect. * Having to remember to git module init, update etc. I always have to look up the commands and never remember what the difference is. * I don't care that there are unlisted files in a submodule, either don't bug me about this in status, or integrate commands in such a way that they work transparently across the main module and submodule. * Related to the previous: Coordinating a single logical change across submodules involves several manual steps and has plenty of scope to go wrong.
junon 6y ago
Submodules are stupid easy, what issues do you have with them?
- ducktective 6y ago
  Once I accidentally added a sub-module to a repo. Had to jump through all kinds of hoops to fix that. - An edit in .gitmodules - An edit in .git/config - A removal in .git/modules (seriously?) https://stackoverflow.com/questions/1260748/how-do-i-remove-a-submodule https://stackoverflow.com/questions/1260748/how-do-i-remove-... https://gist.github.com/myusuf3/7f645819ded92bda6677 https://gist.github.com/myusuf3/7f645819ded92bda6677
  - junon 6y ago
    Deinit didn't work? And that's the only issue you had with them?
vadishev 6y ago
To be honest, I doubt better submodules are gonna help you overtake Git, there's too much inertia in this market. Take for example Mercurial which is a great tool and exists forever, yet it still lost its battle to Git. I'd rather believe that submodules are fixed at some point or an alternative solution appears that works much better. Git subtree is around for a while and there's also Git X-Modules https://gitmodules.com https://gitmodules.com which is modules on the Git server.

amelius 6y ago

This is also my biggest issue with Git. And integrated/streamlined support for large files.

wickedOne 6y ago
what would be a typical large file you want to have in your codebase? usually this shouldn't be a consideration
- _ph_ 6y ago
  I recently had to deal with a PowerPoint document which is slightly larger than 50 megabytes, which in todays terms, isn't very much. Before, I had kept it in SVN and that has no issues storing larger files. It is a bit shocking that Git has issues with not so tiny files.
- diffeomorphism 6y ago
  I want to use it for files not just source code. For instance including graphics, the pdfs generated by latex or just anything. Or storing lots of directories. Currently, I use dropbox for that, but if git could do that...
  - wickedOne 6y ago
    though i agree that there isn't a proper vcs out there for large files (adobe bridge was a nice attempt) git wasn't designed for that and one might wonder whether you want git to be _that_ multi purpose.
    - _ph_ 6y ago
      If you have a significant project, you want to store at least a reasonable amount of media with it. Images, documentation. Git doesn't necessarily have to be the best system for handling high-gigabyte sized binaries, but should at least deal gracefully, with small and medium sized binary files. I am also not sure, why not more effort was spent making Git support large files even well.
      - amelius 6y ago
        By the way, if you're storing large files under version control in Git it is often useful to use the "--depth=1" flag when cloning or pulling repositories. That way you only download the stuff you really need and leave the rest of the history on the server until you need it.
  - sofixa 6y ago
    Git LFS can do that.
    - war1025 6y ago
      Git LFS is basically the CVS of binary file version control. It is a pile of garbage, but it's better than nothing.
- amelius 6y ago
  In addition to what others posted here, sometimes it is nice to put generated files under version control with the source code that generated them. For example, simulation results, deep learning models, graphs that you need to include in your LaTeX documents (which can be considered partly source code, partly generated content). Also, deep learning training data often consists of large image files, and can also be considered "source code", and in any case it can be very useful to put these under version control. And finally it can be useful to put external dependencies as tar-files into your source tree.
  - watwatinthewat 6y ago
    I appreciate you laying this out because it's something I have struggled with and thought I just didn't know the right way to handle it. For writing tests in a deep learning code base, rather than simply including a native data file (image, CSV, whatever), I've taken to writing a fake data creator class. It always feels like overkill when an alternative solution is including a native data file or two that already exists.
- xiphias2 6y ago
  It doesn't matter actually. Tools should be easy to learn and as free of edge cases as possible. An example happening is constant propagation in Rust: it's the same feature, but with every release it can cover more of the code base.
sofixa 6y ago
Git LFS has been a thing for years.
- amelius 6y ago
  Git LFS is not integrated well enough, imho. As I posted elsewhere, this software uses a separate network channel for transferring large files. You can't use it with git-daemon and git-shell, but you'll have to setup another server, configure access control etc. and your repository will be split into different parts of your file system.
- deleted 6y ago
  [deleted]
- aesh2Xa1 6y ago
  Git LFS lacks a "host your own server" setup. That's a big downside. Hosting your own revision history should be simple.
  - overgard 6y ago
    I'm doing that with LFS on a gitlab droplet on digital ocean and its trivial (with like a hundred gb repo). Dunno if it was bad in the past but it's not hard now.
- war1025 6y ago
  If you want a terrible time, have several members of your team not install git lfs, and have them be the main people that commit binary files. LFS uses some sort of internal filtering and tracking to determine which binary files might have changed. It seems to have trouble deciding if there are actually dirty files that need changed. So you can't just say, "Okay, go find all the binary files that didn't actually get moved to LFS and correct them" Instead you end up with random moments where you want to commit a single file and git instead detects 1000 png files that it absolutely could not go on without doing something about. But then the diff is a disaster so good luck understanding that what it is actually mad about is that it wants to move the files into LFS. The only way I finally figured it out was to manually load the object blob and notice one of them was an LFS pointer file. I personally think git annex handles things more elegantly, but lfs won that battle.
vadishev 6y ago
This and also path based access (limiting access to certain directories/files within a branch).

wickedOne 6y ago

been using submodules for years without any issues other than the routine before "deinit" was introduced was somewhat tricky. could you elaborate?

rwmj 6y ago
I've been using git submodules for years but there are some real problems with them: * Changing from a subdirectory to a submodule breaks lots of things like git reset and git bisect. * Having to remember to git module init, update etc. I always have to look up the commands and never remember what the difference is. * I don't care that there are unlisted files in a submodule, either don't bug me about this in status, or integrate commands in such a way that they work transparently across the main module and submodule. * Related to the previous: Coordinating a single logical change across submodules involves several manual steps and has plenty of scope to go wrong.

junon 6y ago

Submodules are stupid easy, what issues do you have with them?

ducktective 6y ago
Once I accidentally added a sub-module to a repo. Had to jump through all kinds of hoops to fix that. - An edit in .gitmodules - An edit in .git/config - A removal in .git/modules (seriously?) https://stackoverflow.com/questions/1260748/how-do-i-remove-a-submodule https://stackoverflow.com/questions/1260748/how-do-i-remove-... https://gist.github.com/myusuf3/7f645819ded92bda6677 https://gist.github.com/myusuf3/7f645819ded92bda6677
- junon 6y ago
  Deinit didn't work? And that's the only issue you had with them?

vadishev 6y ago

To be honest, I doubt better submodules are gonna help you overtake Git, there's too much inertia in this market. Take for example Mercurial which is a great tool and exists forever, yet it still lost its battle to Git. I'd rather believe that submodules are fixed at some point or an alternative solution appears that works much better. Git subtree is around for a while and there's also Git X-Modules https://gitmodules.com https://gitmodules.com which is modules on the Git server.