6 ms·

The file size limit of git is a bit annoying. I've moved to svn for a few projects that required large binaries to be shared and version controlled along with t

The file size limit of git is a bit annoying. I've moved to svn for a few projects that required large binaries to be shared and version controlled along with their source (for hardware design projects) and hit the 100Mbyte (IIRC) file limit. Im sure it could be fixed, but it is a limitation that has prevented me from using git in some cases

profunctor 6y ago
Git lfs is what you want.
- amelius 6y ago
  The problem with this software is that it uses a different channel to transfer the large binaries. If you're using git-shell or git-daemon to host your git repositories on a server, then you can't use that alone to also manage transfer/access of stuff that is managed through git lfs.
- andromeduck 6y ago
  Broken on Windows.
- _ph_ 6y ago
  Why isn't is integrated into Git out of the box? Why do I as a user have to be aware of a separate large file support? Why do I have to install and use a separate tool for this?
- emerged 6y ago
  Everyone immediately says that when this topic comes up, but afaik it doesn’t even transfer or store files as deltas. Each time you change 2 bytes in a multiple GB file, you get to transfer and store an entire duplicate. IMO that is not a genuine solution to large files.
  - NineStarPoint 6y ago
    In fairness, that’s because gitlfs’s primary purpose is to store binaries where changes in the representation don’t actually map to anything useful. There definitely is a separate use case out there for storing large files where only deliberate changes are made to the objects.
    - plorkyeran 6y ago
      Transferring only deltas is a performance optimization, not a question of semantics. Git-lfs needs to be able to perform binary diffs to save you from uploading 2 GB every time you touch a 2 GB file, not so that it can display useful diffs.
      - emerged 6y ago
        The tech to detect binary diffs has existed for a very long time as well. Since the previous revision is present on disk, there isn’t any good reason why they can’t at least delta compress the network transfer and validate hash on the other end. But beyond that, what’s keeping the server from storing these deltas instead of full versions? Only argument I can think of is that server side corruption of a single file would break the chain of deltas and you’d lose every child. But surely that can be solved using some sort of redundancy.
notretarded 6y ago
Put them in an artefact repository and pull them down at build time?
bullen 6y ago
I agree; cvs, svn and git work ok and don't need that much change, what needs improvement is separation between code and media! I'm making a simple .zip over HTTP solution for my MMO now. It just has a naming scheme with dates and you send your last .zip date to the server and the server sends a list for you to download. The good part with dates is that you can consolidate the data retroactively without breaking the system!

profunctor 6y ago

Git lfs is what you want.

amelius 6y ago
The problem with this software is that it uses a different channel to transfer the large binaries. If you're using git-shell or git-daemon to host your git repositories on a server, then you can't use that alone to also manage transfer/access of stuff that is managed through git lfs.
andromeduck 6y ago
Broken on Windows.
_ph_ 6y ago
Why isn't is integrated into Git out of the box? Why do I as a user have to be aware of a separate large file support? Why do I have to install and use a separate tool for this?
emerged 6y ago
Everyone immediately says that when this topic comes up, but afaik it doesn’t even transfer or store files as deltas. Each time you change 2 bytes in a multiple GB file, you get to transfer and store an entire duplicate. IMO that is not a genuine solution to large files.
- NineStarPoint 6y ago
  In fairness, that’s because gitlfs’s primary purpose is to store binaries where changes in the representation don’t actually map to anything useful. There definitely is a separate use case out there for storing large files where only deliberate changes are made to the objects.
  - plorkyeran 6y ago
    Transferring only deltas is a performance optimization, not a question of semantics. Git-lfs needs to be able to perform binary diffs to save you from uploading 2 GB every time you touch a 2 GB file, not so that it can display useful diffs.
    - emerged 6y ago
      The tech to detect binary diffs has existed for a very long time as well. Since the previous revision is present on disk, there isn’t any good reason why they can’t at least delta compress the network transfer and validate hash on the other end. But beyond that, what’s keeping the server from storing these deltas instead of full versions? Only argument I can think of is that server side corruption of a single file would break the chain of deltas and you’d lose every child. But surely that can be solved using some sort of redundancy.

notretarded 6y ago

Put them in an artefact repository and pull them down at build time?

bullen 6y ago

I agree; cvs, svn and git work ok and don't need that much change, what needs improvement is separation between code and media! I'm making a simple .zip over HTTP solution for my MMO now. It just has a naming scheme with dates and you send your last .zip date to the server and the server sends a list for you to download. The good part with dates is that you can consolidate the data retroactively without breaking the system!