4 ms·

naive me's first thought would be: Why not use git? Can someone with more experience in this area explain?

by erikb 9y ago

jhoechtl 9y ago
The git toolset is, out of the box, weak on tabular data. Also out of the box large file support requires one central repository. None of that is true for Dat
- icebraining 9y ago
  out of the box large file support requires one central repository. How so? As far as I know, git doesn't treat large files any differently than small ones.
  - watson 9y ago
    I'll try to answer your question, but it's been a while since I last looked at it, so I might get some/most of this wrong - so don't shoot me: As far as I know git isn't good at storing binary data. Git depends on line breaks to be able to diff and make change sets. If you store a binary file in git and make an update to it - even though that update only changed 1 byte, the entire new version of the file is stored again. Dat uses Rabin fingerprinting to intelligently slice binary files into chunks that are less likely to change. That make dat a lot more efficient at storing, versioning, and syncing videos, images, and other large binary files.
    - icebraining 9y ago
      Conceptually, git doesn't use change-sets, each commit is a snapshot of all the files in the current version. For storage and transmission efficiency, though, it can store and send them in packfiles, which use delta compression based on LibXDiff - which uses Rabin's fingerprint algorithm (as well as other algorithm by Joshua P. MacDonald) for binary files.
  - rspeer 9y ago
    GitHub definitely treats large files differently from small files (it rejects them), and a Git repository you can't push to GitHub is very different from a typical Git repository. Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files.
    - icebraining 9y ago
      a Git repository you can't push to GitHub is very different from a typical Git repository. When comparing to Dat, I don't see how; you can't push any Dat repository to Github. Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files. You don't need centralized extensions for that, though. I use git-annex, which is completely P2P.
      - rspeer 9y ago
        I was responding to the claim that "git doesn't treat large files any differently than small ones". I was not saying anything about dat or git-annex. Now it's clear that you are treating your large files differently from small ones. Good luck with git-annex.
        deleted 9y ago
        [deleted]
        icebraining 9y ago
        Right, but the claim was about git and you responded with Github, which is a different thing.
- erikb 9y ago
  So, git is not good, but Dat isn't good either?
rspeer 9y ago
Git is not good for large files. If the suggested solution is to use git-annex, I will say that git-annex is so poor in usability that the majority response to "you can get the data via git-annex" is "oh well, I'll try some other data then".

naive me's first thought would be: Why not use git? Can someone with more experience in this area explain?

by erikb 9y ago

naive me's first thought would be: Why not use git? Can someone with more experience in this area explain?

The git toolset is, out of the box, weak on tabular data. Also out of the box large file support requires one central repository. None of that is true for Dat

icebraining 9y ago
out of the box large file support requires one central repository. How so? As far as I know, git doesn't treat large files any differently than small ones.
- watson 9y ago
  I'll try to answer your question, but it's been a while since I last looked at it, so I might get some/most of this wrong - so don't shoot me: As far as I know git isn't good at storing binary data. Git depends on line breaks to be able to diff and make change sets. If you store a binary file in git and make an update to it - even though that update only changed 1 byte, the entire new version of the file is stored again. Dat uses Rabin fingerprinting to intelligently slice binary files into chunks that are less likely to change. That make dat a lot more efficient at storing, versioning, and syncing videos, images, and other large binary files.
  - icebraining 9y ago
    Conceptually, git doesn't use change-sets, each commit is a snapshot of all the files in the current version. For storage and transmission efficiency, though, it can store and send them in packfiles, which use delta compression based on LibXDiff - which uses Rabin's fingerprint algorithm (as well as other algorithm by Joshua P. MacDonald) for binary files.
- rspeer 9y ago
  GitHub definitely treats large files differently from small files (it rejects them), and a Git repository you can't push to GitHub is very different from a typical Git repository. Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files.
  - icebraining 9y ago
    a Git repository you can't push to GitHub is very different from a typical Git repository. When comparing to Dat, I don't see how; you can't push any Dat repository to Github. Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files. You don't need centralized extensions for that, though. I use git-annex, which is completely P2P.
    - rspeer 9y ago
      I was responding to the claim that "git doesn't treat large files any differently than small ones". I was not saying anything about dat or git-annex. Now it's clear that you are treating your large files differently from small ones. Good luck with git-annex.
      - deleted 9y ago
        [deleted]
      - icebraining 9y ago
        Right, but the claim was about git and you responded with Github, which is a different thing.
erikb 9y ago
So, git is not good, but Dat isn't good either?

rspeer 9y ago

Git is not good for large files. If the suggested solution is to use git-annex, I will say that git-annex is so poor in usability that the majority response to "you can get the data via git-annex" is "oh well, I'll try some other data then".