4 ms·

The big .git directory is probably binary revisions? Is there any good way around that in git?

by maaaats 13y ago

rickyc091 13y ago
Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.
- masklinn 13y ago
  > Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.
  - Dylan16807 13y ago
    Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.
    - masklinn 13y ago
      > Is that all actually code? Of course not, it's an actual project. There is code, there are data files (text and binary) and there are assets.
Tobu 13y ago
Stripping the binaries with filter-branch, or normalising them using smudge filters (for example, making sure zip-based formats use no compression).
spoiler 13y ago
I'm a bit confused. Whenever I've used git on my projects, I'd make sure the binaries were excluded, using .gitignore Don't other people do that, too? What's the benefit of having binaries stored? I've never needed that; I've never worked on any huge projects, so I might be missing something crucial.
- maaaats 13y ago
  Well, it depends. Images are for instance binaries where a text diff makes little sense, so you have a copy of each version of the image ever used. And many projects use programs where the files are binaries. For instance, I've been on a project where Flash were used and the files checked in. Or PhotoShop PSD files, .ai files etc.
  - spoiler 13y ago
    I usually keep a separate repo for PSD/AI files (completely neglected to consider them as binary files). As for images, icons, fonts and similar, I just have a build script that copies/generates them, if it's needed. I guess I've always been a little bit "obsessed" about the tidiness of my repositories.
- zhemao 13y ago
  If you don't have the source that produced those binaries your only choice is to have them downloadable from somewhere else (which is a real hassle for the developer) or just check them into the repo.
- smsm42 13y ago
  If there is a small number of rarely changing binaries (like icons, tool configs, etc.) then it may not be worth it to move them. Also if space is much cheaper than tool complexity and build time.
heavenlyhash 13y ago
When I can't avoid referring to binary blobs in git, I put them in a separate repo and link them with submodules. It keeps the main repo trim and the whole thing fast while still giving me end-to-end integrity guarantees. I wrote https://github.com/polydawn/mdm/ https://github.com/polydawn/mdm/ to help with this. It goes a step further and puts each binary version in a separate root of history, which means you can pull down only what you need.
namdnay 13y ago
you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful
- heavenlyhash 13y ago
  git is "the stupid content tracker", not "the stupid merge tool". Even for things you have no intention of branching or merging, it still gives you control over versioning... and there's a huge gap between the level of control a hash tree gives you versus trusting some remote server to consistently give you the same file when you give it the same request.
namdnay 13y ago
you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful
namdnay 13y ago
maven!
syntheticlife 13y ago
FWIW: Every time you modify a file in git, it adds a completely new copy of the file. 100mb text file, 100mb binary file - makes no difference. Modify one line, it's a new 100mb entry in your git repo.
- JelteF 13y ago
  No, git has delta compression. It only saves the changes.
  - syntheticlife 13y ago
    Only when git repack operations run.
    - Skinney 13y ago
      which you can run at any time. Git also runs a minor gc every time you do a git push, which does do some compression.

The big .git directory is probably binary revisions? Is there any good way around that in git?

by maaaats 13y ago

The big .git directory is probably binary revisions? Is there any good way around that in git?

Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.

masklinn 13y ago
> Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.
- Dylan16807 13y ago
  Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.
  - masklinn 13y ago
    > Is that all actually code? Of course not, it's an actual project. There is code, there are data files (text and binary) and there are assets.

Tobu 13y ago

Stripping the binaries with filter-branch, or normalising them using smudge filters (for example, making sure zip-based formats use no compression).

spoiler 13y ago

I'm a bit confused. Whenever I've used git on my projects, I'd make sure the binaries were excluded, using .gitignore Don't other people do that, too? What's the benefit of having binaries stored? I've never needed that; I've never worked on any huge projects, so I might be missing something crucial.

maaaats 13y ago
Well, it depends. Images are for instance binaries where a text diff makes little sense, so you have a copy of each version of the image ever used. And many projects use programs where the files are binaries. For instance, I've been on a project where Flash were used and the files checked in. Or PhotoShop PSD files, .ai files etc.
- spoiler 13y ago
  I usually keep a separate repo for PSD/AI files (completely neglected to consider them as binary files). As for images, icons, fonts and similar, I just have a build script that copies/generates them, if it's needed. I guess I've always been a little bit "obsessed" about the tidiness of my repositories.
zhemao 13y ago
If you don't have the source that produced those binaries your only choice is to have them downloadable from somewhere else (which is a real hassle for the developer) or just check them into the repo.
smsm42 13y ago
If there is a small number of rarely changing binaries (like icons, tool configs, etc.) then it may not be worth it to move them. Also if space is much cheaper than tool complexity and build time.

heavenlyhash 13y ago

When I can't avoid referring to binary blobs in git, I put them in a separate repo and link them with submodules. It keeps the main repo trim and the whole thing fast while still giving me end-to-end integrity guarantees. I wrote https://github.com/polydawn/mdm/ https://github.com/polydawn/mdm/ to help with this. It goes a step further and puts each binary version in a separate root of history, which means you can pull down only what you need.

namdnay 13y ago

you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful

heavenlyhash 13y ago
git is "the stupid content tracker", not "the stupid merge tool". Even for things you have no intention of branching or merging, it still gives you control over versioning... and there's a huge gap between the level of control a hash tree gives you versus trusting some remote server to consistently give you the same file when you give it the same request.

namdnay 13y ago

maven!

syntheticlife 13y ago

FWIW: Every time you modify a file in git, it adds a completely new copy of the file. 100mb text file, 100mb binary file - makes no difference. Modify one line, it's a new 100mb entry in your git repo.

JelteF 13y ago
No, git has delta compression. It only saves the changes.
- syntheticlife 13y ago
  Only when git repack operations run.
  - Skinney 13y ago
    which you can run at any time. Git also runs a minor gc every time you do a git push, which does do some compression.