33 ms·

Thanks for verifying my somewhat out of date guesses about smudge performance! Re the python startup time, this is particularly important for smudge/clean filt

by joeyh 12y ago

Thanks for verifying my somewhat out of date guesses about smudge performance! Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)

caust1c 12y ago
Yep. It's really a problem that needs to be fixed in git proper. I'm surpised that github of all people didn't realize this and/or invest the time to do it right. The one major obvious drawback for it being fixed in git is that it's not backwards compatible with old clients though. Doing it in go is probably a good improvement over the existing solutions of git-media and git-fat but I don't think is the final one. Funny enough, although I had thought this since I started working with git-fat, I only recently admitted it[1]. Perhaps if I had admitted it when I first started work on it then there's a chance they would have seen it! :-P [1]https://github.com/cyaninc/git-fat/issues/41#issuecomment-88695218 https://github.com/cyaninc/git-fat/issues/41#issuecomment-88...
clacke2 12y ago
Ok, that means I don't have to check out git-lfs, git-fat or git-bigstore. My annex is 250k symlinks pointing to 250 GiB of data. It's slow enough as it is.
- joeyh 12y ago
  At 250k files in one branch, you are starting to run into other scalability limits in git too, like the innefficient method it uses to update .git/index (rewriting the whole thing).
  - bicolao 12y ago
    There is the new "split-index" mode to avoid this (see "git update-index" man page). The base index will contains 250k files but .git/index only contains entries you update, should be a lot fewer than 250k files.
    - weakish 12y ago
      It seems that `--split-index` is only available at 'update-index'. Can it be used with `add` or via `git config`?

by joeyh 12y ago

caust1c 12y ago

Yep. It's really a problem that needs to be fixed in git proper. I'm surpised that github of all people didn't realize this and/or invest the time to do it right. The one major obvious drawback for it being fixed in git is that it's not backwards compatible with old clients though. Doing it in go is probably a good improvement over the existing solutions of git-media and git-fat but I don't think is the final one. Funny enough, although I had thought this since I started working with git-fat, I only recently admitted it[1]. Perhaps if I had admitted it when I first started work on it then there's a chance they would have seen it! :-P [1]https://github.com/cyaninc/git-fat/issues/41#issuecomment-88695218 https://github.com/cyaninc/git-fat/issues/41#issuecomment-88...

clacke2 12y ago

Ok, that means I don't have to check out git-lfs, git-fat or git-bigstore. My annex is 250k symlinks pointing to 250 GiB of data. It's slow enough as it is.

joeyh 12y ago
At 250k files in one branch, you are starting to run into other scalability limits in git too, like the innefficient method it uses to update .git/index (rewriting the whole thing).
- bicolao 12y ago
  There is the new "split-index" mode to avoid this (see "git update-index" man page). The base index will contains 250k files but .git/index only contains entries you update, should be a lot fewer than 250k files.
  - weakish 12y ago
    It seems that `--split-index` is only available at 'update-index'. Can it be used with `add` or via `git config`?