4 ms·

Git filter-repo: much easier/faster alternative to filter-branch

60 points by dmart 6y ago

MeteorMarc 6y ago
Git filter-repo is mainly from a single author newren, who also is a significant contributor to git itself: https://github.com/git/git/graphs/contributors https://github.com/git/git/graphs/contributors The repo seems open to PR's though, given the many commits from one-time contributors. [Edited] A question: how would one check the integrity of the resulting git repo (apart from an obvious rsync check of the source files in master)? Say, I would like to create a branch from an older commit and rewrite its history with "git rebase -i"; that could fail on a a repo without full integrity.
tomohawk 6y ago
This looks so much nicer than filter-branch, which is slow beyond belief. We've used bfg (which at least performs in reasonable time) to filter out files containing viruses or security tokens/passwords, but there are some definite limitations.
drothlis 6y ago
> If commit messages refer to other commits by ID (e.g. "this reverts commit 01234567890abcdef", "In commit 0013deadbeef9a..."), those commit messages should be rewritten to refer to the new commit IDs Finally, a tool that does this! I wish git rebase could do it.
fmorel 6y ago
I used this earlier this year to split a monorepo into a submodule + 4 other repos. It made it incredibly easy to rewrite history to move files going to different repos, then remove the history of any files not staying in a particular repo. And it was fast. Also, being able to pass in files containing rewrite rules allowed me to easily do multiple dry-runs of the code split ahead of time so I was able to minimize the code freeze for our team.
acemarke 6y ago
I did something similar a couple years ago. I needed to rewrite an entire repo's history so that I could reformat all the code _and_ modernize all the JS syntax throughout every commit in the repo's history (dating back to 2013). I started with `filter-branch`, but found it way too slow (especially on a corporate-controlled Windows machine, where starting up additional processes seems to have a lot of overhead). I concluded that I needed to run the entire filtering logic in a single process to avoid that overhead. Started writing my own with `pygit2`, but then found a repo called `pylter-branch` which did most of that same "loop through commits and reprocess" work for me - I just had to add a lot of additional logic on top for the specific reprocessing I wanted to do. Ended up being able to reprocess about 15K commits in around 4.5 hours. Given the amount of processing I was doing, that was pretty good. I did an extensive writeup [1] on the problem statement, investigation, and techniques I used if anyone's interested. [0] https://github.com/sergelevin/pylter-branch https://github.com/sergelevin/pylter-branch [1] https://blog.isquaredsoftware.com/2018/11/git-js-history-rewriting/ https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...
- gumby 6y ago
  Why did your group want to do a refacctor like that retrospectively rather than just checking in those changes on a forward going basis?
  - acemarke 6y ago
    Heh. It wasn't even the rest of my team - I'm the only one who cared enough to do anything about it :) Per the post, I wanted to auto-reformat the entire codebase, as well as codemod the JS source to update it to more modern syntax (ES modules instead of AMD, `const/let` instead of `var`, shorthand object literals, etc). But, that would have left most of the codebase history showing that the last commit touching a line was "MARK REFORMATTED ALL THE THINGS". Since I was rewriting the entire repo history to strip out all the junk files that were consuming space, which was going to create an entirely new alternate history anyway, I figured I might as well take advantage of the opportunity to redo all the source as well: https://blog.isquaredsoftware.com/2018/11/git-js-history-rewriting/#the-plan-grows https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...
bobbydreamer 6y ago
Wow these are really interesting projects

60 points by dmart 6y ago

Git filter-repo is mainly from a single author newren, who also is a significant contributor to git itself: https://github.com/git/git/graphs/contributors https://github.com/git/git/graphs/contributors The repo seems open to PR's though, given the many commits from one-time contributors. [Edited] A question: how would one check the integrity of the resulting git repo (apart from an obvious rsync check of the source files in master)? Say, I would like to create a branch from an older commit and rewrite its history with "git rebase -i"; that could fail on a a repo without full integrity.

tomohawk 6y ago

This looks so much nicer than filter-branch, which is slow beyond belief. We've used bfg (which at least performs in reasonable time) to filter out files containing viruses or security tokens/passwords, but there are some definite limitations.

drothlis 6y ago

> If commit messages refer to other commits by ID (e.g. "this reverts commit 01234567890abcdef", "In commit 0013deadbeef9a..."), those commit messages should be rewritten to refer to the new commit IDs Finally, a tool that does this! I wish git rebase could do it.

fmorel 6y ago

I used this earlier this year to split a monorepo into a submodule + 4 other repos. It made it incredibly easy to rewrite history to move files going to different repos, then remove the history of any files not staying in a particular repo. And it was fast. Also, being able to pass in files containing rewrite rules allowed me to easily do multiple dry-runs of the code split ahead of time so I was able to minimize the code freeze for our team.

acemarke 6y ago

I did something similar a couple years ago. I needed to rewrite an entire repo's history so that I could reformat all the code _and_ modernize all the JS syntax throughout every commit in the repo's history (dating back to 2013). I started with `filter-branch`, but found it way too slow (especially on a corporate-controlled Windows machine, where starting up additional processes seems to have a lot of overhead). I concluded that I needed to run the entire filtering logic in a single process to avoid that overhead. Started writing my own with `pygit2`, but then found a repo called `pylter-branch` which did most of that same "loop through commits and reprocess" work for me - I just had to add a lot of additional logic on top for the specific reprocessing I wanted to do. Ended up being able to reprocess about 15K commits in around 4.5 hours. Given the amount of processing I was doing, that was pretty good. I did an extensive writeup [1] on the problem statement, investigation, and techniques I used if anyone's interested. [0] https://github.com/sergelevin/pylter-branch https://github.com/sergelevin/pylter-branch [1] https://blog.isquaredsoftware.com/2018/11/git-js-history-rewriting/ https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...

gumby 6y ago
Why did your group want to do a refacctor like that retrospectively rather than just checking in those changes on a forward going basis?
- acemarke 6y ago
  Heh. It wasn't even the rest of my team - I'm the only one who cared enough to do anything about it :) Per the post, I wanted to auto-reformat the entire codebase, as well as codemod the JS source to update it to more modern syntax (ES modules instead of AMD, `const/let` instead of `var`, shorthand object literals, etc). But, that would have left most of the codebase history showing that the last commit touching a line was "MARK REFORMATTED ALL THE THINGS". Since I was rewriting the entire repo history to strip out all the junk files that were consuming space, which was going to create an entirely new alternate history anyway, I figured I might as well take advantage of the opportunity to redo all the source as well: https://blog.isquaredsoftware.com/2018/11/git-js-history-rewriting/#the-plan-grows https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...

bobbydreamer 6y ago

Wow these are really interesting projects