46 ms·

A Git catastrophe cleaned up

274 points by asymmetric 10y ago

soft_dev_person 10y ago
Why not just revert the offending commit? It would be a valid blip in history as mistake made and corrected.
- OJFord 10y ago
  > It occurred to me while I was writing this that it would probably have worked to make one commit on master to remove the back-end files again, and then rebase the entire topic branch onto that commit. But I didn't think of it at the time. And it's not as good as what I did do, which left the history as clean as was possible at that point.
  - barrkel 10y ago
    He could have made the reversion commit on a local branch (branched from master), then rebased interactively to reorder / squash commits. It presupposes that the commits are clean and well-separated though.
  - soft_dev_person 10y ago
    I read that, but it doesn't read like he is aware of the option to revert a commit directly. More that he would remove them manually and commit that. Little difference, but if "as clean as possible" was the goal, I think that would be the cleanest. [edit] added the little words that my keyboard seems to filter out.
    - OJFord 10y ago
      Maybe you're right; I agree that's probably what I would have done. Apart from anything else it allows a clean-slate "this is how it _should_ have been handled in the first place", rather than a separate git-fu move that was necessary or even relevant only because of the original indiscretion.
- rwmj 10y ago
  Another thing is not to fear the non-fast-forward commit. Sure you'll need to tell the other developers (apparently "a hundred" in this case, but more often just a handful of people in the same room) that they have to reset back a few commits before pulling, but that's not the end of the world.
  - bluehex 10y ago
    Not only that but the failure case when a developer pulls before receiving the memo isn't that bad. Git tells you that the branch was force updated and working out what to do from there is usually pretty easy.
- latortuga 10y ago
  Admittedly I did not read the whole post, but I agree that this sounds like the right answer. He made a mistake on master by merging a partial topic branch. The answer is to revert the mistake (git revert abc123) and then either rebase+merge or just merge the whole topic branch. I also don't agree with his conclusion that his way left the history in a better state - reverting the commit actually represents the mistake that was made.
- OskarS 10y ago
  Yeah, that was my instinct as well. The problem was that the first user did that kooky way of bringing the 18 files over with `git checkout`. That's a terrible idea, both because it lead to the catastrophe here, but also because you lose all history of those changes, they appear brand new in that commit (which makes `git bisect` and `git blame` much less useful). No, just `git revert` that terrible commit, then do a regular sensible merge/rebase. Solves all problems in, like, two commands.
guard-of-terra 10y ago
Two fun things about git: It is deterministic, and it doesn't delete anything (readily). This means you can't really have a catastrophe. Just git reflog your way out.
- tomp 10y ago
  That's a bit of an overstatement. You can't have a catastrophe with stuff already committed (but it's very easy to delete in-progress work).
  - guard-of-terra 10y ago
    Only if you never commit and then do checkout -f / reset --hard. Why would you be doing that? Otherwise it's still in reflog. Anything you ever commited is just there.
    - mnarayan01 10y ago
      Unreachable references will be pruned after 30 days (by default) if/when git-gc gets run (which will occasionally happen automatically by default). Not saying it's a problem, just note that they're not actually there forever.
      - lomnakkus 10y ago
        For reflog it's 90 days. (So anything referenced by the reflog will be retained for that long.)
        mnarayan01 10y ago
        AFAICT, for unreachable entries (e.g. as generated by `git commit --amend`) the default is 30 days as defined by `gc.reflogExpireUnreachable`.
        gergoerdi 10y ago
mcherm 10y ago
The author concludes by saying: >I think I've written before that this profusion of solutions is the sign of a well-designed system. The tools and concepts are powerful, and can be combined in many ways to solve many problems that the designers didn't foresee. I disagree. I consider this to be a failure of Git. The set of different options (normal merge, rebase, filter-branch, etc) is complex and not cleanly orthogonal which makes for a very messy "mental model". Even experienced experts would have difficulty finding the clear, simple way to solve this problem and those less experienced would have little chance of proceeding cleanly. I really wish some tool other than Git had "won" the version-control race; I honestly believe Git to be the worst of the contenders in the most recent generation of version control systems (albeit better than the previous generation in important ways).
- stonecodegump 10y ago
  I agree. I know there is a lot to Git and you can pretty much do everything. But whenever such discussions come up about how many options there are and how amazing it is, I just glaze over and completely fail to concentrate. I don't think I have the brain power to actually master Git. Luckily I work in a 2 man team, so rarely do I need to do anything particularly complex. I have my workflow and I do what I can not to stray from it.
- ipioxu15 10y ago
  Git is certainly the best of the version control systems to date, and I'm happy that it won. Experts have no problems with issues like this. The described failure mode also can occur in any existing version control system --- the user is essentially did cp -f ../mybranch/somefile-new.c somefile.c; hg commit somefile.c The history in mybranch touching somefile.c is now partly useless, since all the patches are already applied. Of course, the branch can still be merged, if the conflicts are resolved --- the conflict resolution is then pretty much doing the above command again.
  - mtrn 10y ago
    > Git is certainly the best of the version control systems to date, and I'm happy that it won. Me too. What I found was that people do not recognize what tool git really is. People do not read the manuals and count on their world knowledge acquired from years of svn or other centralized systems. It's not the best sign, that you have to really dig git to be able to use it properly. But once you understood the core ideas, you can basically derive a lot of the workflows yourself, in a very principled and clear manner. Also: once you get a bit more familiar with the data model, one almost cannot not appreciate its beauty, I believe.
mjd 10y ago
The Reddit discussion of this, though brief, was interesting and to the point. https://www.reddit.com/r/git/comments/5i3mpz/another_git_catastrophe_cleaned_up_story_of_a/ https://www.reddit.com/r/git/comments/5i3mpz/another_git_cat...
- coolgeek 10y ago
  This is the author of the blog post we're all discussing
scarface74 10y ago
I think everyone is overlooking the main point. He said that one guy was only making changes to either the front end or back end code. They should have been two separate repos. one for the front end code and one for the back end code.
- icebraining 10y ago
  That's a huge discussion in itself: https://news.ycombinator.com/item?id=10007654 https://news.ycombinator.com/item?id=10007654
  - scarface74 10y ago
    That was a great discussion. As someone said below, I agree if it is independently deployable, it should be in its own repo. If you need to share code between repos you need to create versioned packages - in the c# world - Nuget.
- mnarayan01 10y ago
  If you break up your repos whenever two random people work on disjoint file sets within a branch, you're going to end up with a lot of repos.
- Raphmedia 10y ago
  Where is the line? Would templates files and view-models be in the front-end repo?
  - Ajedi32 10y ago
    I'd say the line is that, if you can independently deploy just a single part of a repository, that part should probably be in a separate repo. In the article, the author stated that "X decided to merge and deploy just the back-end changes". To me, that suggests that perhaps the backend is an independent component entirely decoupled from the front end. At the very least I think you should avoid making changes to multiple independently deployable parts in the same commit.
msvalkon 10y ago
While an unorthodox merge strategy was used, this is what happens when you hole up in a topic branch for a long time. I bet this would've been easier had they merged smaller commits or PR's to master constantly. If one is afraid of deploying unfinished features, don't make them functional until they are ready. Tie them together once finished. Or did I miss something here?
- sangnoir 10y ago
  I'm also very surprised - I thought it was standard to merge master (target upstream branch) into the topic branch frequently (daily seems reasonable). I know some people do not appreciate "merged <branch> into <other branch>" commits in their history, but that is a small price to pay IMO.
  - kosinus 10y ago
    Or rebase. Don't hide merge fixes in merge commits, keep your changes relevant when read in context of current master. Unless you're working on a topic branch together with another dev, but I find that rarely happens in practice.
    - nerdponx 10y ago
      Even so, if you can both adhere to "pull before you push, and use force-with-lease instead of force" the rebase workflow is possible too. However it's more of an abuse of Git rather than a use of it.
    - msvalkon 10y ago
      I find this to be the optimal wow.
    - nogridbag 10y ago
      Yes I just completed a large refactoring in a feature branch which affected lots of files and the only way to stay sane throughout the process was to constantly rebase my work on top of master (within my feature branch). Once my refactoring was complete I squashed it into a single commit to prepare to merge (or rebase) into master. I don't really think it's useful to keep the history of how I implemented that refactoring. Squashing it into a single commit is far easier to revert then if I merged multiple commits into master.
hellofunk 10y ago
The more I read stuff like this, the more I wonder how many problems would just go away for so many people if they used Mercurial instead.
- qznc 10y ago
  How would this problem go away with Mercurial?
  - hellofunk 10y ago
    It's much harder to shoot yourself in the foot with Mercurial. Which is why so many developers and companies prefer it: https://news.ycombinator.com/item?id=10228399 https://news.ycombinator.com/item?id=10228399 https://news.ycombinator.com/item?id=9465084 https://news.ycombinator.com/item?id=9465084
    - mbrock 10y ago
      Could you explain how this specific situation is better with Mercurial? In nearly every thread about Git I hear vague claims that other systems are much easier, but almost never any specific reason to think so.
      - reitanqild 10y ago
        Seconded. And sorry for the brief comment but that would be very interesting.
- hackerboos 10y ago
  Why is Mercurial better than git? It's my understanding that you can still rewrite history with Mercurial.
  - pjmlp 10y ago
    Most of the Mercurial use cases fit better with the metal model that people coming from SVN and similar, are used to. It is only when one wants to do advance tricks that he/she needs to enter "Git-like" land.
luos 10y ago
So if I get this then he just added the files to master, then tried to work on the topic branch? That really seems weird. I don't think it's git's fault. Also he could have done git merge master --accept-theirs if he really wanted some kind of history but I guess it would be worthless.
forrestthewoods 10y ago
I like Perforce. It may not be perfect. But it's idiot proof. "Days since gitastrophe" is a common phrase. There is no Perforce equivalent. You can't blow your leg off. There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks. Thanks Perforce.
- taesis 10y ago
  In my personal experience, life (or, er, people) tend(s) to find a way to mess up any sufficiently complex, shared resource. In other words, I'm happy that it never breaks for you :3 Re. Perforce made easy, I'd suggest hitting up stack overflow or the p4 forums -- the community is much smaller than git's (i.e. practically the entire open source community vs. game developers), so there are fewer people available to write such blog posts (but enough to complain, evidently). I have a number of company internal-blog posts that explain Perforce to fellow newbs, but they can't be made public due to contracts and etc.
- reitanqild 10y ago
  Sorry for ruining the party (seriously). I like Perforce. It may not be perfect. But it's idiot proof. That's because it is almost impossible to do anything, good or bad with it. ;-) "Days since gitastrophe" is a common phrase. Never heard it before. There is no Perforce equivalent. You can't blow your leg off. The equivalent of a hand saw: you will have a harder time cutting off your leg with a hand saw than with a chain saw. There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks. Or because nobody uses it and those who do keeps the knowledge for themselves;-) Seriously: I used perforce for 18 months and I tried to understand why people loved it but no luck. Git isn't exactly perfect but compared to "checking out" a file before even editing it or not being able to commit without access to the server (IIRC) git is perfect. Oh, and not being able to check out a file that someone else has checked out, so if a dev goes on holiday without "checking in" the files first then you have to hack a bit to get ready for work again. The best argument however for git is that even the perforce guys have integrated with it. Thanks Perforce.
fpig 10y ago
I don't understand the problem here, why didn't he just do a merge and resolve the 18 conflicts by using the version of the file from master? And the problem wasn't in checkout-add-commit, that is a trivial issue, the WTF here is producing 406 new commits in a branch without ever thinking of merging master back into it or rebasing on master, to avoid having a giant merge later.
lmm 10y ago
> The next day he wanted to go ahead and merge the front-end changes, but he found himself in “a bit of a pickle”. The merge didn't go forward cleanly, perhaps because of other changes that had been made to master in the meantime. And trying to rebase the branch onto the new master was a complete failure. Many of those 406 commits included various edits to the 18 back-end files that no longer made sense now that the finished versions of those files were in the master branch he was trying to rebase onto. Can one not instead merge master into the feature branch?
- ep103 10y ago
  ding ding ding. The original dev should have made a new feature branch off master. Then they should have dropped the 18 files in, or if the 18 files had a commit history, cherry picked those commits into the new feature branch. The next day, when he was ready for the rest of the changes, merge those into the feature branch. These devs were trying to be clever, and got bit by it :) KISS
- antocv 10y ago
  I also dont see why he didnt just git revert to the parent commit, the commit before the other guy changed 18 files seemingly randomly. Then rebased featurebranch on that, and thats it. So sure the history would contain the mistake, so what.
  - cyberpunk 10y ago
    I'm with ya; I don't see the evil in just force pushing head back to where it was before though -- do you really need 800 useless commits in the history when you're going to just be adding the code back in sanely later anyway? Reverting them seems a bit more risky to be than just rebuilding it properly -- you'll need to write some scripts to make sure you got all the commits and then the whole dev team is going to end up pushing commits which unrevert your reverts just so you can then rebase the feature branch.. Sometimes push -f is the best option ;)
    - antocv 10y ago
peff 10y ago
The simplest solution is: # try the merge; you'll get conflicts on those files git merge topic # discard the versions from the topic branch; # you know you already merged those changes in # the funny "git checkout commit", so any differences # are due to changes on master. git checkout --ours new-file-{1,18} # now you are free to fix up any real conflicts # and resolve the merge git commit This has the advantage of representing the true history. You had two lines of development (the original topic, and the "squashed" history created for deployment), and the merge shows them coming together and choosing the deployment-side content.
- SonicSoul 10y ago
  hmm this is certainly not simple to me! what's new-file-{1,18} for ?
  - antocv 10y ago
    Its the names of files, I believe. so git checkout --ours {name_of_file1,name_of_file2,and_so_on} The {} is standard bash expansion, you can try it like mkdir hello{one,two,three}, so parent poster went with articles use of new-file as the name of the files that were affected.
- cyberpunk 10y ago
  Yeah; I know it sounds evil and this shouldn't be done but why not just make a new branch from the borked master, force push master back to sanity (pre-massive merge), have everyone else cherry pick and push their commits and then just rebase new branch and fix the conflicts in there as you should and merge normally?
  - btym 10y ago
    Because it's selfish. Your time isn't more valuable than that of the developers already working against that branch (and as stated in the article, there were hundreds). It's your job to merge your changes cleanly.
draw_down 10y ago
I hate this shit. Rebasing always causes conflicts and dealing with them is such a giant pain. I get that in this case the designer really brought the pain on themselves but I wish using git didn't require this sort of surgery periodically, which in my experience it does.
- aninhumer 10y ago
  As opposed to what? Conflicts are an inevitable consequence of multiple people working on the same code at the same time, and git is as good or better than the alternatives for handling that.
  - draw_down 10y ago
    In my experience- as opposed to merging, which just works, but nerds get all persnickety about because they hate merge commits. I understand the concern I suppose, but working with `git rebase` just seems so unnecessarily painful. It's really awful.
    - rplst8 10y ago
      rebase uses the same merge logic as merge. What some people don't get is that if there were three commits pulled down form a remote, and you had three commits, and you had conflicts (different ones) on each of those commits, rebase makes you resolve them one by one so it can literally "replay" those commits on top of the tip of the branch you are rebasing onto. That's why rebase is so cool. It lets you solve each conflict during the merge of each commit separately. Then you can do it interactively and squash it all into one commit. Learn the tools, don't complain.
      - aninhumer 10y ago
        > It lets you solve each conflict during the merge of each commit separately. Which is what they're complaining about, compared to just merging and doing it once. Sometimes resolving smaller chunks at a time can be easier, but almost as often it involves a lot of extra effort creating intermediate states for little benefit, and often they don't even make sense. I much prefer to keep an accurate history, rather than invent one that I think looks neater (the same goes for squashing commits). And honestly I think a lot of what people object to can be mitigated by learning how to use git log properly.
kazinator 10y ago
> X decided to merge and deploy just the back-end changes, and then, once that was done and appeared successful, to merge the remaining front-end changes. > "What should X have done in the first place to avoid the pickle?" 0. (Of course, not develop a 406 patch changeset and then have to pick it apart. Make smaller pushes, frequently.) 1. Create a topic branch right there at the tip where the 406 changes are locally committed. 2. Then use git's interactive rebase to rewrite this branch such that just the back-end commits are picked first, followed by the front end. 3. Make a back-end topic branch from the last back-end commit and test that. If it's cool, master can be rebased to that and pushed to origin/master upstream. 4. Test remaining front-end changes, rebase master to them, push. Also: 3. a) If back-end changes need fixing, fix them on the back-end-topic branch. Then rebase the original topic to the back end topic to pick up these changes "under" it. (I.e. replay the front end over the new back end, and install as new front end).
- sirclueless 10y ago
  What to do with commits that touch both back-end and front-end code in step 2.? This amounts to manual run of git-filter-branch, which is a feature the author was unaware of. Without that tool this task is daunting. If the author is willing to manually rewrite potentially hundreds of commits that touch back-end code, then there are a lot of straightforward solutions. We're looking for something more elegant.
  - kazinator 10y ago
    I have dealt with that sort of thing, and roughly as follows. Having somehow identified these commits, I would mark them "edit" in the interactive rebase workflow. Then when such a commit is applied and the interactive rebase stops on it, I would use "git reset --patch HEAD^" to undo some of its changes in the index (while keeping the working tree the same). Then do a "git commit --amend" to overwrite the commit with pruned one. At that point, all the rejected changes appear as local modifications. Another "git commit -a" creates a new commit out of them; so the original became two. If the commit needs to become three you can just "git commit --patch" twice. Or just reset the whole original commit to HEAD^ and do several "git commit --patch" ops. (Use -c <original sha> to re-use the commit message as a basis for the new commit messages). Typically I will do a rebase pass through the commit stack to do nothing but these splits in that pass. Then when things are separated out, do another interactive rebase to do the re-ordering and squashing.
rurban 10y ago
400 commits not cleanly applying? Not a big deal. I routinely merge 1000-2000 commits and rebase 30 active branches to that also. The solution is git rerere. It stores all the resolved merge resolutions forever, and cp or rb apply cleanly then, without any trouble. Eg https://medium.com/@porteneuve/fix-conflicts-only-once-with-git-rerere-7d116b2cec67#.8pj73vnex https://medium.com/@porteneuve/fix-conflicts-only-once-with-...
gragas 10y ago
>and published the changes to `master` -_-
cousin_it 10y ago
In a corporate environment, I think I prefer a simpler workflow with a plain old centralized VCS and without using any branches at all. As code gets written, each commit goes on the trunk behind a feature flag (which you need anyway). That way each commit can benefit from continuous builds and testing, and other people can notice problems early. Branches would only be used for releases. I've worked like that for years on some pretty big projects, and it never caused complicated problems like in the OP. The only caveat is that you need a strong safety net against breaking the trunk (lots of tests, mandatory code review, etc.)
- deleted 10y ago
  [deleted]
krupan 10y ago
People in the discussion here saying, somewhat blindly it appears to me, that using mercurial would have avoided this mess. I'm a huge mercurial fan and have dealt with some tricky situations similar to this (but never this situation exactly) and I'm not so sure how mercurial would have handled it. The best I can say is that I've never known even the most adventurous mercurial user to use checkout (actually revert in mercurial) on individual files in that way. Is that something git people do more often? Aside from that, it'd be fun to see how mecurial handles this, but I'm not sure from reading the original post if I could exactly reproduce it. Mercurial would let you do the checkout (revert) trick that started it all. I can imagine it causing merge conflicts as described. Mercurial does let you specify how to resolve merge conflicts for the whole merge, or you can tell it not to resolve conflicts at all and then you can run hg resolve on a file-per-file (or glob of files) basis and tell it to pick default (equivalent of master) for the files you want. I didn't quite follow the git way of doing this he described with .gitattributes, but using hg resolve sounds easier (but neither are things a non-expert user of either tool would know). In the end some other solutions were proposed. I would not recommend using checkout (revert in mercurial) either. I don't know of a filter-branch equivalent in mercurial, but that sounds like a cool way to deal with this. In mercurial I probably would have reached for graft (equivalent of git's cherry pick), which isn't very different from git.
marcinkuzminski 10y ago
IMHO it's odd approach to the problem. I'd rather ask the author (who knows best in this case) to split this into two separate parts that can be nicely merged.
isaac_is_goat 10y ago
Why wouldn't they reintegrate the mainline development branch with the new branch if it was so long lived? And/Or have the new code behind a feature flag so you could potentially have it deployed but disabled? So many ways this could have been avoided with some basic forward thinking...
SadWebDeveloper 10y ago
The problem with Git is that everyone is trying to use Git as central repository rather than distributed as if it were SVN, personally i blame GitHub for promoting among the new developers the wrong tool for the job causing all this unnecessary drama. Git is the best version control system if and only if the project has a good leader checking everyone merges before commiting and letting everyone knows who is working on what and what parts will affect.
cyberpunk 10y ago
> But I couldn't think of anything, so I asked Rik Signes. Rik immediately said that X should have used git-filter-branch to separate the 406 commits into two branches, branch A with just the changes to the 18 back-end files and branch B with just the changes to the other files. (The two branches together would have had more than 406 commits, since a commit that changed both back-end and front-end files would be represented in both branches.) Then he would have had no trouble landing branch A on master and, after it was deployed, landing branch B. Well. Okay. That's a technical solution and it'd work, it's probably no less time consuming than going and fixing the code in a new branch and merging cleanly (every time I end up needing to filter branch stuff I have to RTFM on it, and it takes ages) -- this problem is NOT a technical one though; it's a process one. Why are you landing 400 commits in one go? Half of those were on files which then start causing merge conflicts for your team and wasted a huge amount of your time? Use feature flags, fix your conflicts in branch, don't merge anything into master unless it's using the 'merge' button on github/gitlab/gogs/whatever. And really think/discuss/roundtable about how you're introducing features because it sounds like this is running away from you a bit here.. It doesn't need to be this complex, and these kinds of messes can't really be put on the tools -- although git certainly makes it easy to set a lot of things on fire..
- edibleEnergy 10y ago
  > this problem is NOT a technical one though; it's a process one. No, the cause of the problem was a process one, but now it's become a technical problem. That isn't really helpful when someone has already created a git nightmare like what's described here.
  - cyberpunk 10y ago
    Hmm, I wasn't trying to come off that way, although I wasn't trying to be helpful with the specifics of their git nightmare since happily they sorted it out anyway: I was only trying to suggest that the only way you "clean up" from a nightmare such as that is to start looking at what seems to be an obviously fragile way of dealing with their code since that's the real problem.. Unfucking the repo, while mildly technically interesting should really be the smaller of the two lessons learned from such a mess... No?
bowmessage 10y ago
Worth noting that cherry-pick takes commit ID ranges in the form xxxxxx..yyyyyy, it may have simplified the driver. Thanks for the tip re: --keep-redundant-commits!
jdonaldson 10y ago
Reading this was like watching a traffic accident in slow motion. I could hear myself yelling at the author as if he were a student driver: "Use filter-branch!!! Use filter-branch!!! NOOOOOOOO NOT merge union with manual deletes!!!" But... he went and did it anyways. Honestly, reading back through commit logs, you always find the part where the driver runs off the road, plows through a clearly marked gate, runs on a train track for a mile or two, then merges back onto the main street, carrying part of a mailbox and a deer carcass. You can fault git if you want, but it seems like some of these cases just arrive naturally no matter what cvs is used. It would be great to have a "git education" repo that contains situations just like these to work through... sort of a "drivers ed" for managing a repo.
chetanahuja 10y ago
I humbly (re)submit this for your consideration https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/
guomanmin 10y ago
Participate in Atlassian Research My name is Angela and I do research for Atlassian. I’m kicking off a round of discussions with people who use Git tools. Ideally, I’d like to talk to people that sit on a team of 3 or more. If this is you, I would love to talk to you about your experience with <using> Git tools, or just some of the pain points that are keeping you up at night when doing your jobs. We’ll just need 30 mins of your time, and as a token of my thanks to those that participate, I’d like to offer a US$50 Amazon gift voucher.    If you’re interested, just shoot me an email with your availability over the next few weeks and we can set up a time to chat for 30 minutes. Please also include your timezone so we can schedule a suitable time (as I’m located in San Francisco). Hope to talk to you soon! Cheers,   Angela Guo aguo@atlassian.com

274 points by asymmetric 10y ago

soft_dev_person 10y ago

Why not just revert the offending commit? It would be a valid blip in history as mistake made and corrected.

OJFord 10y ago
> It occurred to me while I was writing this that it would probably have worked to make one commit on master to remove the back-end files again, and then rebase the entire topic branch onto that commit. But I didn't think of it at the time. And it's not as good as what I did do, which left the history as clean as was possible at that point.
- barrkel 10y ago
  He could have made the reversion commit on a local branch (branched from master), then rebased interactively to reorder / squash commits. It presupposes that the commits are clean and well-separated though.
- soft_dev_person 10y ago
  I read that, but it doesn't read like he is aware of the option to revert a commit directly. More that he would remove them manually and commit that. Little difference, but if "as clean as possible" was the goal, I think that would be the cleanest. [edit] added the little words that my keyboard seems to filter out.
  - OJFord 10y ago
    Maybe you're right; I agree that's probably what I would have done. Apart from anything else it allows a clean-slate "this is how it _should_ have been handled in the first place", rather than a separate git-fu move that was necessary or even relevant only because of the original indiscretion.
rwmj 10y ago
Another thing is not to fear the non-fast-forward commit. Sure you'll need to tell the other developers (apparently "a hundred" in this case, but more often just a handful of people in the same room) that they have to reset back a few commits before pulling, but that's not the end of the world.
- bluehex 10y ago
  Not only that but the failure case when a developer pulls before receiving the memo isn't that bad. Git tells you that the branch was force updated and working out what to do from there is usually pretty easy.
latortuga 10y ago
Admittedly I did not read the whole post, but I agree that this sounds like the right answer. He made a mistake on master by merging a partial topic branch. The answer is to revert the mistake (git revert abc123) and then either rebase+merge or just merge the whole topic branch. I also don't agree with his conclusion that his way left the history in a better state - reverting the commit actually represents the mistake that was made.
OskarS 10y ago
Yeah, that was my instinct as well. The problem was that the first user did that kooky way of bringing the 18 files over with `git checkout`. That's a terrible idea, both because it lead to the catastrophe here, but also because you lose all history of those changes, they appear brand new in that commit (which makes `git bisect` and `git blame` much less useful). No, just `git revert` that terrible commit, then do a regular sensible merge/rebase. Solves all problems in, like, two commands.

guard-of-terra 10y ago

Two fun things about git: It is deterministic, and it doesn't delete anything (readily). This means you can't really have a catastrophe. Just git reflog your way out.

tomp 10y ago
That's a bit of an overstatement. You can't have a catastrophe with stuff already committed (but it's very easy to delete in-progress work).
- guard-of-terra 10y ago
  Only if you never commit and then do checkout -f / reset --hard. Why would you be doing that? Otherwise it's still in reflog. Anything you ever commited is just there.
  - mnarayan01 10y ago
    Unreachable references will be pruned after 30 days (by default) if/when git-gc gets run (which will occasionally happen automatically by default). Not saying it's a problem, just note that they're not actually there forever.
    - lomnakkus 10y ago
      For reflog it's 90 days. (So anything referenced by the reflog will be retained for that long.)
      - mnarayan01 10y ago
        AFAICT, for unreachable entries (e.g. as generated by `git commit --amend`) the default is 30 days as defined by `gc.reflogExpireUnreachable`.
        gergoerdi 10y ago

mcherm 10y ago

The author concludes by saying: >I think I've written before that this profusion of solutions is the sign of a well-designed system. The tools and concepts are powerful, and can be combined in many ways to solve many problems that the designers didn't foresee. I disagree. I consider this to be a failure of Git. The set of different options (normal merge, rebase, filter-branch, etc) is complex and not cleanly orthogonal which makes for a very messy "mental model". Even experienced experts would have difficulty finding the clear, simple way to solve this problem and those less experienced would have little chance of proceeding cleanly. I really wish some tool other than Git had "won" the version-control race; I honestly believe Git to be the worst of the contenders in the most recent generation of version control systems (albeit better than the previous generation in important ways).

stonecodegump 10y ago
I agree. I know there is a lot to Git and you can pretty much do everything. But whenever such discussions come up about how many options there are and how amazing it is, I just glaze over and completely fail to concentrate. I don't think I have the brain power to actually master Git. Luckily I work in a 2 man team, so rarely do I need to do anything particularly complex. I have my workflow and I do what I can not to stray from it.
ipioxu15 10y ago
Git is certainly the best of the version control systems to date, and I'm happy that it won. Experts have no problems with issues like this. The described failure mode also can occur in any existing version control system --- the user is essentially did cp -f ../mybranch/somefile-new.c somefile.c; hg commit somefile.c The history in mybranch touching somefile.c is now partly useless, since all the patches are already applied. Of course, the branch can still be merged, if the conflicts are resolved --- the conflict resolution is then pretty much doing the above command again.
- mtrn 10y ago
  > Git is certainly the best of the version control systems to date, and I'm happy that it won. Me too. What I found was that people do not recognize what tool git really is. People do not read the manuals and count on their world knowledge acquired from years of svn or other centralized systems. It's not the best sign, that you have to really dig git to be able to use it properly. But once you understood the core ideas, you can basically derive a lot of the workflows yourself, in a very principled and clear manner. Also: once you get a bit more familiar with the data model, one almost cannot not appreciate its beauty, I believe.

mjd 10y ago

The Reddit discussion of this, though brief, was interesting and to the point. https://www.reddit.com/r/git/comments/5i3mpz/another_git_catastrophe_cleaned_up_story_of_a/ https://www.reddit.com/r/git/comments/5i3mpz/another_git_cat...

coolgeek 10y ago
This is the author of the blog post we're all discussing

scarface74 10y ago

I think everyone is overlooking the main point. He said that one guy was only making changes to either the front end or back end code. They should have been two separate repos. one for the front end code and one for the back end code.

icebraining 10y ago
That's a huge discussion in itself: https://news.ycombinator.com/item?id=10007654 https://news.ycombinator.com/item?id=10007654
- scarface74 10y ago
  That was a great discussion. As someone said below, I agree if it is independently deployable, it should be in its own repo. If you need to share code between repos you need to create versioned packages - in the c# world - Nuget.
mnarayan01 10y ago
If you break up your repos whenever two random people work on disjoint file sets within a branch, you're going to end up with a lot of repos.
Raphmedia 10y ago
Where is the line? Would templates files and view-models be in the front-end repo?
- Ajedi32 10y ago
  I'd say the line is that, if you can independently deploy just a single part of a repository, that part should probably be in a separate repo. In the article, the author stated that "X decided to merge and deploy just the back-end changes". To me, that suggests that perhaps the backend is an independent component entirely decoupled from the front end. At the very least I think you should avoid making changes to multiple independently deployable parts in the same commit.

msvalkon 10y ago

While an unorthodox merge strategy was used, this is what happens when you hole up in a topic branch for a long time. I bet this would've been easier had they merged smaller commits or PR's to master constantly. If one is afraid of deploying unfinished features, don't make them functional until they are ready. Tie them together once finished. Or did I miss something here?

sangnoir 10y ago
I'm also very surprised - I thought it was standard to merge master (target upstream branch) into the topic branch frequently (daily seems reasonable). I know some people do not appreciate "merged <branch> into <other branch>" commits in their history, but that is a small price to pay IMO.
- kosinus 10y ago
  Or rebase. Don't hide merge fixes in merge commits, keep your changes relevant when read in context of current master. Unless you're working on a topic branch together with another dev, but I find that rarely happens in practice.
  - nerdponx 10y ago
    Even so, if you can both adhere to "pull before you push, and use force-with-lease instead of force" the rebase workflow is possible too. However it's more of an abuse of Git rather than a use of it.
  - msvalkon 10y ago
    I find this to be the optimal wow.
  - nogridbag 10y ago
    Yes I just completed a large refactoring in a feature branch which affected lots of files and the only way to stay sane throughout the process was to constantly rebase my work on top of master (within my feature branch). Once my refactoring was complete I squashed it into a single commit to prepare to merge (or rebase) into master. I don't really think it's useful to keep the history of how I implemented that refactoring. Squashing it into a single commit is far easier to revert then if I merged multiple commits into master.

hellofunk 10y ago

The more I read stuff like this, the more I wonder how many problems would just go away for so many people if they used Mercurial instead.

qznc 10y ago
How would this problem go away with Mercurial?
- hellofunk 10y ago
  It's much harder to shoot yourself in the foot with Mercurial. Which is why so many developers and companies prefer it: https://news.ycombinator.com/item?id=10228399 https://news.ycombinator.com/item?id=10228399 https://news.ycombinator.com/item?id=9465084 https://news.ycombinator.com/item?id=9465084
  - mbrock 10y ago
    Could you explain how this specific situation is better with Mercurial? In nearly every thread about Git I hear vague claims that other systems are much easier, but almost never any specific reason to think so.
    - reitanqild 10y ago
      Seconded. And sorry for the brief comment but that would be very interesting.
hackerboos 10y ago
Why is Mercurial better than git? It's my understanding that you can still rewrite history with Mercurial.
- pjmlp 10y ago
  Most of the Mercurial use cases fit better with the metal model that people coming from SVN and similar, are used to. It is only when one wants to do advance tricks that he/she needs to enter "Git-like" land.

luos 10y ago

So if I get this then he just added the files to master, then tried to work on the topic branch? That really seems weird. I don't think it's git's fault. Also he could have done git merge master --accept-theirs if he really wanted some kind of history but I guess it would be worthless.

forrestthewoods 10y ago

I like Perforce. It may not be perfect. But it's idiot proof. "Days since gitastrophe" is a common phrase. There is no Perforce equivalent. You can't blow your leg off. There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks. Thanks Perforce.

taesis 10y ago
In my personal experience, life (or, er, people) tend(s) to find a way to mess up any sufficiently complex, shared resource. In other words, I'm happy that it never breaks for you :3 Re. Perforce made easy, I'd suggest hitting up stack overflow or the p4 forums -- the community is much smaller than git's (i.e. practically the entire open source community vs. game developers), so there are fewer people available to write such blog posts (but enough to complain, evidently). I have a number of company internal-blog posts that explain Perforce to fellow newbs, but they can't be made public due to contracts and etc.
reitanqild 10y ago
Sorry for ruining the party (seriously). I like Perforce. It may not be perfect. But it's idiot proof. That's because it is almost impossible to do anything, good or bad with it. ;-) "Days since gitastrophe" is a common phrase. Never heard it before. There is no Perforce equivalent. You can't blow your leg off. The equivalent of a hand saw: you will have a harder time cutting off your leg with a hand saw than with a chain saw. There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks. Or because nobody uses it and those who do keeps the knowledge for themselves;-) Seriously: I used perforce for 18 months and I tried to understand why people loved it but no luck. Git isn't exactly perfect but compared to "checking out" a file before even editing it or not being able to commit without access to the server (IIRC) git is perfect. Oh, and not being able to check out a file that someone else has checked out, so if a dev goes on holiday without "checking in" the files first then you have to hack a bit to get ready for work again. The best argument however for git is that even the perforce guys have integrated with it. Thanks Perforce.

fpig 10y ago

I don't understand the problem here, why didn't he just do a merge and resolve the 18 conflicts by using the version of the file from master? And the problem wasn't in checkout-add-commit, that is a trivial issue, the WTF here is producing 406 new commits in a branch without ever thinking of merging master back into it or rebasing on master, to avoid having a giant merge later.

lmm 10y ago

> The next day he wanted to go ahead and merge the front-end changes, but he found himself in “a bit of a pickle”. The merge didn't go forward cleanly, perhaps because of other changes that had been made to master in the meantime. And trying to rebase the branch onto the new master was a complete failure. Many of those 406 commits included various edits to the 18 back-end files that no longer made sense now that the finished versions of those files were in the master branch he was trying to rebase onto. Can one not instead merge master into the feature branch?

ep103 10y ago
ding ding ding. The original dev should have made a new feature branch off master. Then they should have dropped the 18 files in, or if the 18 files had a commit history, cherry picked those commits into the new feature branch. The next day, when he was ready for the rest of the changes, merge those into the feature branch. These devs were trying to be clever, and got bit by it :) KISS
antocv 10y ago
I also dont see why he didnt just git revert to the parent commit, the commit before the other guy changed 18 files seemingly randomly. Then rebased featurebranch on that, and thats it. So sure the history would contain the mistake, so what.
- cyberpunk 10y ago
  I'm with ya; I don't see the evil in just force pushing head back to where it was before though -- do you really need 800 useless commits in the history when you're going to just be adding the code back in sanely later anyway? Reverting them seems a bit more risky to be than just rebuilding it properly -- you'll need to write some scripts to make sure you got all the commits and then the whole dev team is going to end up pushing commits which unrevert your reverts just so you can then rebase the feature branch.. Sometimes push -f is the best option ;)
  - antocv 10y ago

peff 10y ago

The simplest solution is: # try the merge; you'll get conflicts on those files git merge topic # discard the versions from the topic branch; # you know you already merged those changes in # the funny "git checkout commit", so any differences # are due to changes on master. git checkout --ours new-file-{1,18} # now you are free to fix up any real conflicts # and resolve the merge git commit This has the advantage of representing the true history. You had two lines of development (the original topic, and the "squashed" history created for deployment), and the merge shows them coming together and choosing the deployment-side content.

SonicSoul 10y ago
hmm this is certainly not simple to me! what's new-file-{1,18} for ?
- antocv 10y ago
  Its the names of files, I believe. so git checkout --ours {name_of_file1,name_of_file2,and_so_on} The {} is standard bash expansion, you can try it like mkdir hello{one,two,three}, so parent poster went with articles use of new-file as the name of the files that were affected.
cyberpunk 10y ago
Yeah; I know it sounds evil and this shouldn't be done but why not just make a new branch from the borked master, force push master back to sanity (pre-massive merge), have everyone else cherry pick and push their commits and then just rebase new branch and fix the conflicts in there as you should and merge normally?
- btym 10y ago
  Because it's selfish. Your time isn't more valuable than that of the developers already working against that branch (and as stated in the article, there were hundreds). It's your job to merge your changes cleanly.

draw_down 10y ago

I hate this shit. Rebasing always causes conflicts and dealing with them is such a giant pain. I get that in this case the designer really brought the pain on themselves but I wish using git didn't require this sort of surgery periodically, which in my experience it does.

aninhumer 10y ago
As opposed to what? Conflicts are an inevitable consequence of multiple people working on the same code at the same time, and git is as good or better than the alternatives for handling that.
- draw_down 10y ago
  In my experience- as opposed to merging, which just works, but nerds get all persnickety about because they hate merge commits. I understand the concern I suppose, but working with `git rebase` just seems so unnecessarily painful. It's really awful.
  - rplst8 10y ago
    rebase uses the same merge logic as merge. What some people don't get is that if there were three commits pulled down form a remote, and you had three commits, and you had conflicts (different ones) on each of those commits, rebase makes you resolve them one by one so it can literally "replay" those commits on top of the tip of the branch you are rebasing onto. That's why rebase is so cool. It lets you solve each conflict during the merge of each commit separately. Then you can do it interactively and squash it all into one commit. Learn the tools, don't complain.
    - aninhumer 10y ago
      > It lets you solve each conflict during the merge of each commit separately. Which is what they're complaining about, compared to just merging and doing it once. Sometimes resolving smaller chunks at a time can be easier, but almost as often it involves a lot of extra effort creating intermediate states for little benefit, and often they don't even make sense. I much prefer to keep an accurate history, rather than invent one that I think looks neater (the same goes for squashing commits). And honestly I think a lot of what people object to can be mitigated by learning how to use git log properly.

kazinator 10y ago

> X decided to merge and deploy just the back-end changes, and then, once that was done and appeared successful, to merge the remaining front-end changes. > "What should X have done in the first place to avoid the pickle?" 0. (Of course, not develop a 406 patch changeset and then have to pick it apart. Make smaller pushes, frequently.) 1. Create a topic branch right there at the tip where the 406 changes are locally committed. 2. Then use git's interactive rebase to rewrite this branch such that just the back-end commits are picked first, followed by the front end. 3. Make a back-end topic branch from the last back-end commit and test that. If it's cool, master can be rebased to that and pushed to origin/master upstream. 4. Test remaining front-end changes, rebase master to them, push. Also: 3. a) If back-end changes need fixing, fix them on the back-end-topic branch. Then rebase the original topic to the back end topic to pick up these changes "under" it. (I.e. replay the front end over the new back end, and install as new front end).

sirclueless 10y ago
What to do with commits that touch both back-end and front-end code in step 2.? This amounts to manual run of git-filter-branch, which is a feature the author was unaware of. Without that tool this task is daunting. If the author is willing to manually rewrite potentially hundreds of commits that touch back-end code, then there are a lot of straightforward solutions. We're looking for something more elegant.
- kazinator 10y ago
  I have dealt with that sort of thing, and roughly as follows. Having somehow identified these commits, I would mark them "edit" in the interactive rebase workflow. Then when such a commit is applied and the interactive rebase stops on it, I would use "git reset --patch HEAD^" to undo some of its changes in the index (while keeping the working tree the same). Then do a "git commit --amend" to overwrite the commit with pruned one. At that point, all the rejected changes appear as local modifications. Another "git commit -a" creates a new commit out of them; so the original became two. If the commit needs to become three you can just "git commit --patch" twice. Or just reset the whole original commit to HEAD^ and do several "git commit --patch" ops. (Use -c <original sha> to re-use the commit message as a basis for the new commit messages). Typically I will do a rebase pass through the commit stack to do nothing but these splits in that pass. Then when things are separated out, do another interactive rebase to do the re-ordering and squashing.

rurban 10y ago

400 commits not cleanly applying? Not a big deal. I routinely merge 1000-2000 commits and rebase 30 active branches to that also. The solution is git rerere. It stores all the resolved merge resolutions forever, and cp or rb apply cleanly then, without any trouble. Eg https://medium.com/@porteneuve/fix-conflicts-only-once-with-git-rerere-7d116b2cec67#.8pj73vnex https://medium.com/@porteneuve/fix-conflicts-only-once-with-...

gragas 10y ago

>and published the changes to `master` -_-

cousin_it 10y ago

In a corporate environment, I think I prefer a simpler workflow with a plain old centralized VCS and without using any branches at all. As code gets written, each commit goes on the trunk behind a feature flag (which you need anyway). That way each commit can benefit from continuous builds and testing, and other people can notice problems early. Branches would only be used for releases. I've worked like that for years on some pretty big projects, and it never caused complicated problems like in the OP. The only caveat is that you need a strong safety net against breaking the trunk (lots of tests, mandatory code review, etc.)

deleted 10y ago
[deleted]

krupan 10y ago

People in the discussion here saying, somewhat blindly it appears to me, that using mercurial would have avoided this mess. I'm a huge mercurial fan and have dealt with some tricky situations similar to this (but never this situation exactly) and I'm not so sure how mercurial would have handled it. The best I can say is that I've never known even the most adventurous mercurial user to use checkout (actually revert in mercurial) on individual files in that way. Is that something git people do more often? Aside from that, it'd be fun to see how mecurial handles this, but I'm not sure from reading the original post if I could exactly reproduce it. Mercurial would let you do the checkout (revert) trick that started it all. I can imagine it causing merge conflicts as described. Mercurial does let you specify how to resolve merge conflicts for the whole merge, or you can tell it not to resolve conflicts at all and then you can run hg resolve on a file-per-file (or glob of files) basis and tell it to pick default (equivalent of master) for the files you want. I didn't quite follow the git way of doing this he described with .gitattributes, but using hg resolve sounds easier (but neither are things a non-expert user of either tool would know). In the end some other solutions were proposed. I would not recommend using checkout (revert in mercurial) either. I don't know of a filter-branch equivalent in mercurial, but that sounds like a cool way to deal with this. In mercurial I probably would have reached for graft (equivalent of git's cherry pick), which isn't very different from git.

marcinkuzminski 10y ago

IMHO it's odd approach to the problem. I'd rather ask the author (who knows best in this case) to split this into two separate parts that can be nicely merged.

isaac_is_goat 10y ago

Why wouldn't they reintegrate the mainline development branch with the new branch if it was so long lived? And/Or have the new code behind a feature flag so you could potentially have it deployed but disabled? So many ways this could have been avoided with some basic forward thinking...

SadWebDeveloper 10y ago

The problem with Git is that everyone is trying to use Git as central repository rather than distributed as if it were SVN, personally i blame GitHub for promoting among the new developers the wrong tool for the job causing all this unnecessary drama. Git is the best version control system if and only if the project has a good leader checking everyone merges before commiting and letting everyone knows who is working on what and what parts will affect.

cyberpunk 10y ago

> But I couldn't think of anything, so I asked Rik Signes. Rik immediately said that X should have used git-filter-branch to separate the 406 commits into two branches, branch A with just the changes to the 18 back-end files and branch B with just the changes to the other files. (The two branches together would have had more than 406 commits, since a commit that changed both back-end and front-end files would be represented in both branches.) Then he would have had no trouble landing branch A on master and, after it was deployed, landing branch B. Well. Okay. That's a technical solution and it'd work, it's probably no less time consuming than going and fixing the code in a new branch and merging cleanly (every time I end up needing to filter branch stuff I have to RTFM on it, and it takes ages) -- this problem is NOT a technical one though; it's a process one. Why are you landing 400 commits in one go? Half of those were on files which then start causing merge conflicts for your team and wasted a huge amount of your time? Use feature flags, fix your conflicts in branch, don't merge anything into master unless it's using the 'merge' button on github/gitlab/gogs/whatever. And really think/discuss/roundtable about how you're introducing features because it sounds like this is running away from you a bit here.. It doesn't need to be this complex, and these kinds of messes can't really be put on the tools -- although git certainly makes it easy to set a lot of things on fire..

edibleEnergy 10y ago
> this problem is NOT a technical one though; it's a process one. No, the cause of the problem was a process one, but now it's become a technical problem. That isn't really helpful when someone has already created a git nightmare like what's described here.
- cyberpunk 10y ago
  Hmm, I wasn't trying to come off that way, although I wasn't trying to be helpful with the specifics of their git nightmare since happily they sorted it out anyway: I was only trying to suggest that the only way you "clean up" from a nightmare such as that is to start looking at what seems to be an obviously fragile way of dealing with their code since that's the real problem.. Unfucking the repo, while mildly technically interesting should really be the smaller of the two lessons learned from such a mess... No?

bowmessage 10y ago

Worth noting that cherry-pick takes commit ID ranges in the form xxxxxx..yyyyyy, it may have simplified the driver. Thanks for the tip re: --keep-redundant-commits!

jdonaldson 10y ago

Reading this was like watching a traffic accident in slow motion. I could hear myself yelling at the author as if he were a student driver: "Use filter-branch!!! Use filter-branch!!! NOOOOOOOO NOT merge union with manual deletes!!!" But... he went and did it anyways. Honestly, reading back through commit logs, you always find the part where the driver runs off the road, plows through a clearly marked gate, runs on a train track for a mile or two, then merges back onto the main street, carrying part of a mailbox and a deer carcass. You can fault git if you want, but it seems like some of these cases just arrive naturally no matter what cvs is used. It would be great to have a "git education" repo that contains situations just like these to work through... sort of a "drivers ed" for managing a repo.

chetanahuja 10y ago

I humbly (re)submit this for your consideration https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/

guomanmin 10y ago

Participate in Atlassian Research My name is Angela and I do research for Atlassian. I’m kicking off a round of discussions with people who use Git tools. Ideally, I’d like to talk to people that sit on a team of 3 or more. If this is you, I would love to talk to you about your experience with <using> Git tools, or just some of the pain points that are keeping you up at night when doing your jobs. We’ll just need 30 mins of your time, and as a token of my thanks to those that participate, I’d like to offer a US$50 Amazon gift voucher.    If you’re interested, just shoot me an email with your availability over the next few weeks and we can set up a time to chat for 30 minutes. Please also include your timezone so we can schedule a suitable time (as I’m located in San Francisco). Hope to talk to you soon! Cheers,   Angela Guo aguo@atlassian.com