15 ms·
Jenkin developers accidentally do "git push --force" to over 150 repos on github
- nathanb 13y agoAlthough the responsible developer's reaction and attitude are both commendable, one element of his response annoyed me: his continued assertion that he should not have been allowed to do this thing that he did. I think it is a truism that systems which allow users to do interesting and clever things must also allow users to do remarkably stupid and wrong things. Rather than focusing on how to prevent a user from doing a silly thing like this, I think a well-designed tool would easily allow the user to undo the silly thing he just did. Unfortunately, all too often when a Bad Thing happens, the discussion tends to center around how to prevent this Bad Thing from happening again. This is often why those who make mistakes end up being demonized; their mistake has now limited the actions that every other user of the system can make, because the system will be modified to prevent other users from making the same mistake.
- btown 13y agoThat said, sometimes Bad Thing Prevention is something that should have been implemented long ago, and was hitherto unrecognized. My canonical example is increased regulations and requirements for entry when an untrained student was seriously injured in a university machine shop; training is something that can be attained, just as "being the one at the top of the organization with force push access" is something that can be attained. There's no loss of abilities, just a more-optimal mapping of abilities to the people who know how to use them.
- bad_user 13y agoGiven that Git is such an integral part of the workflow, I find unnacceptable the ability to push to the central repo for people that can't handle Git well. Git is distributed, if you're using it like it's SVN you're doing it wrong. And yes, we all make mistakes. But by Git's nature, lo and behold, everybody has a copy.
- ihateloggingin 13y agoThis has nothing to do with git. It has to do with github. Git already prohibits forced pushes if you use 'git init --shared' to create your repo.
- kohlhofer 13y agogreat sentiment.
- finnw 13y agohttp://www.youtube.com/watch?v=-8oH8JHaE00 http://www.youtube.com/watch?v=-8oH8JHaE00
- astral303 13y agoMore systems need to implement undo (and scalable undo). Hiding behind "are you sure? (Y/N)" type of prompts and saying "Well, you really should've checked that thrice before hitting Enter" is not good enough.
- aristidb 13y agoYes! And git already gives all the pieces to make this work. (Specifically the reflog.)
- jlgreco 13y agoBeing able to write custom update hooks also allows you to do a lot of neat things (in this case, an update hook could prevent non-fastforward pushes).
- crbnw00ts 13y agoYou can already do this with git config, but unfortunately it's a giant hammer that affects the entire repo at once. It is of course possible to write a post-receive hook that denies non-fastforward pushes to specific branches, but really, git should include a config option out of the box to do this at a more granular level. A common workflow is the expectation that people should be able force-push to their own topic branches all they like, but never to the default branch (except in extreme circumstances). Being able to easily set this via a config option would be very helpful in keeping working repos more "safe" from this kind of thing.
- jlgreco 13y agoIdeally personal topic branches, not yet ready for world consumption, would be pushed but to a different repo. This would allow the main repo to have a global "No force pushes!" setting, and would prevent branch name collisions (how many developers have a scratch branch sitting around named 'scratch'?)
- 13y ago
- Silhouette 13y agoI think it is a truism that systems which allow users to do interesting and clever things must also allow users to do remarkably stupid and wrong things. Rather than focusing on how to prevent a user from doing a silly thing like this, I think a well-designed tool would easily allow the user to undo the silly thing he just did. Isn't that exactly what a revision control system is for?
- darkarmani 13y ago> Isn't that exactly what a revision control system is for? Unfortunately, some allow you to re-write all history.
- Silhouette 13y agoRight. So if your revision control system -- your failsafe in the event of unpredicted mistakes -- is configured such that you can lose data permanently, I think it is fair to ask whether anything can be changed to improve how that tool operates. This doesn't mean you can't also ask what else might be done to avoid any unfortunate repetition of the same problem situation, of course.
- jrochkind1 13y agoNothing was or could have been lost permanently. The commits were all still there, it's just that none of them were in the commit history for any branches anymore. And the entire history including sufficient info to restore the state prior to the `push -f` is all there in the reflog. It's just a pain in the butt to restore it all, especially if in the meantime people started making new commits on top. But if someone has commit privileges to a repo, they have the ability to mess it up, I don't see any real way around that. The UI of the client side tool(s) that made it so easy to make this mistake can perhaps be blamed.
- eru 13y ago> But if someone has commit privileges to a repo, they have the ability to mess it up, I don't see any real way around that. Force push capabilities make it even easier to mess up than just plain commit priviliges.
- darkarmani 13y ago> his continued assertion that he should not have been allowed to do this thing that he did. Why? How does taking away force permission on repos he doesn't commit to prevent him from doing clever things?
- jwecker 13y agoThe fact that there's no immediate obvious answer is what makes it by definition clever.
- mhurron 13y agoGood access controls prevent people from doing things they shouldn't. This includes: -People that shouldn't have access -People that do have access but can make mistakes or shouldn't have complete, unrestricted access -Processes that shouldn't have access -Processes that do have access going haywire He is right, done properly, this probably shouldn't have been allowed to happen. edit: it's almost readable now
- nathanb 13y agoFrom my reading of his post, the one big failure of the system is that he had an access he didn't think he had. This is sort of like me typing find . | xargs -n0 rm -rf in the root directory and thinking "it's OK because it will only delete files I have permission to delete" but not realizing that I'm root. I could certainly see the argument that the system should have made him more aware of the permissions he had. But it seems like instead he's arguing that he shouldn't have been given those permissions to begin with, which goes back to my point. Because one person was given many permissions and used them irresponsibly, now everyone's default permissions are more restrictive.
- kevrone 13y agoI'm pleased that the reaction in this thread is generally with understanding towards the developer. We've all had facepalm moments like this, and probably not as publicly. But I don't think Bad Thing Prevention is necessarily to be avoided. Undo buttons are sweet, but not everything can be undone. An anecdote: During the first month of my first job in finance I accidentally deleted a few mappings in a database which put the entire company at risk to the tune of about $45million. Complete bone-head move, I was new to the game. It all turned out fine and I got off with a slap on the wrist, but my manager was flogged. When markets are involved, something like that has no undo button. Instead, we put some proper controls in place (tests, mostly) to make sure said mappings were always logically consistent. As far as I know, that brand of problem has not occurred in the 6 years since.
- munificent 13y ago> I think a well-designed tool would easily allow the user to undo the silly thing he just did. Agreed. From a usability perspective, undo is always best. Especially undo that the user: 1) knows about, 2) trusts, and 3) is fast. That encourages exploration and lets users fix mistakes. > their mistake has now limited the actions that every other user of the system can make, because the system will be modified to prevent other users from making the same mistake. I don't think this is a black and white issue. It's not about denying all users the ability to express X ever again. I think it's more about having systems that can tell if X is an unusual or heavyweight action and say, "Hey, you're about to do X, which impacts a lot of stuff and you've never done before. Are you sure that's what you mean?" It's velvet rope permissions, not a locked door.
- gcb0 13y agothat's why a have a easy to type password to unlock my local private key, and i set my keymanager to never cache the password. It is always nice to have that last chance to review your changes to the VCS. ...not that it would have helped in this case as he was probably using it heedlessly in a jenkins plugin.
- icebraining 13y agoI don't think this is a black and white issue. It's not about denying all users the ability to express X ever again. I think it's more about having systems that can tell if X is an unusual or heavyweight action and say, "Hey, you're about to do X, which impacts a lot of stuff and you've never done before. Are you sure that's what you mean?" Yes, but git already does that, by having the user type "--force". Asking would eliminate the possibility of scripting the action.
- davvolun 13y agoThat's not entirely accurate, e.g. "git push --force --yes" would still be perfectly scriptable.
- bereft_orange 13y ago
- Havoc 13y agoWell I think one needs both permission control and easy undo options. His insistence that he should not have had right access does come across as a lame excuse, but he has a point. e.g. Take the other poster in the thread who was surprised to find that others have access to his project beyond the half a dozen he knew about - that to me indicates that something isn't quite right permissions-wise.
- tinco 13y agoThere is a very simple undo, just force push the branch you just deleted? It's not like force pushing is some terrible destructive thing, it just puts the repo in a state people don't expect.
- grmarcil 13y agoIs version control really an area where the ability to do "interesting and clever" things is desirable?
- lectrick 13y ago> must also allow users to do remarkably stupid and wrong things. These systems can at least put up speed bumps ("are you SURE you want to do this potentially work-imploding operation?") for those sorts of things.
- ihateloggingin 13y agoSpeed bumps such as being required to specify --force?
- ihateloggingin 13y ago> his continued assertion that he should not have been allowed to do this thing that he did. Well, if the repositories were created with "git init --shared" then it wouldn't have been allowed. I think it's a valid position to believe that this should be the default on github, although obviously in this case, there might be blame-shifting motivation to that position. > Rather than focusing on how to prevent a user from doing a silly thing like this, I think a well-designed tool would easily allow the user to undo the silly thing he just did. Git does make it easy for the user to undo this. It provides the reflog for that. It also provides information in the output of the push command. (However, github does not allow users to access the remote reflog.) In any case, you can blame github, and you can blame the developer, but at least we should all be clear that you can't blame git.
- pearjuice 13y agoCan anyone mirror the link? Google Groups AJAX hell is not loading here.
- DevUps 13y agoOhhhhh... that feeling.
- blktiger 13y agoWe've been using the prevent force push hook for Atlassian's Stash on my team to avoid situations like this. If you really need to force push, you can get around it by deleting the branch and re-creating the branch.
- jlgreco 13y agoI think this is the best setup to run with. There really are few cases where a non-fastforward push to a public repo is called for.
- sarreph 13y agoIt's called... Force Push?
- howeyc 13y agoI don't understand, does everyone in the organization have full commit privileges to all repositories?
- jessaustin 13y agoYou don't need a release manager, until you do.
- mercurial 13y agoThat's some facepalm right here. That said, I do appreciate the reaction of the developer, taking responsibility for his mistake, starting remedial actions right away and discussing preventive measures for the future.
- zygomega 13y agoAnd ditto the reaction of the community. No aggressive denunciations or vindictive chat - just calm and helpful trouble shooting. +1 for the culture they have created.
- pekk 13y agoWhat are you supposed to do if jerks join your community? Excommunicate them?
- teraflop 13y agoThere's a talk called "How to Protect Your Open Source Project From Poisonous People" [1] that offers some suggestions. [1] http://www.slideshare.net/vishnu/how-to-protect-yourhow-to-protect-your-open-source-project-from-poisonous-people http://www.slideshare.net/vishnu/how-to-protect-yourhow-to-p...
- neeee 13y agoVideo: http://youtu.be/Q52kFL8zVoM http://youtu.be/Q52kFL8zVoM
- sylvinus 13y agoMessing up with git and apologizing publicly is hardly being a jerk IMHO.
- jacalata 13y agoHe means none of them are being jerks - for instance a community of jerks might have responded to the announcement of a pretty inconvenient mistake by abusing the guy who made the mistake.
- matwood 13y agoSwitch to Bitbucket. It is easy to selectively disable history rewriting and branch deletion on a per branch basis. I've been told that Github can also do this, but you have to email them.
- jordigh 13y agoSpeaking of bitbucket, the Mercurial devs are working on a completely safe way to edit public commits that doesn't require something like force-push: http://mercurial.selenic.com/wiki/ChangesetEvolution http://mercurial.selenic.com/wiki/ChangesetEvolution The lead dev for that project just got hired by Facebook (actually, almost all of the core hg team is getting hired by FB), and the feature itself is like 85% complete. Soon we will have a safe way to collaboratively edit history!
- emn13 13y agoIt's such a shame git's pretty much won the DVCS wars - hg has so many neat features and a much cleaner user interface - but it seems that's just no longer relevant, given the difference in adoption.
- jordigh 13y agoMercurial is down, but not dead. Vive la résistance!
- twic 13y agoI think of Mercurial being to Git as FreeBSD is to Linux, Greece was to imperial Rome, and Britain is to America. Less populous, less dominant, but providing a source of creativity and a sophisticating influence.
- _stephan 13y agoFacebook seems to be switching from Git to HG for performance/scaling reasons: http://www.reddit.com/r/IAmA/comments/1nl9at/i_am_a_member_of_facebooks_hhvm_team_a_c_and_d/ccjubpq http://www.reddit.com/r/IAmA/comments/1nl9at/i_am_a_member_o...
- smoyer 13y agoEveryone needs at least one frowning picture of themselves to post along with an apology. Props to Luca for the no-nonsense apology.
- M4N14C 13y agoSomeone did a git yolo on a major project?
- mey 13y agoThis kind of situation has been one of my concerns about deploying git internally in a large corporate environment. What steps are appropriate to block this kind of update on a central repo? I don't see a human patch manager scaling.
- nonchalance 13y agoThe old-school way (cronjob + tarball) works well. A modern alternative is only allow a bot to write to the central repo. As an example, the rust guys use a bot: https://github.com/mozilla/rust/pull/10403 https://github.com/mozilla/rust/pull/10403
- potch 13y agoIf you set up a git server yourself, you can disable --force pushes globally: http://randyfay.com/content/avoiding-git-disasters-gory-story http://randyfay.com/content/avoiding-git-disasters-gory-stor...
- dubcanada 13y agoAn easy way to do this is to use the fork/pull request part with a build bot to push commits to the actual master once approved. That way nobody has access to the main repo, all they can do is pull request in, and you can do what ever you want to your own fork.
- qznc 13y agoWe have often inexperienced students, who do get push permission to our central repos. So far they all behaved reasonable, however we had one accident. The student used Eclipse with its git integration. He tried to push, but it failed (non-fast-forward). Apparently, Eclipse tries to be helpful and says something like "If you want to push anyways, use the 'force' checkbox". Not knowing what a "force push" is, the student followed Eclipse's advice. Thankfully, the distributed nature of git is very helpful. You can just force push again from another repo to undo the force push.
- andor 13y agoYes! Eclipse's JGit is afwul. I did a project with 9 others inexperienced in Git. In the final crunch phase, they managed to delete commits via force push multiple times a day! JGit also completely messed up the history with merge commits, instead of doing fetch and rebase.
- jessaustin 13y agoChristopher Orr on the list suggested the following, which makes a great deal of sense to me: One thing I sometimes do for repos where I have commit access, but don't want to push, is to clone the repository via HTTP. That way, any accidental attempts at pushing won't use my ssh key, and so GitHub will prompt me for my password.
- encoderer 13y agoYou can also just override the push URL for any remote, then have a separate remote for pushes. So you'd have like "origin" for pulls only, and "pushorigin" for pushes.
- pnathan 13y agoOuch, that hurts. On the plus side, they are looking into process fixing so it doesn't happen again. Not only that, the flameware level is zero (at this point). Major props to the Jenkins/Cloudbees people! Further, because git is a DVCS, so long as someone has a valid recent clone, the pain is much less than, say, a centralized repo being corrupted.
- InAnEmergency 13y agoI wonder if it's related to this: https://groups.google.com/d/msg/jenkinsci-dev/-Yk0UFfSPZc/GzOu5b1AP7QJ https://groups.google.com/d/msg/jenkinsci-dev/-Yk0UFfSPZc/Gz...
- Estragon 13y agoCan the history simply be replayed into the repositories with the erroneous pushes skipped?
- 0x0 13y agoYes, unless others have pushed local commits based on outdated clones in the mean time; then someone will have to rebase and rework (or just blindly merge and repush)
- kohsuke 13y agoMan, this is not how we wanted to be on Hacker News...
- emn13 13y agoI can only imagine. Good luck! At least everyone's commending your community on keeping it's cool and making the best of it; so hey, that looks professional.
- StefanKarpinski 13y agoMaybe a high-profile incident like this will finally convince GitHub to let you disallow force pushing to specific branches in repos – like your project's master branch. I asked about this years ago and iirc was told there were no plans to implement such controls.
- npinguy 13y agoThe issue isn't force-pushing. The issue is pushing to multiple branches. This is a local developer's Git issue, not a Github issue: http://stackoverflow.com/questions/948354/git-push-default-behavior http://stackoverflow.com/questions/948354/git-push-default-b... but one they've thankfully changed: http://stackoverflow.com/questions/13148066/warning-push-default-is-unset-its-implicit-value-is-changing-in-git-2-0 http://stackoverflow.com/questions/13148066/warning-push-def... Summary: By default Git push used to (they've changed it) push ALL your branches, not just your current one. It is pretty much the first thing I've historically reconfigured whenever starting on any new project. This guy didn't do it, and paid the cost.
- aferreira 13y agoAlways always _always_ use git push <remote> <branch>. Always. Did I mention always? :)
- cjh_ 13y agoAfter my first bad force push (silly git push default on an unfamiliar machine) I now 'enforce' the `git push remote from_branch:to_branch` syntax on myself when force pushing, I also have adopted the habit of always accompanying any --force with a --dry-run.
- markeganfuller 13y agoMaybe that should be the default. It does a dry run and if you're sure you add --do or something.
- filipedeschamps 13y agoFunny fact: Jenkins is all about continuous integration.
- yannk 13y agoIt would be nice if Github was giving a UI control over force-push behavior. If you open a ticket they'll manually flip this repository settings on or off for you, but still... Maybe this is the wake up call: it's dangerous and doesn't scale?
- Groxx 13y agoI predict this will happen a LOT more often with git 2.0 changing the default push behavior to pushing all branches. If you've ever --force pushed over a temporary remote branch (e.g. to show something to a coworker), 2.0 will run the risk of overriding everything. Hooray!
- 0x0 13y agoIsn't the new git2 default to only push the currently checked out branch?
- tacticus 13y agoyes but only if there is a same named branch on the remote end. i kinda prefer the upstream (names can be different but i set up an explicit link) option to simple
- Axsuul 13y agoDon't you mean it will happen less often? Git default push behavior in 2.0 will be 'simple'.
- jrochkind1 13y agoNobody's yet commented on... what is it you can do to force push to ~180 different git repos at once? Normally, it would be difficult to do this even if you tried, no? 180 repos would usually be in a 180 different clone directories; what can you do that somehow force pushes them all at once, even ones you didn't mean to be working on at all? In the thread, the person who made the mistake seems to suggest[1] it has something to do with a 'gerrit replication plug-in'? I am not familiar with gerrit. Anyone? Mistakes will happen; but for a mistake to have this level of consequence took amplification, apparently by the 'gerrit replication plugin'? I'd be inclined to blame that software, rather than human error (or github permissions or lack thereof! If you have write access to a bunch of repos, of course you can mess them all up, no github permissions will ultimately change that. The curious thing remains how you manage to mess up a couple hundred of them in one click). [1] https://groups.google.com/d/msg/jenkinsci-dev/-myjRIPcVwU/v8PWC60lNEIJ https://groups.google.com/d/msg/jenkinsci-dev/-myjRIPcVwU/v8...
- thwest 13y ago`git submodule foreach` is capable using git alone.
- deleted 13y ago[deleted]
- StefanKarpinski 13y ago> If you have write access to a bunch of repos, of course you can mess them all up, no github permissions will ultimately change that. This is simply not the case. Force pushing is literally the only operation you can do via git protocols that have a destructive effect on the remote repo. Moreover, it is generally considered best a practice never to force push public branches. Thus, preventing forced pushes on branches that are considered public would, in fact, prevent any sort of real destruction to a repo. People could still push new garbage to it, but they can't make old history inaccessible.
- wereHamster 13y agoDeleting a branch is also a destructive operation last time I checked. Force push could also be implemented by deleting the branch and pushing a different SHA1 into the same name.
- kohsuke 13y agoBy the way, it is "Jenkins", not "Jenkin".
- Aqueous 13y agoI'm not sure if it contributed to this fiasco, but GitHub's team system is a bit backwards and screwed up. You create teams of people, that automatically have either pull, push/pull, or administrative permissions on every repository they touch. Then, when you go to the 'collaborators' section, you simply add those teams of people, without being able to see what access level you are awarding to that specific repository! It makes no sense. Either the permissions should be RIGHT next to the team name in the drop down or (much better) GitHub should decouple groups from the permissions they have, and permissions should be awarded on a per-repository basis. That you way you are granting X group of people Y permissions on Z repository. You are always explicitly granting a permissions level to a specific group of people, and no permissions are automatically added that you haven't explicitly signed off on for that repository. Maybe there's some logic behind it, but I doubt it, and I'm really not sure if GitHub completely thought it through.
- tjdetwiler 13y agoGithubs response: Good news from GitHub: they have extracted the full list of SHA-1 before the forced push ! Many thanks to Nathan Witmer :-) See below the full CSV with the SHA-1. He created as well a branch named 'recovery' that points to the candidate point for restoring the master branch. Hope this will help to sort out the remaining repos. Luca. > Hi Luca. > > Oh man, that sinking feeling! > > But, no worries: I've gone through each of the repos listed above and done the following : > > * retrieved the SHA for the previous `master` before you force-pushed > * created a branch called `recovery` pointing to each former master > > In some cases, these are the same. > > I can go further and reset the master refs to their former shas if you'd like, or you can recover these yourself. To do so, in each repo: > > $ git fetch > $ git checkout master > $ git reset --hard origin/recovery > $ git push --force origin master > > I've attached a csv containing the shas (former master and forced master) for each branch, for your reference. > > Good luck! > > Nathan.
- LandoCalrissian 13y agoThat was incredibly nice of them.
- lightyrs 13y agoGitHub represents a new ideal in customer service. I hope other companies will begin to follow their example.
- gcb0 13y agowhy couldn't they do those steps themselves? ...i'm not really familiar with github's git. I am having a really hard time understanding why a distributed VCS with hundreds of users needs the admin to the central repo to do anything for them.
- cjh_ 13y agoThe central repository is the only point that knows (with 100% certainty) which ref the branch was set to before the force push. Each distributed node only knows the ref the branch was pointing at the last time they fetched, so their certainty on what the branch looked like before the force push is much lower.
- Radim 13y agoAn unfortunate accident. Makes me wonder though -- what is the worst a developer with full git commit access can do? (without having access to the actual server) If someone sets out to mess up a repo ON PURPOSE, he can do force push. Can he also somehow trigger GC to preclude recovery? Do something else? What are the attack vectors?
- pearjuice 13y agoTotal mismanagement of repository access. Surprised this is the first time.
- Osiris 13y agoSometimes I have had to do a git push --force, usually when I want to amend a commit I pushed just a few minutes before and I know that no one else has used it. However, I always do a --dry-run first on a force push just to verify that it's only going to update the one branch I expect it to.
- runn1ng 13y agoCan someone explain to me what exactly happened? I don't really know what Jenkin is and what "git push --force" does.
- parkrrr 13y agoJenkins is build server software. It's existence is related to the story only because it was a Jenkins developer. 'git push --force' pushes your copy of the repository to the remote, regardless of what the remote has. He basically told GitHub, "Hey here's the /real/ copy of the repository, the copy you have is totally wrong" and it overwrote all the history because his copy was a few months out of date.
- runn1ng 13y agoOh. Ovewriting history magic. One part of Git I never really liked.
- Pxtl 13y agoSo when do we do the obvious and start running version control on our version control history? Git the git? Revert our VC mistakes as easily as we revert our code ones?
- circa 13y agoLeroy Jenkins?
- bradbatt 13y agoThat's what I came here to type. Well, actually... Leeeeeeeeeeerooooy Jenkinnnnns!!!! http://www.youtube.com/watch?v=LkCNJRfSZBU http://www.youtube.com/watch?v=LkCNJRfSZBU /goddammitleroy
- Kiro 13y agoWhat does the --force flag do? I don't understand the description at http://git-scm.com/docs/git-push http://git-scm.com/docs/git-push
- ihateloggingin 13y agoIt deletes the remote branch entirely, and then uploads the local branch.
- justinhj 13y agoThis link just opens a google groups page with no content on my iPhone :(
- jonico 13y agoForce pushes are dangerous but turning them off completely prevents some valid use cases like pruning problematic IP or removing large binaries accidentally committed to a git repo. For those reasons, CollabNet ships with a Gerrit plugin that does not prevent history rewrites but records them in an immutable backup refs, sends emails and logs it properly. Recovery can be done on a self service basis and administrators can still delete those backup refs forever. The plugin also treats branch deletions like that. More details can be found on http://luksza.org/2012/cool-git-stuff-from-collabnet-potsdam-team/ http://luksza.org/2012/cool-git-stuff-from-collabnet-potsdam...
- JSno 13y agoshould it be "Jenkins" instead of "Jenkin" in title?
- timmow 13y agoThe number one feature I wish github would add is a setting to prevent force pushing to master, but allow a forced push to all other branches. This would allow us to use a branching model like http://scottchacon.com/2011/08/31/github-flow.html http://scottchacon.com/2011/08/31/github-flow.html but we would still be able to rebase branches after code review, without the danger of rewriting history on the master branch. You can do something similar with client side hooks, but it would be nice if it was supported server side.