4 ms·
How do people manage hundreds of Git repositories?
I have worked with several large companies using git as their version control system. One hosts several hundred git repositories in bitbucket stash, the other has around a hundred repositories in github.
We recently ran into an issue with joda-time where upgrading past java8u60 caused apps to fail on startup. So my team is tasked with getting all of our microservices working with the latest jdk.
What tools are there to modify and commit similar changes across 100+ repositories at once?
I have found gitwalk https://www.npmjs.com/package/gitwalk which does a subset of what I want focused on github. But google has not revealed many other tools that can do discovery and transformations across 100s of repositories.
- tedmiston 9y agoIt probably depends on exactly what and how complex your “transformation” is. You definitely can do lots of automation with $ git itself in terms of committing, merging, opening PRs on GitHub via $ hub, etc. Another idea is if you can refactor the common bits into a shared standard library, for instance the design pattern is something like a system with a bunch of plugins, then most of the plugin repos become very tiny. Ideally you’d also have some sort of infrastructure automation around deployment.
- Sevii 9y agoWe have shared libraries and maven plugins setup. This is a situation where I really wish we used a monorepo and I could just do one commit across 100 microservices using grep and other tools to update pom.xml files. I have read whitepapers from google on how they support this kind of operation, but I need it for many git repos.
- ynezz 9y agoThis sounds like StackOverflow question. Anyway, if you're proficient in Java, why don't you write small app which will iterate over all Git repo dirs, modifies the files, makes commit, push and creates PRs? Not a big deal, right?
- Sevii 9y agoThat is the current plan, combined with plenty of manual work verifying jenkins jobs and deployments.
- assafmo 9y agoIf you can do it if there was just one repo, then juat clone those 100 repos under the same parent directory and execute the script in the parent directory
- assafmo 9y agoand then in a loop commit the changes
- tenken 9y agoHave you looked at a tool like: https://github.com/mixu/gr https://github.com/mixu/gr
- Sevii 9y agoGoing to look at it, thanks.
- codeonfire 9y agoGrep + sed or replace in Files for the changes. And gitall to commit them all. There is no need to get fancy if simple tools will do the job.
- sethish 9y agoI haven't found anything off-the-shelf that helps. I manage a project with > 50k repositories in a GitHub organization. For the past few years I've been developing software that helps with my [particular goals](https://github.com/gitenberg-dev/gitberg https://github.com/gitenberg-dev/gitberg), but I'm afraid there isn't much there that may help you.
- ericfrederich 9y agoThis looks like a one-off thing you need to do semi-manually and methodically. Every step of the way you need to balance the effort of doing something with a script vs doing it manually. How long would it take to manually make the change in all 100+ repos... vs. how long would it take to write a script to do it. Would the script be robust enough to properly make the change in like 90% of them... etc. These are things you need to ask yourself in your endeavor. You'd want to do all this work on a new similarly named branch (or branches depending on your workflow) for every repo. Perhaps write a script to validate changes to all 100+ repos on that branch. Then when all the work is done... do a massive merge.
- frou_dh 9y ago`mr` is good for keeping track of, and working with, many repos in bulk. http://myrepos.branchable.com/ http://myrepos.branchable.com/ IIRC it has a good man page explaining more advanced use.
- zeveb 9y agoIt's too late for you, but I think that it's useful for other folks to see your example and realise why multiple repos are a bad idea. In the general case, each organisation should have a single repo containing all of its code. There are times to violate this rule, but unless you're in extraordinary circumstances, more than a dozen repos in an organisation is an extremely bad mistake.
- allwein 9y agoCan you elaborate on this a bit? What are the advantages to keeping everything in a single repo for the organization vs. a different repo per project?
- zeveb 9y agoBasically, having multiple repos means that one person can change a dependency of someone else's code without fixing the dependent code. This is unavoidable with independent organisations, but is easily avoidable within an organisation which uses unit & integration testing within a single repo (the use of a single repo means that a suite of changes may be applied atomically). In general, a process which allows one person in an organisation to stymie another's progress is a broken system.
- kadirt 9y agoYour argumentation feels very flawed. Having multiple repos is a good thing. You have an abstraction, you only get relevant branches/tags for each project. The code commits and pulls faster. You can add project based permissions. Hooks can automatically test each new branch. And maybe you do want keep some of the projects with dependencies on old sdk's. The management required to keep it working sounds horrible