3 ms·

> Fundamentally any workflow where you have multiple long-lived branches and want to cross-merge between them (e.g. branches for several mostly-independent long

by patrec 6y ago

> Fundamentally any workflow where you have multiple long-lived branches and want to cross-merge between them (e.g. branches for several mostly-independent long-term features, per-client branches) is impossible with rebase I absolutely agree with you, if you have long lived branches (for example if you have several supported versions of a product and you need to apply security backports etc) you should use merges. We both agree that long lived branches are to be avoided if possible, so although I further agree that merge is better if you want to integrate across branches, I place no value on that in the scenarios I normally have to deal with (no inherent necessity for long lived branches). I want feature branches to be short lived and integration to happen (predominantly) via master, because that offers significant benefits. For the cases where I really needed something from A in B, rebasing or cherry-picking has not been much of a hassle for me in practice. My feeling is that the majority of open source or commercial projects should not have long lived branches (although some undoubtedly need them). > Merging rather than rebasing also makes bisect (especially automated bisect) much more effective Not really: there is exactly one method I have found that works well for bisecting robustly, and this applies equally to merge and rebase based commits: you need to to record which commits have passed CI in some way that's trivial to use for bisect skip (e.g. by CI rewriting the head commit's message to indicate it was tested and passed). This is pretty easy to set up and very useful. If you have a merge based workflow you can also use merges commits to a blessed branch (like master) indicate that there are no broken commits (hence my previous question, how do you bisect just on master merges? What I was getting at is that git annoyingly makes it somewhat awkward to say "skip bisect everything that's not a merge into master"). The only other alternative to get robust bisect is to test all commits via CI, but that's typically not practical for anything but small projects. > Why would that ever be what you want, and how does rebase make it any easier? See above: in my experience feature branches that get merged (rather than rebased) will be still be full of intermediate broken commits. If you have set up CI sensibly, your merge commits to master should all be non-broken, so this is a good way to skip unreliable commits. Rebase does not make bisect harder or easier (well, other than that it offers a convenient way to tag CI passes in the commit message), this was just to back up my claim that bisect in git is unnecessarily awkward to use. > if you rebase a branch that has a semantic conflict with master then it's very common for the rewritten commits to not even compile, so you end up with lots of non-compiling commits scattered through your history. I'd really like to understand better how you or (more likely?) your co-workers end up in this situation and why you think it's related to a merge based vs rebase based workflow. Linu[sx] use a merge flow workflow but insists on a lot of rebasing to clean up local history, for example. I assume what's happening is something like this: someone has done work on a branch that lived long enough to accumulate a larg-ish number and also diverge enough from master to cause conflicts. When they then rebase master into their branch, they don't both to fix the problems at the commit they occurred, and instead just make the head of the branch work again. So you have a bunch of intermittent broken commits and they pollute history. Is that it? But you can have the exact same thing happen with a merge based workflow, firstly because you can never prevent someone from (badly) rebasing master into their private local branch and secondly because it is also quite likely that someone who couldn't be asked to rebased cleanly but uses merges exclusively would merge master multiplie times into their branch without making the merge commits necessarily pass tests, instead they'd probably also concentrate their efforts on making the head of the feature branch pass. > But if you're happy with having foo deployed on version X and bar deployed on version Y then I'd say that foo and bar are separate systems and the VCS model should reflect that Kinda. I think the correct way to handle this is make it painful and requite people to duplicate the code in repo. You sometimes need two versions of X in production in parallel for a longer amount of time (e.g. if you are transitioning to a new architecture, and you need to run it as a shadow system for some time first to gather confidence), but it should not be a common thing and I find having both in the working tree works well for these cases. > In the case of an internal library there most likely isn't a single consistent version, because different components will depend on different versions of it That's a big no-no in my book. Maybe there is some scale at which this is the lesser evil because otherwise you make it too painful to refactor stuff, but in general I think you should make it painful for people to do this: all versions should be in master and there should be strong pressure to avoid having more than one version of anything in master, certainly for anything but a short transition periods. There are massive downsides to allowing people to use random versions of internal libraries: security concerns, people ending up on some completely outdated version of a lib that then suddenly breaks completely for them and they don't have the time budget to rewrite all their crap, the massive cost of not being able to look into a single repo and know that if you see "import foolib" it's the foolib in the same source tree. I've seen this being accepted practice and it being not accepted practice, and I found the latter brought big benefits. I don't think it's necessary for master to correspond 100% to production, but it should be quite close. If you land something in master and it's not shipped by the end of the day, I'd say that's generally a bad thing and you should consider reverting. > I don't think presenting them as a single repository is helpful for that. It is helpful because all the normal tooling works. You found a problem in production and would like to see where it came from: if production is either a monorepo or a "super-repo" composed of subrepos, you can just bisect. You can just git log the whole thing to understand recent history etc. You can git grep and it will show you all the uses of something, as opposed you have to figure out how to navigate dozens of different things done by different teams. I have worked on projects with monorepos from the start, multi-repos and multi-repo transitioning to mono-repo. Based on that I think monorepos are great, and wherever possible I would strongly encourage their use: everything being in one place and under a single tool is super-super useful. I think a good way to deal with subrepos should preserve a lot of the advantages of monorepos where monorepos are less applicable, so it's very annoying git has terrible support for this.

lmm 6y ago
> I want feature branches to be short lived and integration to happen (predominantly) via master, because that offers significant benefits. For the cases where I really needed something from A in B, rebasing or cherry-picking has not been much of a hassle for me in practice. I think having everything close to master offers benefits, but when it means people start doing things like feature flags then that's a higher cost than the benefits, so feature branches need to live long enough to implement a feature enough to tell whether it works - which is probably a week or two even in the best cases. Going from "everything other than master is private, you ignore what other people are working on until it hits master" to "all pushed feature branches are public unless otherwise stated, you pay attention to what other people are doing and pull their branches whenever you think that would be helpful" is a significant change in mentality, but I found it to be the biggest benefit of moving to DVCS - perhaps because resolving conflicts is disproportionately frustrating work, IME. > you need to to record which commits have passed CI in some way that's trivial to use for bisect skip (e.g. by CI rewriting the head commit's message to indicate it was tested and passed). This is pretty easy to set up and very useful. In my experience you don't need full CI, because people mostly test the code they've just written at least a little bit. Even if they broke something in one commit, they'll usually fix it in the next. > But you can have the exact same thing happen with a merge based workflow, firstly because you can never prevent someone from (badly) rebasing master into their private local branch You can't prevent someone who's determined to put broken commits in their history from doing so, sure. But you can ask everyone to not use rebase and they'll do it, and you can ask them to only commit code that compiles (or passes tests) and they'll do that most of the time (and if you have the occasional isolated non-compiling commit, that's not such a big problem). > secondly because it is also quite likely that someone who couldn't be asked to rebased cleanly but uses merges exclusively would merge master multiplie times into their branch without making the merge commits necessarily pass tests, instead they'd probably also concentrate their efforts on making the head of the feature branch pass. The head of the feature branch is the merge commit, that's the whole point. They can't do any more work on their feature until they've at least made things compile - the first time they try to go through the edit-test cycle, they have to fix anything that they broke when merging. So sometimes a lazy person will make a non-compiling merge and then fix it in the following commit, but that's not too bad because it only leaves a single isolated non-compiling commit in the history. > Kinda. I think the correct way to handle this is make it painful and requite people to duplicate the code in repo. You sometimes need two versions of X in production in parallel for a longer amount of time (e.g. if you are transitioning to a new architecture, and you need to run it as a shadow system for some time first to gather confidence), but it should not be a common thing and I find having both in the working tree works well for these cases. I'm talking about the case where X and Y are separate systems; if your repository has "the current version of X" and "the current version of Y" then that can end up being pretty misleading because what you don't have in your repo is "what X thinks is the current version of Y" i.e. the version of the interface/client code to Y that X is using. > That's a big no-no in my book. Maybe there is some scale at which this is the lesser evil because otherwise you make it too painful to refactor stuff, but in general I think you should make it painful for people to do this: all versions should be in master and there should be strong pressure to avoid having more than one version of anything in master, certainly for anything but a short transition periods. If the library is shared between two systems that are versioned/released/deployed separately then it's necessarily normal to have multiple different versions of the library in production at the same time. IMO at that point it's best for the library to have its own proper release cycle in its own repository and the two systems to have their own repositories, and then each repo's history is an accurate reflection of the thing it's a repository for, but you're prevented from trying to view a combined history of the library and the systems that use it in a naive way, because such a history would always be misleading. If the library's users are all the same team (i.e. part of the same standup etc.) - then it's usually better to treat everything that uses it as a single system and deploy it all at the same time, and then you can keep them all in the same repository and have a single consistent history. But if the library is shared by independent teams then you can't enforce that versioning and deployment are done together. > There are massive downsides to allowing people to use random versions of internal libraries: security concerns, people ending up on some completely outdated version of a lib that then suddenly breaks completely for them and they don't have the time budget to rewrite all their crap, the massive cost of not being able to look into a single repo and know that if you see "import foolib" it's the foolib in the same source tree. > I don't think it's necessary for master to correspond 100% to production, but it should be quite close. If you land something in master and it's not shipped by the end of the day, I'd say that's generally a bad thing and you should consider reverting. All that's true; in the scenario where you're sharing an internal library between multiple systems you actually need to treat that library as a first-class project with semver, release notes, a security update/LTS policy and all the rest of it. But as soon as your organisation gets too big to deploy everything every time someone wants to release a feature, what do you do? It's now inevitable that you will have different versions of foolib running in production, so IMO the best thing is to have a repo structure that reflects that. I don't think you're actually contradicting what I've said: one repo for each system that's versioned and deployed together, if that system is your whole organisation and your whole codebase then great. > You found a problem in production and would like to see where it came from: if production is either a monorepo or a "super-repo" composed of subrepos, you can just bisect. You can just git log the whole thing to understand recent history etc. You can git grep and it will show you all the uses of something, as opposed you have to figure out how to navigate dozens of different things done by different teams. At the point where you decouple the versioning/releasing of different components, the things you want your tools to do become different. You can't check if a method is unused by grepping, because there may be an older version of an intermediate library that's using it. You don't want to bisect in the details of another team's changes, so bisect landing you on either the point where your commit broke or the point where you commit an upgrade of the library that you're using and that broke things is fine, probably better than landing on an internal commit in that library. Even if you do want to do the fix in that library yourself, coming up with a test case for it that's specific to the library rather than using the rest of your system is something you need to do anyway, because that library's test suite should be self-contained. The way of working is different enough that I don't want to use the same interface to it.

by patrec 6y ago

lmm 6y ago

> I want feature branches to be short lived and integration to happen (predominantly) via master, because that offers significant benefits. For the cases where I really needed something from A in B, rebasing or cherry-picking has not been much of a hassle for me in practice. I think having everything close to master offers benefits, but when it means people start doing things like feature flags then that's a higher cost than the benefits, so feature branches need to live long enough to implement a feature enough to tell whether it works - which is probably a week or two even in the best cases. Going from "everything other than master is private, you ignore what other people are working on until it hits master" to "all pushed feature branches are public unless otherwise stated, you pay attention to what other people are doing and pull their branches whenever you think that would be helpful" is a significant change in mentality, but I found it to be the biggest benefit of moving to DVCS - perhaps because resolving conflicts is disproportionately frustrating work, IME. > you need to to record which commits have passed CI in some way that's trivial to use for bisect skip (e.g. by CI rewriting the head commit's message to indicate it was tested and passed). This is pretty easy to set up and very useful. In my experience you don't need full CI, because people mostly test the code they've just written at least a little bit. Even if they broke something in one commit, they'll usually fix it in the next. > But you can have the exact same thing happen with a merge based workflow, firstly because you can never prevent someone from (badly) rebasing master into their private local branch You can't prevent someone who's determined to put broken commits in their history from doing so, sure. But you can ask everyone to not use rebase and they'll do it, and you can ask them to only commit code that compiles (or passes tests) and they'll do that most of the time (and if you have the occasional isolated non-compiling commit, that's not such a big problem). > secondly because it is also quite likely that someone who couldn't be asked to rebased cleanly but uses merges exclusively would merge master multiplie times into their branch without making the merge commits necessarily pass tests, instead they'd probably also concentrate their efforts on making the head of the feature branch pass. The head of the feature branch is the merge commit, that's the whole point. They can't do any more work on their feature until they've at least made things compile - the first time they try to go through the edit-test cycle, they have to fix anything that they broke when merging. So sometimes a lazy person will make a non-compiling merge and then fix it in the following commit, but that's not too bad because it only leaves a single isolated non-compiling commit in the history. > Kinda. I think the correct way to handle this is make it painful and requite people to duplicate the code in repo. You sometimes need two versions of X in production in parallel for a longer amount of time (e.g. if you are transitioning to a new architecture, and you need to run it as a shadow system for some time first to gather confidence), but it should not be a common thing and I find having both in the working tree works well for these cases. I'm talking about the case where X and Y are separate systems; if your repository has "the current version of X" and "the current version of Y" then that can end up being pretty misleading because what you don't have in your repo is "what X thinks is the current version of Y" i.e. the version of the interface/client code to Y that X is using. > That's a big no-no in my book. Maybe there is some scale at which this is the lesser evil because otherwise you make it too painful to refactor stuff, but in general I think you should make it painful for people to do this: all versions should be in master and there should be strong pressure to avoid having more than one version of anything in master, certainly for anything but a short transition periods. If the library is shared between two systems that are versioned/released/deployed separately then it's necessarily normal to have multiple different versions of the library in production at the same time. IMO at that point it's best for the library to have its own proper release cycle in its own repository and the two systems to have their own repositories, and then each repo's history is an accurate reflection of the thing it's a repository for, but you're prevented from trying to view a combined history of the library and the systems that use it in a naive way, because such a history would always be misleading. If the library's users are all the same team (i.e. part of the same standup etc.) - then it's usually better to treat everything that uses it as a single system and deploy it all at the same time, and then you can keep them all in the same repository and have a single consistent history. But if the library is shared by independent teams then you can't enforce that versioning and deployment are done together. > There are massive downsides to allowing people to use random versions of internal libraries: security concerns, people ending up on some completely outdated version of a lib that then suddenly breaks completely for them and they don't have the time budget to rewrite all their crap, the massive cost of not being able to look into a single repo and know that if you see "import foolib" it's the foolib in the same source tree. > I don't think it's necessary for master to correspond 100% to production, but it should be quite close. If you land something in master and it's not shipped by the end of the day, I'd say that's generally a bad thing and you should consider reverting. All that's true; in the scenario where you're sharing an internal library between multiple systems you actually need to treat that library as a first-class project with semver, release notes, a security update/LTS policy and all the rest of it. But as soon as your organisation gets too big to deploy everything every time someone wants to release a feature, what do you do? It's now inevitable that you will have different versions of foolib running in production, so IMO the best thing is to have a repo structure that reflects that. I don't think you're actually contradicting what I've said: one repo for each system that's versioned and deployed together, if that system is your whole organisation and your whole codebase then great. > You found a problem in production and would like to see where it came from: if production is either a monorepo or a "super-repo" composed of subrepos, you can just bisect. You can just git log the whole thing to understand recent history etc. You can git grep and it will show you all the uses of something, as opposed you have to figure out how to navigate dozens of different things done by different teams. At the point where you decouple the versioning/releasing of different components, the things you want your tools to do become different. You can't check if a method is unused by grepping, because there may be an older version of an intermediate library that's using it. You don't want to bisect in the details of another team's changes, so bisect landing you on either the point where your commit broke or the point where you commit an upgrade of the library that you're using and that broke things is fine, probably better than landing on an internal commit in that library. Even if you do want to do the fix in that library yourself, coming up with a test case for it that's specific to the library rather than using the rest of your system is something you need to do anyway, because that library's test suite should be self-contained. The way of working is different enough that I don't want to use the same interface to it.