6 ms·

What is a merge queue, and does your team need one?

69 points by dbalatero 3y ago

gregmfoster 3y ago
If a merge queue is light and fast, all repos could benefit from having it enabled by default
dave4420 3y ago
I presume merge queues are useless if you have flaky tests?
- jakub_g 3y ago
  A good MQ impl allows you to make the MQ checks different than in PR. And you choose for MQ builds only a subset of tests which are known to be non-flaky. Merge queue builds should be fast and rock stable, following the 80/20 rule.
- mfenniak 3y ago
  They're super annoying if you have flaky tests. If the tests are such that you can automate retrying and get to a reasonably low false-positive rate, it can be usable.
  - fosterfriends 3y ago
    I like adopting a merge queue as a forcing function to deal with flaky tests.
    - henryfjordan 3y ago
      Why jump to a merge queue? Github has had a feature to grey-out the merge button if the tests aren't passing for years.
  - ankitdce 3y ago
    Excited to see this thread. Aviator also offers a high performance MergeQueue with flaky test support: https://docs.aviator.co/mergequeue/managing-flaky-tests-in-mergequeue https://docs.aviator.co/mergequeue/managing-flaky-tests-in-m... Disclaimer: I'm the co-founder of Aviator
- masklinn 3y ago
  Not useless, you get sprurious rejections and have to resubmit, like running CI. and you can retry—on-failure to increase the odds. Some test frameworks support rerunning flaky tests. They’re a good justification to work on that tho.
- wizofaus 3y ago
  From what I've observed semantic merge issues always get picked up at compile time - hence only the compile phase need pass to allow a queued merge through, but I guess if you're working a lot with weakly-typed or interpreted languages it'd be different. It's hard not to think there should be some way of automatically flagging likely issues where, say, commits in one PR refer to particular identifiers that were modified in parallel PRs, but it definitely goes well beyond what diff-merge tools typically do today.
  - skissane 3y ago
    > From what I've observed semantic merge issues always get picked up at compile time - hence only the compile phase need pass to allow a queued merge through, but I guess if you're working a lot with weakly-typed or interpreted languages it'd be different. I agree that 90% of the time, with strongly typed languages, a compile is enough. But even in those languages, you can still get semantic merge issues which aren’t compilation-related. Suppose your app has a config file. The first PR renames one of the configuration keys in the config file, the second PR adds some new code which reads that configuration key. The combination compiles fine, but the second PR fails in integration tests, because the config key it was expecting wasn’t there any more. Of course this is a sign of a design flaw - the name of the config key should occur only once in the program (say as a constant), so both PRs would use the constant, and the second PR would get the change in constant value from the first, or else the first renames the constant so the second doesn’t compile. Or even a higher-level API which deals with the semantics of what the config represents instead of just key-value strings. But a lot of large legacy code bases contain design flaws like this, and fixing them can be a lot of work, so they don’t get fixed, at least not any time soon.
    - wizofaus 3y ago
      Sure, there are always going to be occasions where a combination of two bits of work done separately cause unexpected behaviour changes - but generally if there are integration/automated E2E tests that are capable of picking that sort of thing up I'd expect they typically get triggered after the mainline merge anyway. OTOH, I can understand at least wanting to ensure that it's not possible for two consecutive merges to the mainline branch to cause it stop building.
nrclark 3y ago
If somebody from Github is reading this: when will merge queues land in Github Enterprise Server? I work on a project that badly needs the feature.
- gregmfoster 3y ago
  FWIW, graphite.dev's MQ supports GitHub Enterprise Server
  - dijit 3y ago
    Just looked at graphite.dev and it looks like it only works with Github. Wild that an entire product can exist that depends on another commercial SaaS offering to have any value at all. That feels absurdly risky to me. Like, I am getting anxiety from the thought of working on that product. Then again, I guess AWS exists and we all make our peace with it. We really don’t like solid foundations do we.
    - masklinn 3y ago
      It’s risky but at the same time it allows creating a significant spike of value without spreading thin. Like, github PRs are pretty shit, which is an opportunity for reviewable.
      - maccard 3y ago
        > Like, github PRs are pretty shit, which is an opportunity for reviewable. I've used a bunch of VCS's and managed solutions, and PR's are probably the _best_ I've had the pleasure of using. Working with perforce, everyone is one (or a small number) of long lived branch(es), chucking around WIP code as shelves. PlasticSCM/Unity DevOps Version Control (seriously, that's what they renamed it to) provides absolutely no support for a merge-based workflow other than some half-baked tools that barely work.
        masklinn 3y ago
        > I've used a bunch of VCS's and managed solutions, and PR's are probably the _best_ I've had the pleasure of using. That is genuinely sad.
deleted 3y ago
[deleted]
tragic95 3y ago
I'm convinced that all repositories with more than a few hundred PRs definitely need a merge queue. Glad to see discussion in this space, I've been frustrated by the lack of merge queues at previous companies.
- nvarsj 3y ago
  Agreed. It's a common problem that many teams don't even realise they have. Same thing with stacked PRs. It's a wonder Github hasn't shipped with merge queue+stacked PRs by now.
  - pperi11 3y ago
    Graphite.dev is stacked PRs for github
jacobegold 3y ago
What's the difference between a merge queue and a pull queue?
- fosterfriends 3y ago
  It's like a rebase queue, but with more ref-log, and less tree inversion. Pull queues counter-intuitively lean on force-push-with-lease sub-commands, whereas monorepo merge queues have remote build caches for... sorry I blacked out.
  - kgwxd 3y ago
    I bought that hook line and sinker :) I was even feeling a real blackout start at the word "remote". Comedic timing is hard to pull off in text. Is any of that even real?
    - gregmfoster 3y ago
      nah, not real, I was just being silly :)
peterhadlaw 3y ago
[flagged]
johanneskanybal 3y ago
[flagged]
- gregmfoster 3y ago
  why
bryanlarsen 3y ago
If your fancy SaaS git repo thingy doesn't support merge queue's, maintain your own using the `--update-refs` option to git. https://andrewlock.net/working-with-stacked-branches-in-git-is-easier-with-update-refs/ https://andrewlock.net/working-with-stacked-branches-in-git-...
- tomasreimers 3y ago
  Hmm, I think there may be some confusion. A merge queue != stacked branches.
  - gregmfoster 3y ago
    yeah, it's really interesting, I've seen this confusion before
  - bryanlarsen 3y ago
    Sure, but we use a stacked branch to solve the same problem that a merge queue solves - preventing semantic merge conflicts by predetermining a merge order.
    - zuckerborgggg 3y ago
      [flagged]
- renewiltord 3y ago
  Oh, `--update-refs` is cool. I think this is what I want sometimes. I'll have a commit I want to test results of against branchA and branchB (e.g. I have `master` and `branchB` with two different implementations of some thing and I want to test some code in ref Y against it. Previously, I had to create a separate branch that was master+Y and branchB+Y and I might alter the branchB and have to rebase branchB+Y on it. Actually I think it doesn't solve the problem now that I wrote it out, but it is interesting nonetheless.
ilyt 3y ago
Think if you have that many developers you need to start doing it Linux kernel style and have maintainers responsible for certain parts of the project making sure merges go in fine. And maybe have actual communication in other means than via comments on tickets. The whole hypothetical example could be avoided by typing "hey, I'm changing this field to be that, anything against?" on team's chat > Since my_branch was merged before it was rebased onto someone_elses_branch, this error goes unnoticed and BAM - main is broken! This is known as a semantic merge conflict — a merge is technically possible, but results in a regression. Do people not merge master into their branch before pushing? All that seems like mostly problem caused by having too long living branches. I'm also confused how that even fixes anything, someone still have to go back and fix that code.
- luketheobscure 3y ago
  > Do people not merge master into their branch before pushing? All that seems like mostly problem caused by having too long living branches. On projects with a high commit velocity and a large team, simply merging main and running the tests is not enough. The likelihood that a new commit lands on main before your tests finish is high, and restarting the process every time there's a new commit is time consuming. Merge queues are the only way to ensure that main does not break from conflicts that arise "in between" commits.
  - pperi11 3y ago
    You could just use smaller PRs See: https://graphite.dev/blog/the-ideal-pr-is-50-lines-long https://graphite.dev/blog/the-ideal-pr-is-50-lines-long
  - throwbadubadu 3y ago
    > Do people not merge master into their branch before pushing? No, they rebase! (Sorry, couldn't resist, yeah the other half does this (!attention, opinion!) ugly cross merges that make the commit graph look like my cables under the desk... > Merge queues are the only way to ensure that main does not break from conflicts that arise "in between" commits. Not the only one (though you may argue this is a degenerated queue): use merge locks! Yeah, we really use git svn style, the locking is done via a chat channel. No, please don't laugh. Acquire the lock, merge develop or rebase, check everything again, release lock :D :D (No again: I am not suggesting this to anyone!! :D )
hlieberman 3y ago
Or use zuul. That solves your problem.
JohnDont 3y ago
You have everything well explained here : https://blog.mergify.com/whats-a-merge-queue-and-why-use-it/ https://blog.mergify.com/whats-a-merge-queue-and-why-use-it/
- zuckerborgggg 3y ago
  [flagged]
deleted 3y ago
[deleted]
thcipriani 3y ago
The first description I read of a merge queue came from Graydon Hoare, the creator of the Rust programming language, describing a CI system his team cobbled together in 2001[0]. He called this system "The Not Rocket Science Rule Of Software Engineering" Zuul, of course, had this back in 2012.[1] 2023: GitHub catching up! [0]: <https://graydon2.dreamwidth.org/1597.html https://graydon2.dreamwidth.org/1597.html> [1]: <https://zuul-ci.org/ https://zuul-ci.org/>
kathlam 3y ago
Large teams need things beyond a merge queue, they need a merge graph: https://trunk.io/blog?post=trunk-merge https://trunk.io/blog?post=trunk-merge

69 points by dbalatero 3y ago

gregmfoster 3y ago

If a merge queue is light and fast, all repos could benefit from having it enabled by default

dave4420 3y ago

I presume merge queues are useless if you have flaky tests?

jakub_g 3y ago
A good MQ impl allows you to make the MQ checks different than in PR. And you choose for MQ builds only a subset of tests which are known to be non-flaky. Merge queue builds should be fast and rock stable, following the 80/20 rule.
mfenniak 3y ago
They're super annoying if you have flaky tests. If the tests are such that you can automate retrying and get to a reasonably low false-positive rate, it can be usable.
- fosterfriends 3y ago
  I like adopting a merge queue as a forcing function to deal with flaky tests.
  - henryfjordan 3y ago
    Why jump to a merge queue? Github has had a feature to grey-out the merge button if the tests aren't passing for years.
- ankitdce 3y ago
  Excited to see this thread. Aviator also offers a high performance MergeQueue with flaky test support: https://docs.aviator.co/mergequeue/managing-flaky-tests-in-mergequeue https://docs.aviator.co/mergequeue/managing-flaky-tests-in-m... Disclaimer: I'm the co-founder of Aviator
masklinn 3y ago
Not useless, you get sprurious rejections and have to resubmit, like running CI. and you can retry—on-failure to increase the odds. Some test frameworks support rerunning flaky tests. They’re a good justification to work on that tho.
wizofaus 3y ago
From what I've observed semantic merge issues always get picked up at compile time - hence only the compile phase need pass to allow a queued merge through, but I guess if you're working a lot with weakly-typed or interpreted languages it'd be different. It's hard not to think there should be some way of automatically flagging likely issues where, say, commits in one PR refer to particular identifiers that were modified in parallel PRs, but it definitely goes well beyond what diff-merge tools typically do today.
- skissane 3y ago
  > From what I've observed semantic merge issues always get picked up at compile time - hence only the compile phase need pass to allow a queued merge through, but I guess if you're working a lot with weakly-typed or interpreted languages it'd be different. I agree that 90% of the time, with strongly typed languages, a compile is enough. But even in those languages, you can still get semantic merge issues which aren’t compilation-related. Suppose your app has a config file. The first PR renames one of the configuration keys in the config file, the second PR adds some new code which reads that configuration key. The combination compiles fine, but the second PR fails in integration tests, because the config key it was expecting wasn’t there any more. Of course this is a sign of a design flaw - the name of the config key should occur only once in the program (say as a constant), so both PRs would use the constant, and the second PR would get the change in constant value from the first, or else the first renames the constant so the second doesn’t compile. Or even a higher-level API which deals with the semantics of what the config represents instead of just key-value strings. But a lot of large legacy code bases contain design flaws like this, and fixing them can be a lot of work, so they don’t get fixed, at least not any time soon.
  - wizofaus 3y ago
    Sure, there are always going to be occasions where a combination of two bits of work done separately cause unexpected behaviour changes - but generally if there are integration/automated E2E tests that are capable of picking that sort of thing up I'd expect they typically get triggered after the mainline merge anyway. OTOH, I can understand at least wanting to ensure that it's not possible for two consecutive merges to the mainline branch to cause it stop building.

nrclark 3y ago

If somebody from Github is reading this: when will merge queues land in Github Enterprise Server? I work on a project that badly needs the feature.

gregmfoster 3y ago
FWIW, graphite.dev's MQ supports GitHub Enterprise Server
- dijit 3y ago
  Just looked at graphite.dev and it looks like it only works with Github. Wild that an entire product can exist that depends on another commercial SaaS offering to have any value at all. That feels absurdly risky to me. Like, I am getting anxiety from the thought of working on that product. Then again, I guess AWS exists and we all make our peace with it. We really don’t like solid foundations do we.
  - masklinn 3y ago
    It’s risky but at the same time it allows creating a significant spike of value without spreading thin. Like, github PRs are pretty shit, which is an opportunity for reviewable.
    - maccard 3y ago
      > Like, github PRs are pretty shit, which is an opportunity for reviewable. I've used a bunch of VCS's and managed solutions, and PR's are probably the _best_ I've had the pleasure of using. Working with perforce, everyone is one (or a small number) of long lived branch(es), chucking around WIP code as shelves. PlasticSCM/Unity DevOps Version Control (seriously, that's what they renamed it to) provides absolutely no support for a merge-based workflow other than some half-baked tools that barely work.
      - masklinn 3y ago
        > I've used a bunch of VCS's and managed solutions, and PR's are probably the _best_ I've had the pleasure of using. That is genuinely sad.

deleted 3y ago

[deleted]

tragic95 3y ago

I'm convinced that all repositories with more than a few hundred PRs definitely need a merge queue. Glad to see discussion in this space, I've been frustrated by the lack of merge queues at previous companies.

nvarsj 3y ago
Agreed. It's a common problem that many teams don't even realise they have. Same thing with stacked PRs. It's a wonder Github hasn't shipped with merge queue+stacked PRs by now.
- pperi11 3y ago
  Graphite.dev is stacked PRs for github

jacobegold 3y ago

What's the difference between a merge queue and a pull queue?

fosterfriends 3y ago
It's like a rebase queue, but with more ref-log, and less tree inversion. Pull queues counter-intuitively lean on force-push-with-lease sub-commands, whereas monorepo merge queues have remote build caches for... sorry I blacked out.
- kgwxd 3y ago
  I bought that hook line and sinker :) I was even feeling a real blackout start at the word "remote". Comedic timing is hard to pull off in text. Is any of that even real?
  - gregmfoster 3y ago
    nah, not real, I was just being silly :)

peterhadlaw 3y ago

[flagged]

johanneskanybal 3y ago

[flagged]

gregmfoster 3y ago
why

bryanlarsen 3y ago

If your fancy SaaS git repo thingy doesn't support merge queue's, maintain your own using the `--update-refs` option to git. https://andrewlock.net/working-with-stacked-branches-in-git-is-easier-with-update-refs/ https://andrewlock.net/working-with-stacked-branches-in-git-...

tomasreimers 3y ago
Hmm, I think there may be some confusion. A merge queue != stacked branches.
- gregmfoster 3y ago
  yeah, it's really interesting, I've seen this confusion before
- bryanlarsen 3y ago
  Sure, but we use a stacked branch to solve the same problem that a merge queue solves - preventing semantic merge conflicts by predetermining a merge order.
  - zuckerborgggg 3y ago
    [flagged]
renewiltord 3y ago
Oh, `--update-refs` is cool. I think this is what I want sometimes. I'll have a commit I want to test results of against branchA and branchB (e.g. I have `master` and `branchB` with two different implementations of some thing and I want to test some code in ref Y against it. Previously, I had to create a separate branch that was master+Y and branchB+Y and I might alter the branchB and have to rebase branchB+Y on it. Actually I think it doesn't solve the problem now that I wrote it out, but it is interesting nonetheless.

ilyt 3y ago

Think if you have that many developers you need to start doing it Linux kernel style and have maintainers responsible for certain parts of the project making sure merges go in fine. And maybe have actual communication in other means than via comments on tickets. The whole hypothetical example could be avoided by typing "hey, I'm changing this field to be that, anything against?" on team's chat > Since my_branch was merged before it was rebased onto someone_elses_branch, this error goes unnoticed and BAM - main is broken! This is known as a semantic merge conflict — a merge is technically possible, but results in a regression. Do people not merge master into their branch before pushing? All that seems like mostly problem caused by having too long living branches. I'm also confused how that even fixes anything, someone still have to go back and fix that code.

luketheobscure 3y ago
> Do people not merge master into their branch before pushing? All that seems like mostly problem caused by having too long living branches. On projects with a high commit velocity and a large team, simply merging main and running the tests is not enough. The likelihood that a new commit lands on main before your tests finish is high, and restarting the process every time there's a new commit is time consuming. Merge queues are the only way to ensure that main does not break from conflicts that arise "in between" commits.
- pperi11 3y ago
  You could just use smaller PRs See: https://graphite.dev/blog/the-ideal-pr-is-50-lines-long https://graphite.dev/blog/the-ideal-pr-is-50-lines-long
- throwbadubadu 3y ago
  > Do people not merge master into their branch before pushing? No, they rebase! (Sorry, couldn't resist, yeah the other half does this (!attention, opinion!) ugly cross merges that make the commit graph look like my cables under the desk... > Merge queues are the only way to ensure that main does not break from conflicts that arise "in between" commits. Not the only one (though you may argue this is a degenerated queue): use merge locks! Yeah, we really use git svn style, the locking is done via a chat channel. No, please don't laugh. Acquire the lock, merge develop or rebase, check everything again, release lock :D :D (No again: I am not suggesting this to anyone!! :D )

hlieberman 3y ago

Or use zuul. That solves your problem.

JohnDont 3y ago

You have everything well explained here : https://blog.mergify.com/whats-a-merge-queue-and-why-use-it/ https://blog.mergify.com/whats-a-merge-queue-and-why-use-it/

zuckerborgggg 3y ago
[flagged]

deleted 3y ago

[deleted]

thcipriani 3y ago

The first description I read of a merge queue came from Graydon Hoare, the creator of the Rust programming language, describing a CI system his team cobbled together in 2001[0]. He called this system "The Not Rocket Science Rule Of Software Engineering" Zuul, of course, had this back in 2012.[1] 2023: GitHub catching up! [0]: <https://graydon2.dreamwidth.org/1597.html https://graydon2.dreamwidth.org/1597.html> [1]: <https://zuul-ci.org/ https://zuul-ci.org/>

kathlam 3y ago

Large teams need things beyond a merge queue, they need a merge graph: https://trunk.io/blog?post=trunk-merge https://trunk.io/blog?post=trunk-merge