6 ms·

GitHub Merge Queue Silently Reverted Code

121 points by matthewbauer 5mo ago

NelsonMinar 5mo ago
How did a bug like this get past their testing?
- philpem 5mo ago
  Given the push to AI/Copilot from MS I'd be impressed if they still did any testing.
- estimator7292 5mo ago
  They're at 88% uptime this year. I don't think they have testing.
  - tracker1 5mo ago
    Microslop®, unmatched quality.
- tikiman163 5mo ago
  CoPilot broke github actions in such a way that if too many requests to start a workflow came in at the same time it would reduce the capacity of the build server's queue until you did a full manual reboot of the build server and completely reset the queue. Some people are saying they aren't doing testing, I know this to be incorrect in one very important way. When it comes to people, the ways in which a co-worker is going to be wrong are predictable. When it comes to AI written code, there is zero predictability as to what could be wrong. Trying to test AI written code is like writing code with no try catch blocks, and expecting the application to not crash within 15 minutes of giving to the end user. They will try everything you didn't think to explicitly handle, and when it comes to AI there is no try catch block for subtly incorrect or incomplete logic.
matthewbauer 5mo ago
Might want to check your git logs if you are using the Merge Queue feature. This afternoon, we found that Merge Queue was silently reverting changesets in the merge queue. Acknowledged by GitHub, but could be a very hard problem to debug if you aren't looking for it.
- EdwardDiego 5mo ago
  Same here, ffs GH.
orpheansodality 5mo ago
for other folks currently in an incident trying to resolve the chaos this caused, the first commit we've found with issues in our repo is from ~10:30am pacific this morning
cwkcw 5mo ago
The window to audit is 2026-04-23 16:05 to 20:43 UTC, about 4h 38m, and 'merged incorrectly' is unpacked in exactly no detail on the status page. Practical check is to compare the tree of each merge commit in that window against the tree of the rebased-and-tested commit that the queue ran CI on. GitHub Enterprise Cloud with Data Residency is still affected as of the latest status update, so DR customers shouldn't clear new queue merges yet. Would be kinda useful if GitHub published the specific failure mode, because content-drift and parent-linkage and dropped-changes all need different audit scripts.
acid__ 5mo ago
Tip: identify PRs with this problem by looking for discrepancies between the metadata (files/lines affected) for PRs via GitHub API and associated commits in the time window. This was a pain to clean up!
EdwardDiego 5mo ago
This has caused a morning of fun for our team, how do you break one of the most fundamental bits of your system? Time to look at alternatives...
- ramon156 5mo ago
  Self-hosting has been wonders. I thought I needed the social part of GH (i still kind of do, I like seeing what people are working on or liking) but overall its the same experience.
deleted 5mo ago
[deleted]
MarkMarine 5mo ago
4 people spent hours putting our repo back together at my company after this. GH has been unreliable and now they are breaking the core tenant of what I expect from this service.
- ragall 5mo ago
  > core tenant tenet not tenant
  - mazeez 5mo ago
    [flagged]
  - MarkMarine 5mo ago
    [flagged]
    - ragall 5mo ago
      I don't care to associate with people who are offended by being corrected. They can sod off. And I never go to what you Americans call "parties".
      - MarkMarine 5mo ago
        This type of pedantry doesn’t add to the discussion. You’re correcting someone’s spelling on the internet, it’s pointless except to make you feel superior, adds nothing. Make a salient point, bring something to the discussion please. Sorry I hurt your feelings bud.
        ragall 5mo ago
        You didn't hurt my feelings, buddy. And if you don't understand what I did and why... pity.
        chylex 5mo ago
        Yes it does. The original commenter can learn how to spell better which makes clear communication easier, and don't forget that other people read internet discussions and they may be learning new words or realizing they also didn't know how to spell a word correctly. You seem offended at the prospect of people learning how to do something correctly, I don't get it.
minraws 5mo ago
Wait what? How, what? This doesn't compute
classified 5mo ago
How long will users put up with this before they finally leave? Microslop is inventing new failure modes on a weekly basis.
sidewndr46 5mo ago
Github has been doing this for at least 5 years now. Merging a pull request merges whatever the latest commit on that branch is in the specific backend that handles the merge request. It does not merge what was just pushed into that branch.
tracker1 5mo ago
I experienced a weird bug, where the github ci environment was checking out code that was throwing an error on build... the version tag definitely didn't have the code the error was coming from... I messed with the CI workflow yaml and finally cat'd the file in question... the error code was definitely there (2 lines repeated in the file) that was not in the repository. I nuked the release tag and re-created it, and it started working... was just a really weird bug/experience.
nextlsj 5mo ago
[dead]
jonnonz 5mo ago
Whats going on over there, is clippy in charge of the release button?
Randalthorro 5mo ago
Imagine you pay someone to save your family pictures but sometimes they save only half your data. Instead they try to upsell you on AI filters for your pictures. GitHub, return to your core mission.
LiquidityC 5mo ago
Checked all my PRs that merged on the 23:d. Checked out each branch locally, rebased on trunk and ran 'git diff'. One of them had a diff which was the entire change that was marked as merged in the GH web ui. This happened in complete silence and since the PR in question was just code refactor I would most likely never have noticed.
- LiquidityC 5mo ago
  Considering switching to original method of simple git repo hosting and email patches like the kernel does it. A workflow I favor. Turns out there are no hosting services which provide such solutions. Lots of manual hosting setup and config required... Sad that GitHub sort of cast the mold for how git hosting has been done for the past 2 decades.
  - vga1 5mo ago
    [dead]
mushfiq_rahman 5mo ago
wow!
sancho1952007 5mo ago
Been self-hosting Gitea since ~9 months for my critical projects. Originally I did it because I didn't want AI to train on my private code but seeing what's happening to Github these days, it looks like a bonus that I don't face data integrity issues or uptime issues. I still use Github for my side-projects where I can directly push to main. I just hope these issues get fixed soon. Been using Github for many years now and I can't afford losing my Github history.

121 points by matthewbauer 5mo ago

NelsonMinar 5mo ago

How did a bug like this get past their testing?

philpem 5mo ago
Given the push to AI/Copilot from MS I'd be impressed if they still did any testing.
estimator7292 5mo ago
They're at 88% uptime this year. I don't think they have testing.
- tracker1 5mo ago
  Microslop®, unmatched quality.
tikiman163 5mo ago
CoPilot broke github actions in such a way that if too many requests to start a workflow came in at the same time it would reduce the capacity of the build server's queue until you did a full manual reboot of the build server and completely reset the queue. Some people are saying they aren't doing testing, I know this to be incorrect in one very important way. When it comes to people, the ways in which a co-worker is going to be wrong are predictable. When it comes to AI written code, there is zero predictability as to what could be wrong. Trying to test AI written code is like writing code with no try catch blocks, and expecting the application to not crash within 15 minutes of giving to the end user. They will try everything you didn't think to explicitly handle, and when it comes to AI there is no try catch block for subtly incorrect or incomplete logic.

matthewbauer 5mo ago

Might want to check your git logs if you are using the Merge Queue feature. This afternoon, we found that Merge Queue was silently reverting changesets in the merge queue. Acknowledged by GitHub, but could be a very hard problem to debug if you aren't looking for it.

EdwardDiego 5mo ago
Same here, ffs GH.

orpheansodality 5mo ago

for other folks currently in an incident trying to resolve the chaos this caused, the first commit we've found with issues in our repo is from ~10:30am pacific this morning

cwkcw 5mo ago

The window to audit is 2026-04-23 16:05 to 20:43 UTC, about 4h 38m, and 'merged incorrectly' is unpacked in exactly no detail on the status page. Practical check is to compare the tree of each merge commit in that window against the tree of the rebased-and-tested commit that the queue ran CI on. GitHub Enterprise Cloud with Data Residency is still affected as of the latest status update, so DR customers shouldn't clear new queue merges yet. Would be kinda useful if GitHub published the specific failure mode, because content-drift and parent-linkage and dropped-changes all need different audit scripts.

acid__ 5mo ago

Tip: identify PRs with this problem by looking for discrepancies between the metadata (files/lines affected) for PRs via GitHub API and associated commits in the time window. This was a pain to clean up!

EdwardDiego 5mo ago

This has caused a morning of fun for our team, how do you break one of the most fundamental bits of your system? Time to look at alternatives...

ramon156 5mo ago
Self-hosting has been wonders. I thought I needed the social part of GH (i still kind of do, I like seeing what people are working on or liking) but overall its the same experience.

deleted 5mo ago

[deleted]

MarkMarine 5mo ago

4 people spent hours putting our repo back together at my company after this. GH has been unreliable and now they are breaking the core tenant of what I expect from this service.

ragall 5mo ago
> core tenant tenet not tenant
- mazeez 5mo ago
  [flagged]
- MarkMarine 5mo ago
  [flagged]
  - ragall 5mo ago
    I don't care to associate with people who are offended by being corrected. They can sod off. And I never go to what you Americans call "parties".
    - MarkMarine 5mo ago
      This type of pedantry doesn’t add to the discussion. You’re correcting someone’s spelling on the internet, it’s pointless except to make you feel superior, adds nothing. Make a salient point, bring something to the discussion please. Sorry I hurt your feelings bud.
      - ragall 5mo ago
        You didn't hurt my feelings, buddy. And if you don't understand what I did and why... pity.
      - chylex 5mo ago
        Yes it does. The original commenter can learn how to spell better which makes clear communication easier, and don't forget that other people read internet discussions and they may be learning new words or realizing they also didn't know how to spell a word correctly. You seem offended at the prospect of people learning how to do something correctly, I don't get it.

minraws 5mo ago

Wait what? How, what? This doesn't compute

classified 5mo ago

How long will users put up with this before they finally leave? Microslop is inventing new failure modes on a weekly basis.

sidewndr46 5mo ago

Github has been doing this for at least 5 years now. Merging a pull request merges whatever the latest commit on that branch is in the specific backend that handles the merge request. It does not merge what was just pushed into that branch.

tracker1 5mo ago

I experienced a weird bug, where the github ci environment was checking out code that was throwing an error on build... the version tag definitely didn't have the code the error was coming from... I messed with the CI workflow yaml and finally cat'd the file in question... the error code was definitely there (2 lines repeated in the file) that was not in the repository. I nuked the release tag and re-created it, and it started working... was just a really weird bug/experience.

nextlsj 5mo ago

[dead]

jonnonz 5mo ago

Whats going on over there, is clippy in charge of the release button?

Randalthorro 5mo ago

Imagine you pay someone to save your family pictures but sometimes they save only half your data. Instead they try to upsell you on AI filters for your pictures. GitHub, return to your core mission.

LiquidityC 5mo ago

Checked all my PRs that merged on the 23:d. Checked out each branch locally, rebased on trunk and ran 'git diff'. One of them had a diff which was the entire change that was marked as merged in the GH web ui. This happened in complete silence and since the PR in question was just code refactor I would most likely never have noticed.

LiquidityC 5mo ago
Considering switching to original method of simple git repo hosting and email patches like the kernel does it. A workflow I favor. Turns out there are no hosting services which provide such solutions. Lots of manual hosting setup and config required... Sad that GitHub sort of cast the mold for how git hosting has been done for the past 2 decades.
- vga1 5mo ago
  [dead]

mushfiq_rahman 5mo ago

wow!

sancho1952007 5mo ago

Been self-hosting Gitea since ~9 months for my critical projects. Originally I did it because I didn't want AI to train on my private code but seeing what's happening to Github these days, it looks like a bonus that I don't face data integrity issues or uptime issues. I still use Github for my side-projects where I can directly push to main. I just hope these issues get fixed soon. Been using Github for many years now and I can't afford losing my Github history.