7 ms·
Top model scores may be skewed by Git history leaks in SWE-bench
- Traster 1y agoMan I feel so dumb. Why haven't I been doing this in my job, if I could just see the commit that fixed my issue this would all be so easy.
- Noumenon72 1y agoSomeone did comment that it's actually smart to check if something is fixed on the unstable branch, or I suppose in your coworkers' branches. A good task for an LLM.
- falcor84 1y agoOh, you haven't been using `git fetch-future-solution`?
- piskov 1y agoNot “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3 https://arxiv.org/html/2506.12286v3
- stefan_ 1y agoSo the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the bench.
- jsheard 1y ago> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.
- geekymartian 1y agothat was my exact thought. how fitting
- hhh 1y agoVerified has a completely different meaning for this, it's that the questions have verified valid solutions.
- yorwba 1y agoThe "Verified" part of "SWE-Bench Verified" means that there was plain "SWE-Bench" before it, which had actually not been verified at all and included a lot of tasks that didn't really make sense for use as a benchmark: https://openai.com/index/introducing-swe-bench-verified/#adapting-swe-bench-as-a-preparedness-evaluation https://openai.com/index/introducing-swe-bench-verified/#ada... Data contamination stemming from the fact that it's based on already-solved problems in public repositories is a different issue that cannot be addressed by verifying the benchmark questions harder, but only by putting stricter limits on the model under test.
- kronks 1y ago[dead]
- sebzim4500 1y agoThe verified refers to the fact that the benchmark problems were verified by human experts to be reasonable. It says nothing about data contamination, which would depend on the model and would not be the fault of the benchmark.
- blibble 1y ago> I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? I doubt any of the AI company employees are encouraged to go looking for cheating
- lieret 1y ago[On the SWE-bench team] As someone pointed out SWE-bench Verified is a subset of tasks that were reviewed to be solvable (i.e., have enough context in the task description) as well are scored with unit tests that aren't overly specific to rule out valid solutions. We've all read & analyzed a large number of agent trajectories. This loophole seems to be something that popped up with the more recent models and we simply weren't aware of it. As discussed in the github issue, there's a fix in the new version of the SWE-bench containers (currently being rolled out) that makes sure that the relevant commits aren't available. Part of what makes SWE-bench a very interesting benchmark is the enormous action space that agents that compete on it can take. However that also means that there's unexpected things happening when models get better. We're currently working on making all agent runs easily browsable on a website (rather than having to download our AWS buckets) to get even more eyes on the trajectories. Thanks to everyone who uncovered this loophole.
- deleted 1y ago[deleted]
- fine_tune 1y agoI was going to argue "LLM's need code samples to-do well on languages and if we are honest C# is a language mostly held in private repo's" but Github's 2024 report[0] says its the 5th most used language (I'm to lazy to check if this report includes private repo's but I'll assume it doesn't). So kinda neat to see this paper! [0]https://github.blog/news-insights/octoverse/octoverse-2024/#the-most-popular-programming-languages https://github.blog/news-insights/octoverse/octoverse-2024/#...
- yieldcrv 1y ago5th most used language based on private repos that the group making the report has the exclusive direct access to seeing I don't see that contradicting your assumption
- BoorishBears 1y ago"In this year’s Octoverse report, we study how public and open source activity on GitHub..."
- CuriouslyC 1y agoThe big labs are almost certainly using compiler/repl output for generated code as an oracle for RL. I doubt they have C# in the mix.
- tomjakubowski 1y agoWhy do you doubt that? It's a widely used language. And there is even an open source C# REPL.
- deleted 1y ago[deleted]
- CuriouslyC 1y agoBecause RL time is expensive and I don't think the languages which are more popular than C# have such high performance that it's worth bumping their batches for C#.
- teaearlgraycold 1y agoPersonally I don't look at or respect LLM benchmarks at all. I've seen SOTA models fail in incredibly shocking ways even recently. Those moments immediately bring me out of the delusion that LLMs have thinking capacity or an understanding of code.
- phatskat 1y ago> the delusion that LLMs have thinking capacity It’s such a strange delusion too, because it’s easy to get caught up in for a moment and it’s easy to remember “oh no this thing is as smart as a bag of bricks”. What strikes me more is how these companies sell their AI offerings - we watched an OpenAI presentation about spec-driven development recently and the presenter was fairly, idk, fine enough if maybe a bit grandiose. But what really nagged me was the way he ended his presentation with something along the lines of “we’re excited to see AGI continue to grow” and it’s honestly A) depressing and B) downright fraud - there is no current AGI to speak of, it’s all just guessing the string of words that sound best together and this OpenAI rep _knows this_. They know that no amount of up-front spec writing will prevent bugs. They know that their LLM doesn’t “know” anything in an actually meaningful way. They know that calling what they have “AGI” is aspirational at best and lying at worst.
- jasonjmcghee 1y agoVery interested to see the updated results. This could really shake up the leaderboard.
- blurbleblurble 1y agoI hope it does. These coding benchmarks have often seemed frustratingly out of touch with my experience.
- mustaphah 1y agoI speculate something similar (or even worse) is going on with Terminal-Bench [1]. Like, seriously, how come all these agents are beating Claude Code? In practice, they are shitty and not even close. Yes. I tried them. [1] https://www.tbench.ai/leaderboard https://www.tbench.ai/leaderboard
- zaptheimpaler 1y agoIt's honestly ridiculous they left git history lying around during a benchmark, and this benchmark made to ICLR in Jan 2024 and no one has detected this issue until now. I don't really trust any benchmarking or tools or claims from this space when they can make such huge basic errors.
- Nijikokun 1y agoThere was a lot of speculation whether or not the model would use them or even if it would attempt to use them and they noted this months ago. Now they have clear evidence of them doing so. Seems reasonable.
- dolmen 1y agoNext models will use zero-day to escape the sandbox and access the answer.
- lieret 1y ago[On swe-bench team] We read and analyzed a lot of trajectories but seems like only recently models have started to exploit this in a small fraction of instances. But yes, clearly shouldn't have happened (and is now fixed in the new container versions).
- belter 1y agoIn the meawhile, Oracle stock went up 40% in one one day, based on what Wall Street thinks AI might be...in 4 years...Not a bubble at all...
- ksherlock 1y agoThe real bubble will come once interest rates start dropping.
- candiddevmike 1y agoI think Oracle's stock mostly popped due to a delayed reaction with the US GSA contract it secured in July and the revenue guidance probably related to it: https://www.oracle.com/news/announcement/blog/oracle-cloud-cuts-costs-and-propels-missions-for-government-agencies-2025-07-14/ https://www.oracle.com/news/announcement/blog/oracle-cloud-c...
- belter 1y agoLol...That contract has Oracle offering licenses at a discount of 75% and is estimated to make them not more than one 1 Billion. The other big contract on Cloud services the DoD JWCC is $8B to 9B but shared by four vendors (AWS, Microsoft, Google, Oracle) and Oracle orders under it are in the hundreds of millions not even 1 Billion... Wall Street is currently heavily punishing any company who misses their quarter, including NVIDIA!, after beating on their quarter. Oracle had a earnings miss in the current quarter! Their current REALITY is ~$15B quarterly revenue (with cloud infra ~$3B) and only ~$12B in near-term deferred backlog and deferred backlog is NOT revenue. To justify the valuation, this would imply OCI going from ~$18B in FY26 to ~$140B by FY30 that is an insane promise of +$120B in 4 years but back-loaded into the year 3 or year 4. :-)) Capex needs ~$35B next year just to chase GPUs/power and if they miss one quarter the story implodes. The supposed rational, efficient market, is paying near $1T today for back-loaded hopes. Is completely bubble math. Like anybody, including Oracle AND their Customers, have ANY idea of their Capex in 4 years. Complete and total bubble.
- Zacharias030 1y ago
- slacktivism123 1y agoFascinating case showing how LLM promoters will happily take "verified" benchmarks at their word. It's easy to publish "$NEWMODEL received an X% bump in SWE-Bench Verified!!!!". Proper research means interrogating the traces, like these researchers did (the Gist shows Claude 4 Sonnet): https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa669a10 https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa... Commentary: https://x.com/bwasti/status/1963288443452051582 https://x.com/bwasti/status/1963288443452051582, https://x.com/tmkadamcz/status/1963996138044096969 https://x.com/tmkadamcz/status/1963996138044096969
- k__ 1y agoYes, often you see huge gains in some benchmark, then the model is ran through Aider's polyglot benchmark and doesn't even hit 60%.
- Workaccount2 1y agoThe best benchmark is the community vibe in the weeks following a release. Claude benchmarks poorly but vibes well. Gemini benchmarks well and vibes well. Grok benchmarks well but vibes poorly. (yes I know you are gushing with anecdotes, the vibes are simply the approximate color of gray born from the countless black and white remarks.)
- wubrr 1y agothe vibes are just a collection anecdotes
- ryoshu 1y ago"qual"
- diggan 1y ago> The best benchmark is the community vibe in the weeks following a release. True, just be careful what community you use as a vibe-check. Most of the mainstream/big ones around AI and LLMs basically have influence campaigns run against them, are made of giant hive-minds that all think alike and you need to carefully asses if anything you're reading is true or not, and votes tend to make it even worse.
- mbowcut2 1y agoI'm not surprised. People really thought the models just kept getting better and better?
- guerrilla 1y agoMaybe. How would I know?
- jMyles 1y ago...even if the agent did "cheat", I think that having the capacity to figure out that it was being evaluated, find the repo containing the logic of that evaluation, and find the expected solution to the problem it faced... is "better" than anything that the models were able to do a couple years ago.
- segmondy 1y agoThe models are getting better and better.
- ofirpress 1y ago[I'm on the SWE-bench team] Multiple people have looked into this, for example right in that thread: https://github.com/SWE-bench/SWE-bench/issues/465#issuecomment-3258065126 https://github.com/SWE-bench/SWE-bench/issues/465#issuecomme... This issue had affected a tiny fraction of existing agents in a tiny fraction of their runs. And we've now issued a fix. This is a natural part of running a benchmark, I'm sure tiny things like this will keep on getting discovered and we'll keep on fixing them. This doesn't change the overall picture or trends at all.
- comex 1y agoThe comment you link to says that "we only performed a quick preliminary search" and "We do not have a method for automatically checking existing trajectories." In other words, it can't confirm that the issue only "affected a tiny fraction of existing agents in a tiny fraction of their runs" as you say. Are you saying that you have since separately confirmed this? Edit: That said, I’m willing to believe based on the information in the thread that this most likely only affects a tiny fraction of runs.
- typpilol 1y agoYa what he links directly contradicts what he's saying lol
- ofirpress 1y ago[flagged]
- bflesch 1y agoUnfortunately the bank account trajectories are not public, because unscupulous corporations such FAANG who let thousands of engineers wade through my chat messages on their platforms might not shy away from bribing academics to improve benchmarks of their billion-dollar AI initiatives. It's also a bribe if my sibling gets a job with $500k annual salary. Tech is not immune to it.
- epolanski 1y agoThis is beyond sad and shameful.
- falcor84 1y agoIf you believe that you can develop a benchmark that wouldn't have any issues, please do so.
- epolanski 1y agoSo instead of calling out the cheaters we victim blame the benchmarks for leaving traces of exploits?
- Tanjreeve 1y agoThis might be the most annoying habit of corporarte AI that it might be one of the few industries that goes around demanding everyone else provides clear use cases and proof of efficacy for it. 1. If the benchmarks are just testing the ability to get the answers from history then something is clearly wrong with the benchmark. 2. If that's even a possibility then that's going to lower confidence in the ability to deal with the vast majority of problems where you don't already have the answer written down. 3. That's not the customers problem to solve on behalf of the vendor.
- OtherShrezzing 1y agoThat the answers have been available to them in the environment, and they’re still not hitting 100% on this benchmark is a damning indictment of SOTA model performance.
- aurareturn 1y agoAre you going to rail on humans for making this mistake in the first place?
- themafia 1y agoNo because that's the baseline. It's what you do when you have no other choice. Railing against that would be pointless.
- ares623 1y agoi mean, if a human was claiming they could do that and successfully received billions to attempt to do it, and fail to deliver, i'd be railing against that particular human too
- raincole 1y agoIt really isn't. Do you expect SOTA models to answer any answered question on the internet with 100% accuracy? Congrats you just compressed the whole internet (at least a few zettabytes) into a model (a few TB at most?).
- OtherShrezzing 1y agoThe linked ticket isn’t suggesting the commit is in the training data. It’s demonstrating that models run ‘git log’, find the exact code to fix the issue against which they’ll be scored, and then they implement that code as-is. The test environment contains the answers to the questions.
- imiric 1y agoWell, we're dealing with (near) superintelligence here, according to the companies that created the models. Not only would I expect them to regurgitate the answers they were trained on, which includes practically the entire internet, but I would expect them to answer questions they weren't trained on. Maybe not with 100% accuracy, but certainly much higher than they do now. It's perfectly reasonable to expect a level of performance concordant with the marketing of these tools. Claiming this is superintelligence, while also excusing its poor performance is dishonest and false advertising.
- deleted 1y ago[deleted]
- jgalt212 1y agoBaseball players cheat for tens of millions. The stakes are 2-4 orders of magnitude higher here. I'm not surprised in the least.
- jMyles 1y agoRegardless of whether, during this particular evaluation, Claude 4 Sonnet looked at the solution to this particular problem in this particular git repo, this seems like a long-term intractable problem. How can we ever perform this sort of faux-neutral agentic evaluation in an environment where we want agents to have access to the sum total of knowledge (which will necessarily include being able to learn about the evaluation being conducted and its expectations)?
- Aperocky 1y agoepochs ago when random forest was part of machine learning nomenclature, we had a strong claim from an adjacent team in the form of a powerpoint circulated upwards that they had achieved almost perfect prediction accuracy. We relatively quickly identified that the testing set are taken directly from the training set, but the claim has been advertised already so they were more difficult to retract... if it were at all, I left shortly after. The incentives are not aligned with accurate reporting.
- pseudosavant 1y agoIf I was doing those tasks, and I found that someone had already fixed it in a future (from my git state) commit, I'd think I was being pretty smart to use that solution too. Turns out the test shouldn't have the answers included in it?
- bryan0 1y agohah the model should get extra credit for discovering this! > Now I understand the situation perfectly! The issue described in the problem statement is a real bug that was already identified and fixed in later versions of pytest. Since we're working with pytest 5.2.4, we need to apply the same fix. https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa669a10 https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa...
- howenterprisey 1y agoAm I to interpret https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa669a10#file-gistfile0-txt-L235 https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa... as it making a test that only asserts false and saying that the test exercises the function in question? Edit: I misunderstood what was being tested; the test is correct.
- skeezyboy 1y ago> hah the model should get extra credit for discovering this! you whoever included it in the training data should get the credit
- ripped_britches 1y agoEveryone on HN is like “yes I knew it! I was so right in 2021 that LLMs were just stochastic parrots!” Strangely one of the most predictable groups of people
- pessimizer 1y agoBecause they are. But stochastic parrots are awesome.
- ripped_britches 1y agoI challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `mapping` and a `ciphertext`. Decode by replacing each ciphertext character `c` with `mapping[c]` (i.e., mapping maps ciphertext → plaintext). Do not guess; just apply the mapping. Return *ONLY* this JSON (no prose, no extra keys, no code fences): { "decoded_prefix": "<first 40 characters of the decoded plaintext>", "last_10": "<last 10 characters of the decoded plaintext>", "vowel_counts": {"a": <int>, "e": <int>, "i": <int>, "o": <int>, "u": <int>} } Inputs use only lowercase a–z. mapping = { "a":"c","b":"j","c":"b","d":"y","e":"w","f":"f","g":"l","h":"u","i":"m","j":"g", "k":"x","l":"i","m":"o","n":"n","o":"h","p":"a","q":"d","r":"t","s":"r","t":"v", "u":"p","v":"s","w":"z","x":"k","y":"q","z":"e" } ciphertext = "nykwnowotyttbqqylrzssyqcmarwwimkiodwgafzbfippmndzteqxkrqzzophqmqzlvgywgqyazoonieqonoqdnewwctbsbighrbmzltvlaudfolmznbzcmoafzbeopbzxbygxrjhmzcofdissvrlyeypibzzixsjwebhwdjatcjrzutcmyqstbutcxhtpjqskpojhdyvgofqzmlwyxfmojxsxmb" DO NOT USE ANY CODE EXECUTION TOOLS AT ALL. THAT IS CHEATING.
- incr_me 1y agoThat's exactly the sort of thing a "stochastic parrot" would excel at. This could easily serve as a textbook example of the attention mechanism.
- rockwotj 1y agoA friend is starting a company to do evals by just pitting models agent each other in simulations. Their teaser video is good (and humorous!) https://kradle.ai/ https://kradle.ai/
- jbellis 1y agoswe-bench's bigger problems include (1) labs train on the test and (2) 50% of the tickets are from django; it's not a representative dataset even if all you care about is Python. I created a new benchmark from Java commits that are new in the past 6 months to add some variety: https://brokk.ai/power-ranking https://brokk.ai/power-ranking
- zelphirkalt 1y agoCan anyone tell me what is the difficulty in simply not having .git at all during a benchmark run? Why not simply remove anything that is not the code the benchmark runs on? Or just simple oversight?
- sigmoid10 1y agoCoding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.
- diggan 1y agoI think this issue is specifically about the agents looking at "future repository state" (according to the linked issue at least), so while looking at the history might be a normal method for solving issues, running `git log --all` to take a peek at the future which already includes the fix isn't very typical (yet?).
- fp64 1y agoWell, there's legacy code and/or horrible git history that also needs fixing at some point. Also I have witnessed how the history can send you down a wrong path. I don't agree that this is a good argument.
- ActionHank 1y agoThat is a weak argument. The point is to benchmark against a human solving a problem. Typically these problems are posed as a question or a blank project, without that history. You are arguing for a an apples to oranges comparison because the LLM performs better. Rather than a realistic comparison.
- sigmoid10 1y agoYou apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world software development examples from actual PRs in popular repos. These very likely had humans use all the tools at their disposal as well (e.g. web search, git commands, code execution). Assuming an LLM could have solved these just by looking at a piece of code turns out to be very myopic. [1] https://arxiv.org/html/2310.06770v3 https://arxiv.org/html/2310.06770v3