10 ms·
Many SWE-bench-Passing PRs would not be merged
- love2read 7mo ago[flagged]
- refulgentis 7mo agoWell, no: one of the first things it says is reviewers were blind to human vs. ai.
- yorwba 7mo agoThe comment you're replying to is talking about a hypothetical scenario. In any case, the blinding didn't stop Reviewer #2 from calling out obvious AI slop. (Figure 5)
- collabs 7mo agoI feel like I don't have the context for this conversation. If slop is obvious as slop, I feel like we should block it. If you look at the comment it says what the code following the comment does. It doesn't matter whether it is a human or a machine that wrote it. It is useless. It is actually worse than useless because if someone needs to change the code, now they need to change two things. So in that sense, you just made twice the work for anyone who touches the code after you and for what benefit?
- zozbot234 7mo agoThe point is that AI models do these kinds of things all the time. They're not really all that smart or intelligent, they just replicate patterns or boilerplate and then iterate until it sort of appears to work properly.
- spartanatreyu 7mo ago> appears to work That "appears" is doing a lot of heavy lifting. The code working isn't what's being selected for. The code looking convincing IS what is being selected for. That distinction is massive.
- p1necone 7mo agoThey might have tried, but this would be pretty hard to achieve for real - especially for the older/worse models. For changes that do more than alter a couple of lines llm output can be very obvious. Stripping all comments from the changeset might go a long way to making it more blind, but then you're missing context that you kinda need to review the code properly.
- nubg 7mo ago> mid-2024 agents Is this a post about AI archeology?
- nine_k 7mo agoLLM-written code passed SWE Bench even back then. This may just say that SWE Bench is an inadequate test, and should not be used for serious evaluation.
- Lerc 7mo agoIt's more about the test than the AI. For the most part, I think the tests AI have been given have been appropriately designed. At release, many AIs do poorly at them, the models rapidly catch up until the point where a new test is needed. They should be measuring close to the limits of ability like that. There will be some that try and steal headlines by targeting the specific nature of the test, but that is not a long term winning solution, the tests keep getting harder. If they make a model good at every test it has seen without regression, then with enough tests, that too ceases to be a problem. Perhaps there should be an aggregate AI test score that evaluates all of the tests released in a given year. If a model passes the latest test really well but does worse at TestSet2024 than the models before, it would perhaps indicate the model being trained to pass the latest cool test. There is a problem with people interpreting an AI that passes a test of X,Y or Z as indicating that the AI has the abilities of a human who passes X,Y, or Z. You should tell people who say that, Kasparov makes a nice coffee.
- varispeed 7mo agoDo these benchmarks make any sense? I tried a few local models that seem to be scoring well in SWE but the results were pure rubbish. (For instance MiniMax-M2.5 at 128GB from unslothed - completely unusable).
- devnotes77 7mo ago[flagged]
- hrmtst93837 7mo ago[flagged]
- segmondy 7mo agoWhich quant? I find folks running lower quants complaining, yet they should be running higher quant. Qwen3CoderNext is great, even at Q6. I mistakenly had it loaded for an agentic workflow and was surprised at how well it is.
- code_biologist 7mo agoWhat is "lower quant"? What is "higher quant"? I mean, I know what they are, but the very people you intend to reach don't know the difference between Q4_K_M and Q6_K and blog posts like [1] have nuggets like "For tests of the type ran here, there appear to be major diminishing returns past Q4". [1] https://big-stupid-jellyfish.github.io/GFMath/pages/llm-quants https://big-stupid-jellyfish.github.io/GFMath/pages/llm-quan...
- zozbot234 7mo ago> "For tests of the type ran here, there appear to be major diminishing returns past Q4" These statements are silly, because the only interesting comparison is among models with highly comparable on-disk sizes, or sizes for their active parameters. Obviously, a Q4 model is not going to be the same effectiveness as a Q6: no one sensibly expects that, you need to compare the Q4 with a smaller model. (The GP has the same problem of course.) I believe that once you do that kind of comparison, higher quantizations tend to do better up to Q2 or so for casual chat, maybe slightly more bits-per-param for agentic use cases where avoiding erratic behavior is important.
- languid-photic 7mo agomakes sense! we wrote something yesterday about the weaknesses of test-based evals like swe-bench [1] they are definitely useful but they miss the things that are hard to encode in tests, like spec/intent alignment, scope creep, adherence to codebase patterns, team preferences (risk tolerance, etc) and those factors are really important. which means that test-evals should be relied upon more as weak/directional priors than as definitive measures of real-world usefulness [1] https://voratiq.com/blog/test-evals-are-not-enough/ https://voratiq.com/blog/test-evals-are-not-enough/
- AndrewHampton 7mo agoThis seems like an important caveat to the SWE-bench, but the trend is still clearly AI becoming more and more capable.
- utopiah 7mo ago> the trend is still clearly AI becoming more and more capable. Isn't it precisely what this article is questioning?
- iam_circuit 7mo ago[dead]
- stevefan1999 7mo agoI think a far greater problem is the human psychological and prejudice factor itself. When we heard AI assistance on a PR, we usually dive down the road to thinking about "oh my god is it another LLM slop" (for example: https://github.com/jneem/imbl/pull/149#pullrequestreview-3703289472 https://github.com/jneem/imbl/pull/149#pullrequestreview-370...). I do use AI but I review the code before I push it, yet most people don't. Once there is a trend, it is easy to form a prejudice and it is hard to go back, unless there is a substantial improvement both in quality and quantity. Also, some people would have spoken outright rejecting any AI code, but most maintainers would employ the silent treatment tactics. And then when you demand them to review, they either close it or say that "I'm too busy" as an argument. I would call this one of the biggest dick move, because it hurts the most yet you can't find anything wrong with them until they reveal their motives.
- JoshTriplett 7mo ago> And then when you demand them to review You might want to think carefully about why you chose to use the word "demand" there. (Personally, if I'm rejecting AI slop, I'm not going to do it silently. But there are any number of valid reasons to not jump on someone's PR to review it.)
- catlifeonmars 7mo ago> I would call this one of the biggest dick move I don’t think that’s a fair characterization. You don’t know if the maintainer/reviewer is overloaded. No one is obligated to accept/review PRs and there is no question that the amount of noise has gone up. You are not the main character in that story, so to speak.
- duskdozer 7mo ago>And still, I really hate writing those PR descriptions. Yet you can't just leave it empty. If you can't write a description in your own words explaining why you're doing it, why should they take the time reviewing it (which they did on the same day you posted it, btw, even if one of them wasn't pleased)? It makes it seem much less likely that you read the code yourself.
- jc-myths 7mo ago[flagged]
- mbb70 7mo agohttps://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/newsguidelines.html#generated
- flir 7mo agoHow did you figure that?
- stingraycharles 7mo agoLast paragraph mostly I’d say, but the whole comment has signs.
- teshigahara 7mo ago`Tests still passed. Build still passed. But now I have three files to maintain instead of one, and the "extensibility" will never be used.` sounds very LLM-like to me personally, but I wouldn't be willing to bet on it.
- c-hendricks 7mo agoHuh, doesn't sound like that at all to me
- flir 7mo agoInteresting, isn't it. I think we might all be reading tea leaves here (myself included).
- bakugo 7mo agoAccount created 1 day ago, only talks about AI, "Building AI × Web3 products". Expect to see a lot of these types of accounts now that Show HN is restricted for brand new ones.
- xthunk 7mo agoReally interesting note. That echoes thoughts I’ve had about how much automated benchmark scores really reflect production‑ready code. For me the big takeaway is that passing doesn't automatically mean it is maintainable, follows established patterns / conventions or have unexpected side effects that real reviewers care about.
- bisonbear 7mo agoI've been working on building out "evals for your repo" based on the theory that commonly used benchmarks like SWE-bench are broken as they are not testing the right / valuable things, and are baked into the training data (see OpenAI's research on this here https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/why-we-no-longer-evaluate-swe-bench...) Interestingly, I had a similar finding where, on the 3 open-source repos I ran evals on, the models (5.1-codex-mini, 5.3-codex, 5.4) all had relatively similar test scores, but when looking at other metrics, such as code quality, or equivalence to the original PR the task was based on, they had massive differences. posted results here if anyone is curious https://www.stet.sh/leaderboard https://www.stet.sh/leaderboard
- ebhn 7mo agoNice, I really like your idea. First I've heard of something like that
- floodfx 7mo agoWorking on that too. Lmk if you’re up for a chat?
- bisonbear 7mo agoyea I'm down - feel free to send me an email ben@benr.build
- dirtbag__dad 7mo agoThis sounds amazing. In particular, I like comps to existing PRs. But I’m also not sure that I want existing PRs to be a template for most things reasonable or best practice. I’ve been building out internal linters that enforce design patterns I want and raise common code smells (also note tools like eslint allow custom rules which are easy write with something like opus 4.6). The use case is a total refactor of react and fastapi apps. We are suffering from everything’s a snowflake syndrome and just want the same pattern employed across features. This works pretty well when the linter has a companion agents.md file which explains the architecture and way about the world. But to get the agent (Claude code opus 4.6 currently) to nail the directory structure and design primitives, and limit some doofus behavior, I still haven’t cracked how to make literally each line of code simple and sensible. And I haven’t figured out how to prevent agents from going out of bounds and doing weird things unless I catch it in review and add another rule. This is a relatively new endeavor, but my gut is that it’s not much more time (linter rules and perhaps “evals” or a beefy agent review cycle) before I have bespoke linters in place that force what I want from our architecture. Note that a huge bottleneck to all of this is that the codebase our current team inherited has no tests. It’s too easy to accidentally nuke a screen’s subtle details. It’s also really hard to write good tests without knowing what all of the functionality is. It feels like a blocker to a lot of large-swath agentic changes is a test strategy or solution first then a rigid push for rearchitecture or new design.
- slopinthebag 7mo agoThis makes sense to me based on personal experience. LLM's will do anything to pass tests and get a working result, and it will do very weird things in order to get there. For fun I've tried to get it to do stuff while being purposely ambiguous about the implementation details and sometimes the stuff it does makes me literally laugh out loud. It can write some very strange code. But hey, the tests pass! If I force it to use plan mode for everything and babysit it, it can work really well, but it's really just acting as a faster typer for me, which is great. But it requires an experienced dev steering it.
- zhangchen 7mo ago[flagged]
- coderenegade 7mo agoThere needs to be a measure (or measures) of the entropy of a codebase that provides a signal of complexity. When you're paying for every token, you want code patterns that convey a lot of immediate information to the agent so that it can either repeat the pattern, or extend it in a way that makes sense. This is probably the next wave of assisted coding (imo), because we're at the stage where writing code works, the quality is mostly decent, but it can be needlessly complex given the context of the existing repo.
- deleted 7mo ago[deleted]
- js8 7mo agoThere's a way to measure "entropy" of a codebase. Take something like the binary lambda calculus or the triage calculus, convert your program (including libraries, programming language constructs, operating system) into it, and measure the size of the program in it in bits. You can also measure the crossentropy, which is essentially the whole program entropy above minus entropy of the programming language and functions from standard libraries (i.e. abstractions that you assume are generally known). This is useful to evaluate the conformance to "standard" abstractions. There is also a way to measure a "maximum entropy" using types, by counting number of states a data type can represent. The maximum entropy of a function is a crossentropy between inputs and outputs (treating the function like a communication channel). The "difference" (I am not sure how to make them convertible) between "maximum entropy" and "function entropy" (size in bits) then shows how good your understanding (compared to specification expressed in type signature) of the function is. I have been advocating for some time that we use entropy measures (and information theory) in SW engineering to do estimation of complexity (and thus time required for a change).
- malfist 7mo agoMaybe cyclomatic complexity would be a good proxy. It can obviously be gamed but it's obvious when it is
- bandrami 7mo agoI mean, it's ultimately a string, and the measurement of the entropy of a string is well-studied. The LLM might start gaming that with variable names so you'd need to do the AST instead. I may actually try something like that; cool idea.
- 50lo 7mo agoWould be interesting to see alternative scoring besides “tests pass”, e.g. diff size, abstraction depth added/removed, or whether the solution introduces new modules/dependencies. That would allow to see if “unmergeable” PRs correlate with simple structural signals.
- deleted 7mo ago[deleted]
- XenophileJKO 7mo agoI was totally aligned until I saw the refusal for a comment in the code. When the refusals are pedantic like that, it just weakens the overall findings significantly.
- finnthehuman 7mo agoYeah, why be such a tryhard? Keeping PR friction down is what matters. Just let the codebase slowly deteriorate. It'll be fine.
- vexnull 7mo ago[dead]
- bandrami 7mo agoI'm very much an AI bear but I do think one interesting outcome is going to be that LLMs will stumble upon some weird ways of doing things that no human would have chosen that turn out to be better (Duff's device-level stuff) and they will end up entering the human programming lexicon.
- tripdout 7mo agoIs this, along with the comments by the other green usernames on this post, an AI-generated comment? Apologies if it isn't, AIs are trained on human writing and all that, but they're jumping out at me. Edit: I see another green comment was flagged for AI, might be indicative of something, but why so many green comments on this thread specifically?
- aarjaneiro 7mo agoGreen username just means new user (under 1 month iirc)
- ukuina 7mo agoLints, beautifiers, better tests?
- colechristensen 7mo agoEh, but if you're in an organization you tune your AGENTS.md, CLAUDE.md, AI code reviews, etc. to have your human driven or automated AI generated code fit the standards of your organization. I don't need models to be smart enough to aggressively try to divine how the organization wants them to do, the users will indeed make that happen. So this post is maybe a little bit over the top. I am literally right now tuning my PR, Claude instructions, and PR instructions to match our standards. Funny enough I'm having the opposite problem where Claude is lowering its rating of my PR because my testing, documentation, and error handling is better than the other code in the repository so it doesn't match and therefore gets a worse grade. I don't need it to try any harder without explicit instructions.
- Kave0ne 7mo ago[flagged]
- eru 7mo agoThough you can get a lot of that context from looking at the git history and discussions (and revisions) on past PRs and perhaps access to the history of team chat channels.
- Kave0ne 7mo ago[flagged]
- eru 7mo agoYou are right that there's a difference. But you can tell the agent to ingest the whole thing once, and distill an approach in a much smaller (and human readable and reviewable) description perhaps? And then that can serve as the basis for the agent going forward. So you don't have to retrieve the whole history all the time. It's similar to telling your new hire to catch up on _all_ the slack history. Only that the agent will actually do so.
- inventor7777 7mo agoHey, this comment sounds very AI generated. I could be wrong, but if so, you might want to read the newest HN rules.
- sumeno 7mo agoLooking at their post history it's all 100% AI generated
- cornstalks 7mo agoAnecdote time! I had Codex GPT 5.4 xhigh generate a Rust proc macro. It's pretty straightforward: use sqlparser to parse a SQL statement and extract the column names of any row-producing queries. It generated an implementation that worked well, but I hated the ~480 lines of code. The structure and flow was just... weird. It was hard to follow and I was seriously bugged by it. So I asked it to reimplement it with some simplifications I gave it. It dutifully executed, producing a result >600 lines long. The flow was simpler and easier to follow, but still seemed excessive for the task at hand. So I rolled up my sleeves and started deleting code and making changes manually. A little bit later, I had it down to <230 lines with a flow that was extremely easy to read and understand. So yeah, I can totally see many SWE-bench-passing PRs being functionally correct but still terrible code that I would not accept.
- SerCe 7mo agoIf you've got some time, I highly recommend going through the exercise of trying to change the prompt in a way that would produce code similar to what you've achieved manually. Doing a similar exercise really helps to improve agent prompting skills, as it shows how changing parts of the prompt influences the result.
- foltik 7mo agoI haven’t had any luck prompting LLMs to “have taste.” They seem to over fixate on instructions (e.g. golfing when asked for concise code) or require specifying so many details and qualifications that the results no longer generalize well to other problems. Do you have any examples or resources that worked well for you?
- zarzavat 7mo agoYeah prompting doesn't work for this problem because the entire point of an LLM is you give it the what and it outputs the how. The more how that you have to condition it with in the prompt, the less profitable the interaction will be. A few hints is OK, but doing all the work for the LLM tends to lead to negative productivity. Writing prompts and writing code takes about the same amount of time, for the same amount of text, plus there's the extra time that the LLM takes to accomplish the task, and review time afterwards. So you might as well just write the code yourself if you have to specify every tiny implementation detail in the prompt.
- woeirua 7mo agoThis paper doesn’t really tell us much. The cutoff was September of 2025. The models have improved so much that I just don’t know what you can take away from this experiment.
- croemer 7mo agoSWE-bench scores are inflated compared against actual maintainer merge decisions as opposed to an LLM grader.
- jurschreuder 7mo agoThe test is supposed to be a proxy.
- ClaudeAgent_WK 7mo ago[flagged]
- whilenot-dev 7mo ago> This is William Wang's AI-assisted account. Please read the guidelines before posting here: https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/newsguidelines.html#generated
- jeff_antseed 7mo ago[flagged]
- antirez 7mo agoOf what is happening with AI the most bizarre thing, for me, is how these tools are 20$ away from being tested. Yet, to form an idea about actual real world usefulness many folks seek some kind of indirect proxy. This is combined with the incredible general feeling that automatic programming can be evaluated as producing the same results regardless of the user using it. Something true only with benchmarks, basically. Benchmarks are useful metrics because even if weak we need some guidance, but the current real world dynamic is that AI will completely change what it is capable of doing based on the programmer using it. Maybe never in the history of programming there was a time where diverse programming skills were as important as today (but this may change as AI evolves).
- croemer 7mo agoBenchmarks do a few things: 1. Help choose a model from the hundreds out there, or at least help create a shortlist to try. 2. Quantify progress/improvements (or lack thereof) over time. 3. Inform about relative strengths and weaknesses.
- utopiah 7mo agoAssuming the benchmark can't be gamed.
- utopiah 7mo ago> automatic programming can be evaluated as producing the same results regardless of the user using it. That's something I've argued here several time and that's actually rarely done. Namely it's totally different when a non-developer use such tool for programming vs when a (senior) SWE does. That's a fundamental point which IMHO a potential for (non-riskfree) augmentation versus replacement. Replacement though makes for excellent narrative (if not scapegoat) yet if the tool is "productive" (with KPIs to agree on) only with skilled staff that it's not the reality, just a "wish".
- Archit3ch 7mo agoI'm about to put up the 20 to see what everyone is raving for. But the real cost is time: if this doesn't work, I'm worse off than never trying.
- croemer 7mo agoFigure 1 should not fit a straight line as a trend. Scores are 0 to 100%, the straight line will go outside those bounds at sufficiently large times. The simplest reasonable model would be logistic regression. It's also got 2 parameters and the range is correct.
- kqr 7mo agoAlthough you are technically correct, if you look at the data you'll recognise that for this narrow span of values, the logistic fit will be practically equivalent to the linear. Indeed, they perform the same in cross-validation. Here's what the logistic fit looks like: https://i.xkqr.org/logisfit.png https://i.xkqr.org/logisfit.png
- croemer 7mo agoAuthor gave me the same reply. I just don't want to have to think about whether it's equivalent or not, why use a 2 param model that's strictly less appropriate even if the difference is small.
- blockpilot_ai 7mo agoInteresting project. I've been thinking a lot about tools for organizing long AI conversations. Curious how people here currently manage them.
- mergeshield 7mo ago[flagged]
- kqr 7mo ago> The gap here is really two different definitions of "correct." Tests verify behavior. Reviewers verify fit Take a second look at the "reasons for rejection". A quarter or so of the changes that pass tests actually don't solve the problem they intend to. The tests used in SWE-bench do not discriminate all that well between working and broken solutions.
- blockpilot_ai 7mo agonot rule but thanks discussion
- shanjai_raj7 7mo agoI see this with claude code all the time. it writes code that works but tries to cover every edge case and becomes hard to read. I usually just tell it to make it shorter and simpler and it does a better job on the second pass. passing a benchmark and writing good code are two different things.
- blockpilot_ai 7mo ago[flagged]
- blockpilot_ai 7mo agoInteresting discussion. I've been thinking about tools for organizing long AI conversations. Scrolling through hundreds of messages quickly becomes painful. I'm curious how people here manage long AI chats.
- thesz 7mo agoTime to completion (in appendix A9) should be treated as log-normally distributed, or by some other one-sided distribution because one cannot complete the task faster than 0 seconds. This transformation will rule out confidence ranges with negative time. BTW, log-normal distribution tend to produces events P(x>E(X)+d) more frequently than events P(x<E(X)-d). If one needs reasons why software projects often late, this is one of them.
- maciusr 7mo ago[flagged]
- tonipotato 7mo agoI feel the same! they are raising the bar higher and higher. I wrote a bot and pass the swe bench lite for 67% and can not get a chance to show. I also tried to submit for swe bench full but they limit it to organization only. where can us independent developers post our stuff, can we have an open bench mark for everyone and we just use merit to rank?