20 ms·
$500 GPU outperforms Claude Sonnet on coding benchmarks
- memothon 7mo agoI'm always skeptical because you can make it pass the benchmarks, then you use it and it is not practically useful unlike an extremely general model. Cool work though, really excited for the potential of slimming down models.
- yogthos 7mo agoYou obviously have to try it out to see how it works for you, but the trick they use is pretty clever. When you ask an AI to write code, it doesn’t always get it right. Sometimes the code has bugs, sometimes it misunderstands the problem entirely. A naive way to address that is to generate a few solutions and test each one. The odds that at least one works go way up. ATLAS generates multiple attempts, running each through a test suite. Each retry also gets told what went wrong with the previous attempt, so it can try to avoid the same mistake. But this can be pretty slow since you have to run the code in an isolated environment, check the outputs, wait for it to finish. Doing that for every candidate quickly adds up. So ATLAS has another shortcut for avoiding unnecessary testing. Instead of simply generating solutions and testing all of them, it tries to predict which one is most likely correct before running any tests. ATLAS also asks the model for an embedding of what it just wrote which acts as a fingerprint. Two similar pieces of code will produce similar fingerprints. A well-written, confident solution will produce a different fingerprint than a confused, buggy one. These fingerprints get fed into a separate, much smaller neural network called the Cost Field. This little network was trained ahead of time on examples where they already knew which solutions were correct and which were wrong. It learned to assign a score to each fingerprint. Correct solutions get a low score and incorrect ones get a high one. So the process is to generate multiple solutions, get their fingerprints, score each one, and pick the lowest. Only that one gets tested. The Cost Field picks correctly about 88% of the time according to the repo.
- zar1048576 7mo agoReally intriguing set of techniques to improve accuracy by generating multiple solutions. Even with the work to predict the most likely solutions, it's not clear to me based on the description how this could all be done efficiently. Would definitely be really impressive if it pans out on real-world use cases. Will look to kick the tires on this if I can get some time.
- yogthos 7mo agoSeems like the key insight is to train a small model that acts as a heuristic for embeddings that resemble quality code. I imagine a lot depends on how well this model is trained. And you could probably create specialized versions for different languages and domains. Another interesting approach could be to use this set up with a language like Clojure or Common Lisp which facilitates interactive development. If you could hook up the agent directly to a REPL in a running program, then it could run tests with a lot less overhead.
- xyzzy123 7mo agoI'm super confused. The small model "cost field" `rag-api/geometric_lens/cost_field.py` was trained on PASS_TASKS like "Write a function that counts vowels in a string." and FAIL_TASKS like "Write a function that converts a regular expression string to an NFA using Thompson's construction, then converts the NFA to a DFA.". So it seems like it's a difficulty classifier for task descriptions written in English. This is then used to score embeddings of Python code, which is a completely different distribution. Presumably it's going to look at a simple solution, figure out it lands kinda close to simple problems in embedding space and pass it. But none of this helps you solve harder problems, or distinguish between a simple solution which is wrong, and a more complex solution which is correct.
- yogthos 7mo agoI think the goal is to have a light heuristic that helps find plausibly useful solutions. They're still going to go through a testing phase as a next step, so this is just a very simple filter to decide what's even worth testing.
- kimixa 7mo agoI find it's often very language and sector dependent. I still see a massive difference in systems programming (normally c++ and rust) between any open model I've tried and something like sonnet 4.5 (not really tried 4.6). And honestly, even the big models (like Opus 4.6) struggle in many cases. Perhaps these things aren't well represented in the training data for these open models? Every local model I've tried (minimax2.5, GLM-4.7, Quen3, 3.5 and -coder variants) spend so much time trying to get something syntactically sensible and accepted by the compiler that when they've finished they barely seem to have any "momentum" left to actually solve the problems, as pretty much anything but the most trivial change ends up in another loop of actually trying to get it working again, often losing the intent of that change in the process. My fear is that the solution here, having multiple instances all making the same changes for later comparison, would spend a huge amount of time beating it's head against compiler errors, types, memory allocation (NO DON'T JUST SPRINKLE IN A FEW MORE RAW "new" KEYWORDS DAMMIT) before it even gets to the "logic". Having plenty of local GPU power I'd love to be able to actually use that, and I'm already wary about some of the training data use and it's interactions with the license of the code I'm "sending" to the cloud models...
- vidarh 7mo ago> Perhaps these things aren't well represented in the training data for these open models I know from first-hand experience that at least a couple of the SOTA providers use third-party providers for supervised finetuning with instructions that are heavily geared towards a specific set of languages as well. But of course the base dataset from the major providers is likely to be sufficiently better that it matters less, and the big models are good enough at carrying over training that it at least seems like extra training on the core languages they care about at least somewhat carries over (you see this with natural language too - they do really well for many minor languages that make up a miniscule proportion of the training data). (I won't say much more regarding the SFT/RLHF work due to NDAs - plural; I know who one of the providers is; I don't know who the one or more others are as the intermediary I did some work for obscured it well enough that I couldn't really violate the NDA even if I wanted to)
- MattRix 7mo agoI think this is because when you shrink it down, the model ends up space constrained and each “neuron” ends up having to do multiple duties. It can stil be tuned to perform well at specific tasks, but no longer generalizes as well. It’s somewhat unintuitive but models that are larger are often simpler than smaller ones for this same reason.
- negativegate 7mo agoAm I still SOL on AMD (9070 XT) when it comes to this stuff?
- dangus 7mo agoWell, this specific solution was only set up on specific hardware, and is Nvidia dependent, as the readme stares. That doesn’t mean the 9070XT can’t do AI stuff, quite the opposite. ROCm gets better all the time. There are many AI workloads you can do on AMD cards. Is it a card I would choose if I was primarily working on AI? Absolutely not. But it is the card I own and it’s been a great value for gaming.
- dannyw 7mo agoUnfortunately AMD is much worse with supporting AI features like FSR4 on older hardware generations, despite the capability and leaked INT8 models being there. Totally unlike NVIDIA. It’s absurd I have to use open source programs to get INT8 FSR4 support.
- patshead 7mo agoNo, but yes? OmniCoder 9B at Q6 fits on my 9070 XT with 200k+ tokens of context, and it works pretty well with OpenCode. It is for sure the best local model that I've managed to squeeze onto my GPU, and it even works at 120k context at Q3 on an 8GB RX 580 GPU. I can't imagine trying to using this model on either GPU for real work. I can use much bigger and faster models on the $3 Chutes subscription or $10 OpenCode Go subscription. Even so, I am still excited. I don't feel like there was even a model worth using with a tool like OpenCode 6 to 9 months ago. I like the way things are heading, and I am looking forward to seeing how capable coding models of this size are in another 6 to 9 months!
- hrmtst93837 7mo ago[flagged]
- 0xbadcafebee 7mo agoNo? You can run any model that fits in its VRAM, and you can run larger models with layer/MoE offloading. Ask an AI what the best models you can run on that card are, then ask it for newer models than that. Ask what tuning options to pass to llama.cpp, and what the auto-tuning options are. Use ROCm builds. It looks like your card has 16GB VRAM? Start with Qwen 3.5 9B Unsloth GGUFs (UD-Q6_K_XL) and branch out from there.
- riidom 7mo agoNot a word about the tok/sec, unfortunately.
- arjie 7mo agoIt won’t be meaningful considering the architecture: it’s a harness around the model that generated multiple solutions in multiple passes using the test to measure compliance and repair broken solutions. The resulting program won’t be streamed to you because it has existed for minutes as it goes through the cycle. It’s more for an asynchronous use-case. I, too, was interested because I am always eager to use local models in my claw-like. It looks like this could be useful for an async portion of the harness but it wouldn’t work in interactive contexts. Very cool ensemble of techniques, particularly because they’re so accessible. I think I will use this form for reusable portions of web browsing functionality in my personal agent.
- Octoth0rpe 7mo ago> A single patched llama-server runs on K3s, providing both generation with speculative decoding (~100 tok/s) There seems to be at least some detail on that point.
- superkuh 7mo agoIf anyone else was hoping this was using Q8 internally and that converted to Q4 it could fit in 12GB VRAM: unfortunately it's already at Q4_K_M (~9GB) and the the 16GB requirement is from other parts not a 14B@8bit+kv cache/etc you might guess.
- selcuka 7mo agoIt's a race to the bottom. DeepSeek beats all others (single-shot), and it is ~50% cheaper than the cost of local electricity only. > DeepSeek V3.2 Reasoning 86.2% ~$0.002 API, single-shot > ATLAS V3 (pass@1-v(k=3)) 74.6% ~$0.004 Local electricity only, best-of-3 + repair pipeline
- mikestorrent 7mo ago> cheaper than the cost of local electricity only. Can you explain what that means?
- simonw 7mo agoI think they mean that the DeepSeek API charges are less than it would cost for the electricity to run a local model. Local model enthusiasts often assume that running locally is more energy efficient than running in a data center, but fail to take the economies of scale into account.
- littlestymaar 7mo agoI guess it mostly comes from using the model with batch-size = 1 locally, vs high batch size in a DC, since GPU consumption don't grow that much with batch size. Note that while a local chatbot user will mostly be using batch-size = 1, it's not going to be true if they are running an agentic framework, so the gap is going to narrow or even reverse.
- eru 7mo agoWell, different parts of the world also have different electricity prices.
- littlestymaar 7mo agoUsually not multiple orders of magnitude difference though.
- 7mo ago
- ozgurozkan 7mo ago[dead]
- sayYayToLife 7mo ago[dead]
- mmaunder 7mo agoI’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimize your apps and prompts to reduce output tokens.
- XCSme 7mo agoYup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moonshotai-kimi-k2-5-medium/z-ai-glm-5-medium/google-gemini-3-1-flash-lite-preview-medium/ https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...
- wizee 7mo agoIt’s worth also comparing Qwen 3.5, it’s a very strong model. Different benchmarks give different results, but in general Qwen 3.5, GLM 5, and Kimi K2.5 are all excellent models, and not too far from current SOTA models in capability/intelligence. In my own non-coding tests, they were better than Gemini 3.1 flash. They’re comparable to the best American models from 6 months ago.
- XCSme 7mo agoI used qwen 3.5 plus in production, it was really good at instruction following and tool calling.
- deleted 7mo ago[deleted]
- redoh 7mo ago[flagged]
- vidarh 7mo agoWhile I like these models, if you're getting similar results to SOTA models from 6 months ago, I have to question how far you pushed those models 6 months ago. It is really easy to find scenarios were these models really underperform. They take far more advanced harnesses to perform reasonably (and hence the linked project). It's possible to get good results out of them, but it takes a lot of extra work. I badly want to shift more of my work to them, and I'm finding ways of shifting more lower-level loads to them regularly, but they're really not there yet for anything complex.
- limoce 7mo agoThe title should be "Adaptive Test-time Learning and Autonomous Specialization".
- deleted 7mo ago[deleted]
- felixagentai 7mo ago[flagged]
- dang 7mo agoWe've banned this account. Please don't post automated comments to HN. https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/newsguidelines.html#generated
- emp17344 7mo agoYet more evidence that the harness matters more than the model.
- bustah 7mo ago[flagged]
- wiradikusuma 7mo ago[dead]
- 0xbadcafebee 7mo agoThis is specifically an experiment using ablation and multiple passes to improve the end result. Other techniques have been found that do this (like multiple passes through the same layers). But this technique - for this one specific model - seems to be both more performant, but also takes much longer, and requires more complexity. It's unlikely most people would use this technique, but it's interesting.
- eddie-wang 7mo ago[dead]
- itigges22 7mo ago[dead]
- b3ing 7mo agoWill open source or local llms kill the big AI providers eventually? If so when? I can see maybe basic chat, not sure about coding and images yet
- throwaway85825 7mo agoFinancial gravity will kill them when returns don't match stratospheric expectations.
- bluefirebrand 7mo agoI hope so too, but I think it's wishful thinking. Be prepared for the mother of all financial bailouts from the world governments to make sure that doesn't happen
- hollerith 7mo agoI can understand why banks got bailed out by the US gov in 2008, but why would a government feel the need to bail out AI labs? I hope you are not going to say, "to avoid a global recession or depression caused by the popping of the AI bubble". That would be unnecessary and harmful (in its second-order effects), and governments do have advisors who are competent enough in economics to advise against such a move.
- nyargh 7mo agoBold of you to assume competency will overpower politics in our current era.
- hollerith 7mo agoSo far, the country I know best, the US, has been competent enough to avoid massive corporate bailouts except the aforementioned banks in 2008 and GM. The bailout of GM was not motivated by a desire to avoid a recession when a bubble pops. If the AI labs become very influential and powerful, Washington might nationalize them, but that would be very different from bailing them out because they have become unprofitable and cannot attract additional investment from the private sector.
- electroglyph 7mo agowhat's with the weird "Geometric Lens routing" ?? sounds like a made up GPTism
- ottah 7mo agoFeels very pseudo academic.
- bloppe 7mo agoGenerating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.
- d0963319287 7mo ago[flagged]
- sigmoid10 7mo agoProbably want to look at SWE bench pro or terminal bench 2. They cover these longer horizon tasks that need more than just writing a bit of code in one file. And SWE bench pro in particular it is not yet saturated like many other common benchmarks. Normal SWE and LCB are not really useful anymore because they are already being gamed hard so the developers can quote high numbers in a repo readme or press release.
- Bombthecat 7mo agoOh yes! I let my environments now be built by agents via kubectl / helm and let them debug issues. It's amazing! Saves hours of work! I create the basic helm configd settings etc and when there is a conflict or something not working I let an agent fix it!
- bartread 7mo agoI agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally deleted data). Of course, this is not hard to do manually but is is a bloody chore and tends toward error prone. But the agent made short work of it, for which I was very grateful.
- CraigJPerry 7mo ago
- 15minutemail 7mo ago74% on LCB from a single 5060 Ti. I've been paying Anthropic per task and this guy is running it on electricity money, 20 minutes per task is rough for anything interactive though.
- subroutine 7mo agoAt 20 min per task you might as well code it yourself. Bill James needs to write a book on saber-metrics for LLM benchmarks.
- Razengan 7mo agoClaude Code has been bleh or meh at best in my experience. There's so many posts on HN fawning about it lately that it could only be a guerrilla marketing campaign.
- spiderfarmer 7mo ago"I don't get it. Everyone else is wrong."
- Razengan 7mo ago"There's no such thing as astroturfing." ok I use Codex regularly and Claude is shit in comparison, from its constant "Oops you're right!!" backtracking to its crap Electron app (if their AI is so good why can't they make a fucking native app for each OS?) Hell right freakin now I asked it to implement something and got a weird "Something went wrong" API error
- spiderfarmer 7mo ago"Shit", "Crap", "Fucking", "Hell", "Freaking". Maybe you're too easily frustrated. Or your existing code reads like your comments.
- Razengan 7mo agoMaybe you haven't tried any other AI product with an actual preexisting project. Or blindly trust every BS Claude feeds you. I haven't had any such frustrations with Codex Claude is specially annoying because of their submarining and people thinking it's the best
- spiderfarmer 7mo agoI use both, read what I need to read and fix small issues myself. Both Agents are pure magic and none of their issues warrant a tantrum on a public forum.
- tgiba 7mo agoDespite skepticism I love to see experiments like that. If we all are able to run an open source model locally on mid-high end machines I'd be very happy.
- bdbdbdb 7mo agoThis is the kind of innovation I love to see. The big AI companies days are numbered if we can have the same quality in house
- sznio 7mo agoOn that topic, anyone here got a decent local coding AI setup for a 12GB VRAM system? I have a Radeon 6700 XT and would like to run autocomplete on it. I can fit some models in the memory and they run quick but are just a tad too dumb. I have 64GB of system ram so I can run larger models and they are at least coherent, but really slow compared to running from VRAM.
- mongrelion 7mo agoNot the answer that you are looking for, but I am a fellow AMD GPU owner, so I want to share my experience. I have a 9070 XT, which has 16GB of VRAM. My understanding from reading around a bunch of forums is that the smallest quant you want to go with is Q4. Below that, the compression starts hurting the results quite a lot, especially for agentic coding. The model might eventually start missing brackets, quotes, etc. I tried various AI + VRAM calculators but nothing was as on the point as Huggingface's built-in functionality. You simply sign up and configure in the settings [1] which GPU you have, so that when you visit a model page, you immediately see which of the quants fits in your card. From the open source models out there, Qwen3.5 is the best right now. unsloth produces nice quants for it and even provides guidelines [2] on how to run them locally. The 6-bit version of Qwen3.5 9B would fit nicely in your 6700 XT, but at 9B parameters, it probably isn't as smart as you would expect it to run. Which model have you tried locally? Also, out of curiosity, what is your host configuration? [1]: https://huggingface.co/settings/local-apps https://huggingface.co/settings/local-apps [2]: https://unsloth.ai/docs/models/qwen3.5 https://unsloth.ai/docs/models/qwen3.5
- kroaton 7mo agoFor autocomplete, Qwen 3.5 9B should be enough even at Q4_k_m. The upcoming coding/math Omnicoder-2 finetune might be useful (should be released in a few days). Either that or just load up Qwen3.5-35B-A3B-Q4_K_S I'm serving it at about 40-50t/s on a 4070RTX Super 12GB + 64GB of RAM. The weights are 20.7GB + KV Cache (which should be lowered soon with the upcoming addition of TurboQuant).
- mongrelion 7mo ago
- Temporary_31337 7mo agothe headline is pretty stupid - compares a model to a GPU that models run on. Somewhere in that data centre, some part of Sonnet infferencing runs on a 900$ GPU or maybe even cheaper Google tensor
- LuisvelAI 7mo ago[flagged]
- DanielHall 7mo agoThese small models, having been fine-tuned for the test, achieve frighteningly high scores, yet perform abysmally in real-world scenarios.
- mergeshield 7mo ago[flagged]
- josefritzishere 7mo agoThe core problem of AI remains unresolved, with no conceivable path to solvency. The issue is that AI isn't very good. It's OK, sometimes under very narrow criteria. But providing AI in reality very costly. Vague promises of it magically becoming better remain, very optimistic at best and still provide no route to solvency.
- yakbarber 7mo agoAI would have a least worded your comment better
- bilekas 7mo agoWhere is a RTX 5060 Ti 16 GB 500$? Edit : The 8GB seems to hit this price but 16 not so much.
- hedgehog 7mo agoThey were $450 or so until recently, now... good luck.
- paxrel_ai 7mo ago[dead]
- alkonaut 7mo agoGreat, it became a $1000 gpu while you were reading that.
- dwa3592 7mo agoI wonder if it's working out for the benchmark problems only? one expensive and hard lesson we will learn overtime is that you can't compress generality beyond a point.
- Aurornis 7mo agoThis AI-written project is running its own LiveCodeBench on a completely different methodology. The AI-written notes even admit it: > ATLAS scores are from 599 LCB tasks using the full V3 pipeline (best-of-3 + Lens selection + iterative repair) on a frozen 14B quantized model or "pass@k-v(k=3)". Competitor scores are single-shot pass@1 (zero-shot, temperature 0) from Artificial Analysis on 315 LCB problems -- not the same task set, so this is not a controlled head-to-head. Instead of following the LiveCodeBench methodology, it's a harness that spins up a sandbox and spends a long time testing and refining the solution. If you did the same for Sonnet, GPT5.4, or other models they would also get significantly higher scores and they'd do it faster. The AI-coded README is also full of signs of vibecoded slop like the discoveries that some of the complex structures implemented were not actually being used or contributing anything to the output.
- johnwhitman 7mo ago[flagged]
- rldjbpin 7mo ago> coding benchmarks > V3 phases were designed and tuned for LiveCodeBench. only compared on the above benchmark, while this has been identified and being improved for the next version. curious to see how it compares across the board against the base model (Qwen3-14B-Q4_K_M)