5 ms·
My fav coding benchmark for frontier models is to build a simple RTS game in one file (js/html/css). Claude Code with Opus 4.8 in ultracode mode nailed it, the
by senko 5mo ago
My fav coding benchmark for frontier models is to build a simple RTS game in one file (js/html/css). Claude Code with Opus 4.8 in ultracode mode nailed it, the best result so far:
https://bsky.app/profile/senko.net/post/3mmwnrkwboc2v https://bsky.app/profile/senko.net/post/3mmwnrkwboc2v
The prompt was: Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
- jclay 5mo agoIt almost appears as if the code was minified. The variable names are short and formatting looks like it's written to minimize whitespace. Did it write it in this compact format all on it's own?
- rphv 5mo ago"Readability by humans" may no longer be as important as it once was.
- lionkor 5mo agoMaybe it would benefit Anthropic if AI generated code worked, but wasn't readable by humans. That's a nice moat.
- seanw444 5mo agoProprietary stochastic compilers. Hooray.
- orphea 5mo agoOnly if LLMs will start to output object code, skipping text representation.
- selcuka 5mo agoGood variable names are still useful for LLMs to understand context when refactoring.
- rafram 5mo agoLLMs are already bad at reusing existing logic/resources/components, even when they have obvious names. Unreadable code only makes it worse.
- andai 5mo agoA friend sent me something he vibe coded which included a massive webassembly blob in the HTML file. My friend is not a programmer so he was not able to explain to me how it did that.
- unconscionable 5mo agoClaude Design export.
- senko 5mo agoYeah looks extremely compact. I didn't instruct it or told it to use as few lines of code or characters or nothing of the sort. Not sure why it did that. Its own rationale (which is highly suspect, but the only lead I have) is that it defaults to dense style if it has to write a file in a single go. May be a kernel of truth somewhere in there.
- bombcar 5mo agoAnd much code on the web “in production” is minimized.
- AdamN 5mo agominified is fewer tokens than the human-readable version that we would write. It only really makes sense to write in minified js - it's also where alot of code in the wild is since every production site minifies their js which is then consumed by training.
- syspec 5mo agoI just had Opis 4.8 code up something and actually that's exactly how it coded it! It looked gross and minimized, the result was awesome but the code looked pretty awful visually
- dilap 5mo agoDoesn't look minified, just very dense, almost like progcomp code. First time I've seen an LLM spit out that style of code, I'm impressed!
- apitman 5mo agoI like that benchmark. You should throw the results up on GitHub pages so people can try out the games.
- brandly 5mo agoYeah! Host on GitHub pages, so it's easy to click a link and play!
- senko 5mo agoGreat idea! I have a static server of my own, so here's my list (of all the tests I published so far): https://senko.net/vibecode-bench/ https://senko.net/vibecode-bench/
- paulirish 5mo agoWould love to see the prompts, too!
- jmtame 5mo agoSame!
- senko 5mo agoI've updated the page with the prompts, c/p-ing here: Minesweeper: Create a beautiful and fully functional Minesweeper clone in HTML/JS/CSS (all in one file). RTS: Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
- munksbeer 5mo ago
- l3x4ur1n 5mo agoPlayed it to the end. Pretty neat!
- elAhmo 5mo agoWhat is ultracode mode?
- tcoff91 5mo agoit's a brand new mode
- senko 5mo agoIt's a combination of reasoning effort (max) + enabling workflow that orchestrates multiple sub-agents. After some interrogation, here's how it organized the work: 1. Design workflow (rts-game-design, 11 agents, ~13 min) ran first, produced SPEC.md + DESIGN.md: 1.1. Proposals (3 parallel agents): each designed a complete RTS from a different philosophy 1.2 Judge (1 agent): evaluated all three and synthesized one unified design, committing to specific numbers (costs, HP, map size, etc.). 1.3 Deep-dives (6 parallel agents): each wrote an implementation-ready spec for one subsystem, all consistent with the chosen design 1.4 Synthesis (1 agent): merged the design + all six subsystem specs into one conflict-free master spec 2. Code-review workflow (rts-code-review, 25 agents, ~5 min), ran after the main agent had written and tested the code: 2.1 Review (6 agents, read-only Explore type): each scrutinized one dimension and returned structured findings. 2.2. Verify (19 agents): every finding got its own skeptic agent told to try to refute it, Result: 19 flagged → 16 confirmed, 3 rejected as non-bugs. What the main agent did in the main loop: - Wrote all ~2,400 lines of index.html by hand from the spec. - All browser testing/debugging via headless Chrome (I told it to use rodney by @simonw, love the tool :) - Applied all 16 fixes from the review and re-verified them in the browser.
- 33MHz-i486 5mo agoseems like a rube-goldberg esque way to consume 10x tokens. is this really where the industry is heading?
- derac 5mo agoWhy is it Rube Goldbergesque? The process doesn't seem arbitrary.
- digdugdirk 5mo agoDo you have a collection of these benchmark apps saved anywhere? I'd be particularly interested in seeing the relative cost differences between different models in a use case like this.
- senko 5mo agoI'm saving them all as gists here: https://gist.github.com/senko https://gist.github.com/senko But I just vibe-coded a handy list of all the tests I did (unfortunately without the commentary I usually leave in social media posts -- I should add those at some point): https://senko.net/vibecode-bench/ https://senko.net/vibecode-bench/
- jryan49 5mo agoKinda buggy, but impressively nonetheless. How long did it take?
- H3X_K1TT3N 5mo agoThanks for also sharing the prompt. I've been testing claude by asking it to make similar things, so it's useful to see what other people are doing. I do find it interesting that the visual style is pretty similar to things it's produced for me.
- dash2 5mo agoIf you look on the page of games, the style of chatgpt 5.5 is almost identical to the Claude style.
- ammar_x 5mo agoIs there some sort of a leaderboard for this test? Like if you'd give each of Opus 4.8 and GPT 5.5 a score out of 100, what would the scores be?
- senko 5mo agoThere isn't, as I wasn't going for strictness, more like a playful challenge in the vein of Simon's SVG pelican. Between the two, Opus 4.8 seems more capable. But, I suspect the harness plays a large role here. It's possible the result would be as good if Codex ran 10+ agents and spent an hour on it. OpenAI and Anthropic usually fast-follow each other, so I wouldn't be surprised if Codex got the same capability in a couple of days (and even an update to the model), then it'll be a better test. Sooo, let's say, winging it, vibes-based: 85% for Opus 4.8, 75% for GPT 5.5. Compare with GPT 5.3 (let's say 25%) here: https://senko.net/vibecode-bench/2026/rts-codex-5.3.html https://senko.net/vibecode-bench/2026/rts-codex-5.3.html
- shlewis 5mo agoHow much did it cost?
- senko 5mo agoToken equivalent of ~ $20 (I'm on a $100 Max sub).
- jmtame 5mo agoWow, that's impressive. Had fun playing it for 10 minutes locally. Found myself wanting to discover an enemy base :)
- Madmallard 5mo agoOkay now have it implement an authoritative server with reliable netcode and reconnection/disconnection logic, lobbies, and finding games, in-game chat, synchronized state around starting and ending games, resignations and such
- fireant 5mo agoWow that looks really impressive. Both the UI and the content looks good, the game is a bit buggy but still nice!
- veqq 5mo agowow
- egeozcan 5mo agoI've been tasking LLMs to write a traditional AI for a full vibe-coded RTS. I remove the human players and let them battle. I don't know why but I enjoy watching AI players battle so much :) In the repo, I even have a tournament script that calculates ELOs. So far, codex was unmatched. I'll try with Opus 4.8 too. https://egeozcan.github.io/unnamed_rts/game/ https://egeozcan.github.io/unnamed_rts/game/ https://github.com/egeozcan/unnamed_rts/blob/main/src/scripts/tournament.ts https://github.com/egeozcan/unnamed_rts/blob/main/src/script...
- mannanj 5mo agoThis is fun! I look forward to trying this out. Thanks for sharing!
- plutokras 5mo agoI'm happy to report that this game is very fun for natural intelligence entities too. :)
- egeozcan 5mo agoGlad that you liked it! Please fork or note the version you like because I keep breaking it in spectacular ways :)
- skolos 5mo agoHow many times did you try? Same model running multiple times can produce both very good and very bad results. In my benchmark even 10 runs often not enough to tell for sure if one model is better than another.
- senko 5mo agoUsually just once (and I did just one test for this particular one), but I've found the overall quality to be relatively consistent. There's too many confounding variables here, randomness just one of them. So I don't think of it as a definitive test (and reliable ordering), just another data point (along with actual benchmarks, pelicans, etc) to get a sense of the capabilities. For example, I managed to get something out of DeepSeek 4 Flash quantized to 2-bit with Antirez' DwarfStar, used via Pi. Almost kinda worked! :) Which makes me optimistic for using local models for serious development soon - I'd say within a year.
- RobinL 5mo agoNice, I recently found something like this was possible too. Gpt-5.5 one shotted the basic game, but then I added some ai generated graphics/sounds/music and asked it to write then up. It's a vocab building game, playable here (desktop only): https://rupertlinacre.com/vocab_annihilation/ https://rupertlinacre.com/vocab_annihilation/ It kind of blows my mind I can go from: 'I want a fun way to help him learn vocabulary, and I loved total annihilation as a kid' to 'heres a game that's he finds genuinely fun that helps him learn something ' in a few prompts.
- calebgcc 5mo agoI wonder if your previous prompts were part of the new RL fine tuning, and that’s why is now better at this specific question
- zuzululu 5mo agosome reason that website is showing up as high risk and i cannot view it , I had to open it from my mobile phone. it looks quite impressive, I don't use claude currently but hearing good things about it...from codex users ironically
- senko 5mo agoIs that for bsky.app (BlueSky platform) or my personal site (senko.net) where I put up the list of tests? What browser/device was that?
- seidleroni 5mo agoI am absolutely gobsmacked how good the game is! I didn't complete the level fully but I completed all but one of the tasks. This is both smooth and fun and I'm surprised that a modern LLM can do something this well, let alone in a single file. It makes me realize how much the goalposts have been moved. A few years ago (ChatGPT 2? 2.5?) wasn't even able to implement a small Python script I would expect a junior engineer to be capable of producing. Now we're getting the tools to do something like this. You should think about how to "rate" the outputs or at least provide your own rankings.