3 ms·
Basically all the examples of LLM's building impressive things have been because they have human written tests to base the implementation on. If you have an LLM
by slopinthebag 2mo ago
Basically all the examples of LLM's building impressive things have been because they have human written tests to base the implementation on. If you have an LLM write the tests the results are far less impressive or valuable.
- bharatsuthar 2mo agoYes and LLMs are known to cheat on tests written by them.
- slopinthebag 2mo agoIt's not always cheating either. They aren't intelligent, so they don't actually understand the purpose of the tests or can build them to define the actual semantics of the problem space. It's literally just next-token prediction based on the codebase and prompt. Cheating implies that they have agency, and ironically agents don't.
- andai 2mo agoLast year when they added computer use to Claude web I was excited to try it out. I just asked it for a code snippet and it ended up setting up a whole repo in a docker container or something. Even volunteered a test suite. This genuinely amazed me. ...until I checked the tests. It was just console.log("Tests passed!") AGI 2027
- simonw 2mo agoHave you seen that recently? I used to see that happen, but I've not caught it with the more recent (Opus 4.5+, Fable 5, GOT 5.5/5.6) models.
- jryle70 2mo agoShow those known cases? When and which models?