4 ms·
It feels to me that there may be even more potential in flipping this idea around - human coders write tests to exact specifications, then an llm-using coding s
by dwringer 2y ago
It feels to me that there may be even more potential in flipping this idea around - human coders write tests to exact specifications, then an llm-using coding system evolves code until it passes the tests.
- mtndew4brkfst 2y agoPeople who are terminally capitalist have been salivating over this idea basically since this hype curve first started. Someone made a solid joke about it as far back as 2016: https://www.commitstrip.com/2016/08/25/a-very-comprehensive-and-precise-spec/ https://www.commitstrip.com/2016/08/25/a-very-comprehensive-...
- willsmith72 2y agoWell yeah, TDD. Many companies already work this way, and the LLMs are ok at generating the test-passing code. From my experience though, it's better at writing tests (from natural language)
- dwringer 2y agoMy concern with having the LLM's write tests is that it's hard to be convinced that they've written the right tests. Coupling human TDD with a genetic algorithm of some sort that uses LLM's to generate candidate populations of solutions, one could be assured that once a solution gets far enough through the tests [assuming one ever does], it is guaranteed to have the correct behavior (as far as "correct" has been defined in the tests).
- willsmith72 2y agoyes, definitely it's a concern. the idea with llm tests first is tests should be extremely easy to read. of course ideally so should production code, but it's not always possible. if a test is extremely complicated, it could be a code smell or a sign that it should be broken up. this way it's very easy to verify the llm's output (weird typos or imports would be caught by intellisense anyway)