4 ms·
This definitely seems like a potentially powerful approach. The said, maybe I'm missing something obvious, but is the LLM generating both the tests and the impl
by DylanSp 3y ago
This definitely seems like a potentially powerful approach. The said, maybe I'm missing something obvious, but is the LLM generating both the tests and the implementation? If that's the case, then it seems like there could be issues caused by the generated tests not matching what's specified in the initial prompt. Manually writing highly focused unit tests doesn't seem like the best way to work with this, but being able to manually write some sort of high-level, machine-checkable specs might be useful.
- mikeravkine 3y agoThis is exactly what happened when I tried this approach personally: on a non trivial codebase, the tests were just as likely to be wrong as the code itself and during "debugging" the wrong would converge until the test passed but the results were useless. I still have my prompt from before and a Modal account, I'm going to give this one a shot to see if I missed something or if it really only works in the mickey mouse case.