3 ms·
> ...so you need a checker that actually reasons about the world (compiler, linter, SAT solver, ground-truth dataset, etc.). Agree. What do you think about tel
by hu3 2y ago
> ...so you need a checker that actually reasons about the world (compiler, linter, SAT solver, ground-truth dataset, etc.).
Agree. What do you think about telling the LLM to also generate unit tests for the code it spits and then run all tests (including previous application unit tests).
I think this is a way to ensure some level of grounded verification:
- Does code compile?
- Do unit test pass?
AI can then consume test results to help fix their own mistakes.
- dwaltrip 2y agoDefinitely, test runners are a way to ground the model and give it a feedback loop. Not a silver bullet but can be very helpful.
- nojs 2y agoThis works well but only if you eyeball the tests and edit them a bit in my experience. Otherwise it gets lazy and makes them trivial to pass. Also, you’ve often gotta explicitly tell it not to hardcode test cases in the solution to make them pass.
- macrolime 2y agoYet, having the LLM evaluate the tests often works well. Especially if you ask it to look for hardcoded test cases.
- eru 2y ago> Also, you’ve often gotta explicitly tell it not to hardcode test cases in the solution to make them pass. You can use property based testing for that. But I've often run into cases where the AI gets into a vicious spiral of worse and worse code when you keep feeding it the test failures.