3 ms·
> I had some test cases with hard coded numbers that had wobbled and needed updating. I simply asked the LLM to keep rerunning the test and updating the numbers
by teraflop 2y ago
> I had some test cases with hard coded numbers that had wobbled and needed updating. I simply asked the LLM to keep rerunning the test and updating the numbers as necessary.
Why not take this a step farther and incorporate this methodology directly into your test suite? Every time you push a code change, run the new version of the code and use it to automatically update the "expected" output. That way you never have to worry about failures at all!
- ezyang 2y agoIn fact, the test framework I was using at the time (jest) did in fact support this. But the person who had originally written the tests hadn't had the foresight to use snapshot tests for this failing test!
- diggan 2y agoI don't know if your message is a continuation of the sarcasm (I feel like maybe no?), but I'm pretty sure parent's joke is that if you just change the expected values whenever the code changes, you aren't really effectively "testing" anything as much as "recording" outputs.