3 ms·
That will help a bit, but it's not going to fix it. Using GPT4 and Code Interpreter, I have asked it to write a function and test it, given some inputs and exp
by skybrian 3y ago
That will help a bit, but it's not going to fix it.
Using GPT4 and Code Interpreter, I have asked it to write a function and test it, given some inputs and expected outputs. The function returned different values when it tested it, but it lied and said it worked as expected.
You need to read the code and the test outputs yourself. Or maybe have it write an automated test?
Despite this, it seems quite promising. I expect that in a year or two, some IDE's will come with a useful pair programming feature.
- ec109685 3y agoWith code interpreter, it actually runs the code, so how did failing tests get interpreted as being correct?
- skybrian 3y agoI don't know. Why does a language model do anything? The "test" was a print statement, not a unit test. There wasn't a failure message. It had to read the output and compare it to the expected value I gave it. It claimed it got a different result. I guess it didn't really read the result because it strongly expected something else? If you use an assertEquals() that loudly complains, maybe it's less likely to do this? I haven't seen it ignore stack traces.