3 ms·

Friendly PSA: If you're going to use ChatGPT to generate code, do everyone a favor and add something like "...and write a passing unit test" to the prompt. If

Friendly PSA: If you're going to use ChatGPT to generate code, do everyone a favor and add something like "...and write a passing unit test" to the prompt. If you want to really do everyone a favor, consider that TDD has never been easier. Don't write a prompt, write a unit test-- and instruct GPT to write the code that makes it pass. (Don't ask AI to do your work...ask it to finish your work.)

NhanH 3y ago
ChatGPT tends to over mock the module to the point it will always pass the test, so except for utilities module, this isn't helping much either. This is especially common in JS environment. If you think about it, it does make sense why: the prompt tends to have no where near the context of how the module should be delineated. Normally the programmer knows how the abstraction within his program is separated, but we have no good way to tell it to chatGPT. That means unless your program structure is still following one of the common structure and hasn't morphed into your business domain structure, ChatGPT will mock the wrong layer/ abstraction in the test suite.
- jstarfish 3y ago
  I don't know JS, but it works pretty well for me in Python. Where it goes astray, I'll put comments indicating what it's supposed to do and GPT tends to pick up on it. It's like instruction-based inpainting/outpainting. The biggest hurdle is context limit, especially from retained comments. The real benefit to my system is that it weeds out the zero-effort clowns by forcing them to validate their own work before I waste my time trying to do anything with it.
ipaddr 3y ago
Now you have a test that passes but checks something unrelated.
- Vanit 3y ago
  You ask ChatGPT to write that test too (:
  - bheadmaster 3y ago
    Now you have a test that passes its test but the latter checks for something unrelated...
webjunkie 3y ago
This this this. You can ask it to write a test. You can ask it to fix it, if the test does not pass. You can ask it to check whether the code it wrote makes sense, and it will surely find most times that there is something to improve. Urgh, it's not "one prompt and done".

NhanH 3y ago

ChatGPT tends to over mock the module to the point it will always pass the test, so except for utilities module, this isn't helping much either. This is especially common in JS environment. If you think about it, it does make sense why: the prompt tends to have no where near the context of how the module should be delineated. Normally the programmer knows how the abstraction within his program is separated, but we have no good way to tell it to chatGPT. That means unless your program structure is still following one of the common structure and hasn't morphed into your business domain structure, ChatGPT will mock the wrong layer/ abstraction in the test suite.

jstarfish 3y ago
I don't know JS, but it works pretty well for me in Python. Where it goes astray, I'll put comments indicating what it's supposed to do and GPT tends to pick up on it. It's like instruction-based inpainting/outpainting. The biggest hurdle is context limit, especially from retained comments. The real benefit to my system is that it weeds out the zero-effort clowns by forcing them to validate their own work before I waste my time trying to do anything with it.

ipaddr 3y ago

Now you have a test that passes but checks something unrelated.

Vanit 3y ago
You ask ChatGPT to write that test too (:
- bheadmaster 3y ago
  Now you have a test that passes its test but the latter checks for something unrelated...

webjunkie 3y ago

This this this. You can ask it to write a test. You can ask it to fix it, if the test does not pass. You can ask it to check whether the code it wrote makes sense, and it will surely find most times that there is something to improve. Urgh, it's not "one prompt and done".