5 ms·
Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebase
by yuliyp 1y ago
Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative?
As is it's neat that they wrote some code to generate some prompts for an LLM but there's no idea if it actually works.
- motorest 1y ago> Did the author do any analysis of the effectiveness of their tool on something beyond multiplication? Did they look to see if it caught any bugs in any codebases? What's the false positive rate? False negative? I would also add the concern on whether the tests are actually deterministic. The premise is also dubious, as docstring comments typically hold only very high-level descriptions of the implementation and often aren't even maintained. Writing a specification of what a function is expected to do is what writing tests is all about, and with LLMs these are a terse prompt away.
- bluGill 1y agoDocumentation should not be telling your how it is implemented. It should tell you how and why to use the function. Users who care about how it is implemented should be reading the code not the comments. Users who need to find/use a helper and get on with their feature shouldn't.