4 ms·
Treating docstrings as the spec and asking an LLM to flag mismatches feels promising in theory but personally I'd b wary of overfitting to underspecified docs.
by lgiordano_notte 1y ago
Treating docstrings as the spec and asking an LLM to flag mismatches feels promising in theory but personally I'd b wary of overfitting to underspecified docs. Might be useful as a lint-like signal, but hard to see it replacing real tests just yet.
- bluGill 1y agoif that is the only testing you do I agree. However to test that the code works as the docs say is valuable as well. The code often will do more, but it needs to do at least what the docs say.
- lgiordano_notte 1y agoAgreed. Catching mismatches between doc and implementation is still valuable, just wouldn’t want people to rely on it as a safety net when the docs themselves might be inaccurate/incomplete. As a complement to traditional tests though seems like a solid addition.