3 ms·
> Reading code is the hardest part Does anyone have experience using an LLM to extract documentation from code? Or are LLMs bad at it because we have too few
by amelius 2y ago
> Reading code is the hardest part
Does anyone have experience using an LLM to extract documentation from code?
Or are LLMs bad at it because we have too few good examples? :)
- wizzwizz4 2y agoOn the contrary: they're bad at it because our examples are too good. There is information in documentation (requirements, rationales, performance characteristics, safety concerns) that cannot be deduced from the code. The language model anticipates that such information would be present, so it writes it anyway – despite having no idea what it should say. If you're lucky, you get something bland and verbose, but not incorrect. If you're unlucky, you get something dangerously wrong that a later reader assumes is true / important, since "otherwise, why would they have put it in the documentation?". This is part of why I refuse to work on codebases which LLMs have mucked about with. Even the most basic things cannot be relied upon. (Note: refusal can be overridden by suitable application of financial compensation.)