3 ms·
How to verify the meaning of docs? How to deal with the model hallucinations? It would be hell to lose trust to api docs due to those risks.
by zx8080 4y ago
How to verify the meaning of docs? How to deal with the model hallucinations?
It would be hell to lose trust to api docs due to those risks.
- splatzone 4y agoVery good point, but easily solved - just tag the docs as being generated by GPT-4, and make sure whoever reads them knows it.
- didericis 4y agoThat doesn't solve the problem. Documentation of unknown quality is useless noise. People don't understand the unfathomable amount of garbage that's going to be generated in light of all these models. Doesn't matter how accurate they are, the lack of understanding for that remainder percent of inaccuracy is going to cause false confidence and cause errors to compound like mad.
- funfunfunction 4y agoHallucination is definitely a problem but can be somewhat mitigated by good prompting. GPT-4 seems less prone to hallucination. This will be better over time. You can view the prompts used for generating docs here[1] and the prompts used for answering questions here[2] [1]https://github.com/context-labs/autodoc/blob/master/src/cli/commands/index/prompts.ts https://github.com/context-labs/autodoc/blob/master/src/cli/... [2https://github.com/context-labs/autodoc/blob/master/src/cli/commands/query/createChatChain.ts https://github.com/context-labs/autodoc/blob/master/src/cli/...]
- docandrew 4y agoThe way I’m thinking of it is as a junior engineer who can go do busy work for me. I’m not going to accept their “PRs” without a review. Even if it gets me 75% of the way there, that’s still a big time savings for me.