3 ms·
How are you measuring the confidence of the answers? This is one of the biggest challenges I've seen with AI, it provides wrong answers to hard questions, which
by 8organicbits 2y ago
How are you measuring the confidence of the answers? This is one of the biggest challenges I've seen with AI, it provides wrong answers to hard questions, which wastes the user's time.
- engomez 2y agoTake a look at https://docs.inkeep.com/ai-api/openai-chat-completion-endpoint#question-answer-mode https://docs.inkeep.com/ai-api/openai-chat-completion-endpoi... tl;dr we define a JSON schema with a few semantic labels that represent a gradient in confidence. On our end we prompt it with certain examples and guidance for when to use each label. This is generally a better approach than e.g. asking an LLM to give a numeric score. We also have trained embedding-based classifiers as non-LLM heuristics.