7 ms·
What I would love to see in GPT 3 is some sort of a confidence score that they could return, as in how sure their model is that what it returned is accurate and
by anshumankmr 4y ago
What I would love to see in GPT 3 is some sort of a confidence score that they could return, as in how sure their model is that what it returned is accurate and not gibberish. Could this classifier help with that? I am working on a requirement where we are using ElasticSearch to map a query to an article in a knowledge base and then the plan is to send it to GPT 3 to help summarize the article.
Since the ElasticSearch integration is still WIP, I had made a POC to scrape the knowledge base (with mixed results, lots of the content is poorly organized, so the scraped content that would act as prompt to the GPT 3 model wasn't all that good either) and then feed it to GPT 3, but the it couldn't always give the most accurate answers on that. The answers sometimes were spot on, or quite good but other times, not so much. I would say about 30% of the time, it made sense. So if there was a way for me to get if answer was sensible or not, so we could give an error response if the GPT 3's response did not make sense.
The reason why we are doing it cause the client has a huge knowledge base and mapping each question to an answer would be difficult for them.
- ilaksh 4y agoOpenAI's text completion has an option to return "log probability" or something for each token. That might apply. You can also turn down the temperature parameter which reduces hallucinations to some degree.