4 ms·
Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text a
by visarga 3y ago
Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.
- codetrotter 3y agoBut how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China
- Bjartr 3y agoAsk another model if the two statements are in agreement of course! ;)
- omneity 3y agoThis is trivially achievable with function calling, assuming the model you use supports this (which most models do at this point). Define a function `reportFactual(isFactual: boolean)` and you will get standardized, machine-readable answers to do statistics with.
- codetrotter 3y agoI’ve used function calls with OpenAI. But are there any good local LLMs that you can run with Ollama that support function calling?
- omneity 3y agoIf you expect an OpenAI compatible API to use function calls, I don't think Ollama supports it yet (to be confirmed). However you can do it yourself using the appropriate tokens for the model. I know that Llama3, various Mistrals and Command-R support function calling out of the box. Here are the tokens to achieve this in Mixtral 8x22 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1#function-calling-and-special-tokens https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1... Pass function definitions in the system prompt.
- indigodaddy 3y agoI think llamafile supports openai compatible api.. https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile
- inimino 3y agoSimpler yet, just tell the model "Reply with 'Yes' or 'No'."
- CuriouslyC 3y agoPlot twist, you can ask LLMs to provide a bayesian prior on their belief in the truth value (from multiple perspectives, again) then plug that into a variety of algorithms.