4 ms·
But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China
by codetrotter 2y ago
But how do you know if the LLMs agree, when all of them word the response differently
For example
LLM 1: Yes, it is true that fireworks were invented in China
LLM 2: Fireworks were indeed invented in China
- Bjartr 2y agoAsk another model if the two statements are in agreement of course! ;)
- omneity 2y agoThis is trivially achievable with function calling, assuming the model you use supports this (which most models do at this point). Define a function `reportFactual(isFactual: boolean)` and you will get standardized, machine-readable answers to do statistics with.
- codetrotter 2y agoI’ve used function calls with OpenAI. But are there any good local LLMs that you can run with Ollama that support function calling?
- omneity 2y agoIf you expect an OpenAI compatible API to use function calls, I don't think Ollama supports it yet (to be confirmed). However you can do it yourself using the appropriate tokens for the model. I know that Llama3, various Mistrals and Command-R support function calling out of the box. Here are the tokens to achieve this in Mixtral 8x22 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1#function-calling-and-special-tokens https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1... Pass function definitions in the system prompt.
- indigodaddy 2y agoI think llamafile supports openai compatible api.. https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile
- inimino 2y agoSimpler yet, just tell the model "Reply with 'Yes' or 'No'."
- CuriouslyC 2y agoPlot twist, you can ask LLMs to provide a bayesian prior on their belief in the truth value (from multiple perspectives, again) then plug that into a variety of algorithms.