5 ms·

Andrew, you and your friends should be proud. It's really encouraging to see people, especially in your generation, thinking seriously about the problem of misi

Andrew, you and your friends should be proud. It's really encouraging to see people, especially in your generation, thinking seriously about the problem of misinformation. There are fundamental challenges you all will face in this idea: - most current LLMs are trained on large amounts of web data that itself contains facts, opinions, and misinformation. These things are treated equally, so I would expect the LLM to get common facts right, but also to represent opinions or misinformation as facts when they are pervasive. - LLMs "hallucinate" and tend not to know when to say "I don't know" or to not try to fact-check something that is not factual in nature. ...in short, I would expect LLMs to be an unreliable fact checker, which has the potential to do as much harm as good.

dweinus 2y ago
Thinking out loud... I don't think these problems can be solved. If you are going to do it anyway, I would suggest: - Using a RAG architecture on top of a database of factual information. Wikipedia is probably your best bet. It is not 100% factual or correct either, but maybe as good as it gets. Scaling RAG to wikipedia size is not trivial, but I think it can be done. - Prompting the LLM to cite its sources so people can fact-check the fact-checker - Prompting the LLM to say it is unsure when something does not have a clear answer. I don't expect this to be reliable, but maybe somewhat better
- helloduck1234 2y ago
  Yea ok, we already have the citing thing done, and are going to start working on the RAG architecture soon.
- CuriouslyC 2y ago
  There's a whole art to prompting a LLM to say it's unsure. I need to write a blog post about this, it's deep.
  - visarga 2y ago
    Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.
    - codetrotter 2y ago
      But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China
      - Bjartr 2y ago
        Ask another model if the two statements are in agreement of course! ;)
      - omneity 2y ago
        This is trivially achievable with function calling, assuming the model you use supports this (which most models do at this point). Define a function `reportFactual(isFactual: boolean)` and you will get standardized, machine-readable answers to do statistics with.
        codetrotter 2y ago
        I’ve used function calls with OpenAI. But are there any good local LLMs that you can run with Ollama that support function calling?
        omneity 2y ago
        If you expect an OpenAI compatible API to use function calls, I don't think Ollama supports it yet (to be confirmed). However you can do it yourself using the appropriate tokens for the model. I know that Llama3, various Mistrals and Command-R support function calling out of the box. Here are the tokens to achieve this in Mixtral 8x22 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1#function-calling-and-special-tokens https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1... Pass function definitions in the system prompt.
- netrap 2y ago
  You don't have to solve it, you just have to try...
helloduck1234 2y ago
Yea, thank you for your feedback!
- reaperman 2y ago
  > “…which has the potential to do as much harm as good.” I find this is one of the more difficult things for people to learn to fully integrate into their psyche. Many people never learn to truly care about this and everything it means. They go on forever primarily caring about what’s good for them personally.
  - paulcole 2y ago
    Strong disagree. Put yourself first nearly 100% of the time. Nobody cares about you so don’t think others are doing anything but prioritizing themselves. I mean look at the world. Essentially everybody puts themselves first and it’s clear as day. Don’t trick yourself into being the sap doing things for the greater good. And who cares if there’s equal potential for harm and good? The harm might be less than we imagine and the good might be better than we think it could be. “This might be bad” is a terrible reason to not do something. Nearly everything might be bad! People are pretty resilient. They can generally deal with you being selfish.
    - spencerflem 2y ago
      That's grim. Don't light yourself on fire to keep others warm or nothing, but in my experience most people are hoping to make the world better where they can. This sort of hustle culture belief is definitely present in the world, especially among finance and us techie types, but there's tons of examples of people Not acting like this. Teachers don't do it for the pay, etc. There's a reason meaningful jobs tend to pay less, and its because so many people want to do useful helpful things that badly. Anyways point is, that I want to explicitly condemn this type of thinking. Yeah don't let fear of doing the wrong thing paralyze you but also think through the consequences
      - paulcole 2y ago

dweinus 2y ago

Thinking out loud... I don't think these problems can be solved. If you are going to do it anyway, I would suggest: - Using a RAG architecture on top of a database of factual information. Wikipedia is probably your best bet. It is not 100% factual or correct either, but maybe as good as it gets. Scaling RAG to wikipedia size is not trivial, but I think it can be done. - Prompting the LLM to cite its sources so people can fact-check the fact-checker - Prompting the LLM to say it is unsure when something does not have a clear answer. I don't expect this to be reliable, but maybe somewhat better

helloduck1234 2y ago
Yea ok, we already have the citing thing done, and are going to start working on the RAG architecture soon.
CuriouslyC 2y ago
There's a whole art to prompting a LLM to say it's unsure. I need to write a blog post about this, it's deep.
- visarga 2y ago
  Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.
  - codetrotter 2y ago
    But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China
    - Bjartr 2y ago
      Ask another model if the two statements are in agreement of course! ;)
    - omneity 2y ago
      This is trivially achievable with function calling, assuming the model you use supports this (which most models do at this point). Define a function `reportFactual(isFactual: boolean)` and you will get standardized, machine-readable answers to do statistics with.
      - codetrotter 2y ago
        I’ve used function calls with OpenAI. But are there any good local LLMs that you can run with Ollama that support function calling?
        omneity 2y ago
        If you expect an OpenAI compatible API to use function calls, I don't think Ollama supports it yet (to be confirmed). However you can do it yourself using the appropriate tokens for the model. I know that Llama3, various Mistrals and Command-R support function calling out of the box. Here are the tokens to achieve this in Mixtral 8x22 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1#function-calling-and-special-tokens https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1... Pass function definitions in the system prompt.
netrap 2y ago
You don't have to solve it, you just have to try...

helloduck1234 2y ago

Yea, thank you for your feedback!

reaperman 2y ago
> “…which has the potential to do as much harm as good.” I find this is one of the more difficult things for people to learn to fully integrate into their psyche. Many people never learn to truly care about this and everything it means. They go on forever primarily caring about what’s good for them personally.
- paulcole 2y ago
  Strong disagree. Put yourself first nearly 100% of the time. Nobody cares about you so don’t think others are doing anything but prioritizing themselves. I mean look at the world. Essentially everybody puts themselves first and it’s clear as day. Don’t trick yourself into being the sap doing things for the greater good. And who cares if there’s equal potential for harm and good? The harm might be less than we imagine and the good might be better than we think it could be. “This might be bad” is a terrible reason to not do something. Nearly everything might be bad! People are pretty resilient. They can generally deal with you being selfish.
  - spencerflem 2y ago
    That's grim. Don't light yourself on fire to keep others warm or nothing, but in my experience most people are hoping to make the world better where they can. This sort of hustle culture belief is definitely present in the world, especially among finance and us techie types, but there's tons of examples of people Not acting like this. Teachers don't do it for the pay, etc. There's a reason meaningful jobs tend to pay less, and its because so many people want to do useful helpful things that badly. Anyways point is, that I want to explicitly condemn this type of thinking. Yeah don't let fear of doing the wrong thing paralyze you but also think through the consequences
    - paulcole 2y ago