4 ms·

Hmmm, an AI that only uses factual encyclopaedia data for its training set. If they can also completely eliminate hallucinations, they could be onto something

Hmmm, an AI that only uses factual encyclopaedia data for its training set. If they can also completely eliminate hallucinations, they could be onto something useful.

ipsum2 3y ago
If you train a language model on a factual dataset, it will still hallucinate.
- justinclift 3y ago
  That's why I specifically said "if they can also eliminate that...". Sure it won't come from just having a good data set... they'll need to do other stuff as well. Hopefully not impossible. :)
  - skissane 3y ago
    One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database. Another approach that can help, is after you generate the response, you then submit the response to an LLM (even the same LLM), with a prompt to check it for errors (errors of reasoning, misrepresentation of citations, etc). Often, LLMs can do a decent job on catching their own errors, including hallucinations. The more advanced an LLM is, the less likely it is to hallucinate, and the more likely it is to pick up on its own mistakes. Llama-7b hallucinates a lot more than GPT-4, and is far less likely to detect itself doing so. You can't make an LLM foolproof, but you can't make humans foolproof either. Humans "hallucinate" too – e.g. students write essays with erroneous claims, cited to sources which never actually make those claims.
    - justinclift 3y ago
      > You can't make an LLM foolproof ... Heh, we need "post-LLM" AI's then that don't hallucinate at all. ;)
    - dragonwriter 3y ago
      > One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database. Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query.
      - skissane 3y ago
        > Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query. They can produce rather different results – the LLM can do a better job than just an embedding at picking out what's truly important in the prompt, which can increase the relevance of the returned documents. I've seen before where a vector search returns documents which contain some similar language to the prompt but which are actually talking about something completely unrelated. LLMs are less likely to do that sort of thing, because they have a much better understanding of what's important in the prompt and what's just fluff/waffle
Cheer2171 3y ago
Do you think the reason hallucinations occur in LLMs is because the training data includes the hallucination? You are very wrong. Hallucinations are not a training data problem, at least in the way you are implying.
- justinclift 3y ago
  > Do you think the reason hallucinations occur in LLMs is because the training data includes the hallucination? No. What a bizarre take. Why would anyone think that? Having a training data set that's not based on fiction sounds like it could be a useful thing. If they can somehow get rid of the hallucinations too it could be good.
  - Cheer2171 3y ago
    > Why would anyone think that? Sorry (not sorry) for assuming that the second sentence in your comment was related to the first.
    - justinclift 3y ago
      Hmmm, I've just added the word "also" in there as it seems you're not the only person to misunderstand what I meant. Hopefully it's clearer now. ;)
barbazoo 3y ago
Are hallucinations really only a problem with "non factual" training data? Looking at all the made up library methods that ChatGPT uses in the code it produces, it made me think this is a general LLM/GPT problem.
- rileymat2 3y ago
  Yes, weirdly, often the hallucination is how I would have liked them to expose the API.
  - barbazoo 3y ago
    There's a AI powered devtool in this idea somewhere
- thfuran 3y ago
  It is a fundamental problem with how gpt works.
observationist 3y ago
Just because it's printed in a book doesn't mean it's any more factual than any other source of information. Many studies and reviews have found Wikipedia to be more accurate overall than Encyclopedia Britannica. In fact, because it's in a book, it's harder to correct, and more insidious because of opinions like the one you just expressed. In both cases, you should not treat the information as canonically or authoritatively accurate or factual. Biases, gaps, outright lies and fabrication exist in any large collection of human writing.
- breck 3y ago
  > Many studies and reviews have found Wikipedia to be more accurate overall than Encyclopedia Britannica I'm not doubting this, but do you have citations?
  - RandallBrown 3y ago
    I realize this isn't the most unbiased source, but it does have lots of citations with studies comparing Wikipedia and Britannica. https://en.wikipedia.org/wiki/Reliability_of_Wikipedia https://en.wikipedia.org/wiki/Reliability_of_Wikipedia
    - wolverine876 3y ago
      That comment is an amazing demonstration of the intellectual failure of Internet users (as you somewhat acknowledge) and the reason Wikipedia is so popular. If you know it's heavily biased, maybe find something else?
      - RandallBrown 3y ago
        The post I'm replying to was asking for a list of citations where Wikipedia was equally accurate to Encyclopedia Britannica. That wiki article has several of those citations (as well as Britannica's attempted refutations). I understand that you can't read every link in a comment, but you certainly don't have to post a comment like this if you haven't at least looked into what I posted.
HankB99 3y ago
That will be interesting. I suspect that the hallucinations come from combining unrelated data in ways that stretch reality. However, using the general Internet as a source of Truth seems like it was never a good idea.
fcsp 3y ago
I took a brief look at their online article about the British empire. While it does briefly mention Jamaica requiring "conquest" in the origins section, it seems mostly oblivious to the consequences of the empire's "commercial ambitions" for the local populations. Not sure that would form a great "factual" source of truth to train an AI on.
HughParry 3y ago
I've got a 20 year old set of their books in the basement. It's got facts which were wrong at the time of writing. At the end of the day it's just a single centralised source
dragonwriter 3y ago
An LLM that only used “factual encyclopedia data” as its training set would probably be very good at producing content that looked like encyclopedia articles, but bad at anything else. Probably wouldn’t be any less prone to confabulation than any other LLM, and given how limited in coverage the sum of any factual encyclopedia is compared to typical LLM training sets, would have even a spottier base of factual information that most LLM’s have to work from. > If they can completely eliminate hallucinations Yes, but that’s orthogonal to the training set.
- justinclift 3y ago
  > Yes, but that’s orthogonal to the training set. Of course. I didn't say otherwise. > An LLM that ... Note that I also wrote "AI" rather than "LLM". ;) Hopefully there will be better alternatives (to LLMs) at some point that don't hallucinate. My point is that having one of those (something that doesn't hallucinate) trained on factual data sounds like it'd be useful. :)
  - dragonwriter 3y ago
    > Hopefully there will be better alternatives (to LLMs) at some point that don’t hallucinate. There are AIs that don’t “hallucinate” (a bad metaphor), e.g., most of the technologies developed in the previous rounds of AI – like expert systems. I doubt that we will find more advanced, equally-or-more general AI techniques than LLMs that don’t “hallucinate”, what we’ll probably do is find better ways to build systems around LLMs (or more future AI architectures) that allow them to resolve “hallucination” better before presenting results. It’s not as if natural intelligences don’t both literally hallucinate and, more relevant to what is described as “hallucination” in AIs, confabulate when pressed to answer without access to facts, and we know with them that grounding in physical reality via sensory data and having access to factual references doesn’t mitigate those effects (and we’ve also been able to demonstrate the latter with LLMs.)
- nickpsecurity 3y ago
  You’re missing that you can reinforce information by repeating specific, data sets. They could use some repetitions of it to drill in the knowledge, put in a lot of content that’s less reliable in a different style, and re-run high-quality content afterwards to reinforce it. The AI’s already accidentally do that for topics people repeat a lot on the Internet, esp political. ChatGPT used to fight with me over some of those. That it would not relent or stop redirecting me showed that they can be trained to follow authoritative sources, too.

Hmmm, an AI that only uses factual encyclopaedia data for its training set. If they can also completely eliminate hallucinations, they could be onto something useful.

ipsum2 3y ago

If you train a language model on a factual dataset, it will still hallucinate.

justinclift 3y ago
That's why I specifically said "if they can also eliminate that...". Sure it won't come from just having a good data set... they'll need to do other stuff as well. Hopefully not impossible. :)
- skissane 3y ago
  One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database. Another approach that can help, is after you generate the response, you then submit the response to an LLM (even the same LLM), with a prompt to check it for errors (errors of reasoning, misrepresentation of citations, etc). Often, LLMs can do a decent job on catching their own errors, including hallucinations. The more advanced an LLM is, the less likely it is to hallucinate, and the more likely it is to pick up on its own mistakes. Llama-7b hallucinates a lot more than GPT-4, and is far less likely to detect itself doing so. You can't make an LLM foolproof, but you can't make humans foolproof either. Humans "hallucinate" too – e.g. students write essays with erroneous claims, cited to sources which never actually make those claims.
  - justinclift 3y ago
    > You can't make an LLM foolproof ... Heh, we need "post-LLM" AI's then that don't hallucinate at all. ;)
  - dragonwriter 3y ago
    > One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database. Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query.
    - skissane 3y ago
      > Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query. They can produce rather different results – the LLM can do a better job than just an embedding at picking out what's truly important in the prompt, which can increase the relevance of the returned documents. I've seen before where a vector search returns documents which contain some similar language to the prompt but which are actually talking about something completely unrelated. LLMs are less likely to do that sort of thing, because they have a much better understanding of what's important in the prompt and what's just fluff/waffle

Cheer2171 3y ago

Do you think the reason hallucinations occur in LLMs is because the training data includes the hallucination? You are very wrong. Hallucinations are not a training data problem, at least in the way you are implying.

justinclift 3y ago
> Do you think the reason hallucinations occur in LLMs is because the training data includes the hallucination? No. What a bizarre take. Why would anyone think that? Having a training data set that's not based on fiction sounds like it could be a useful thing. If they can somehow get rid of the hallucinations too it could be good.
- Cheer2171 3y ago
  > Why would anyone think that? Sorry (not sorry) for assuming that the second sentence in your comment was related to the first.
  - justinclift 3y ago
    Hmmm, I've just added the word "also" in there as it seems you're not the only person to misunderstand what I meant. Hopefully it's clearer now. ;)

barbazoo 3y ago

Are hallucinations really only a problem with "non factual" training data? Looking at all the made up library methods that ChatGPT uses in the code it produces, it made me think this is a general LLM/GPT problem.

rileymat2 3y ago
Yes, weirdly, often the hallucination is how I would have liked them to expose the API.
- barbazoo 3y ago
  There's a AI powered devtool in this idea somewhere
thfuran 3y ago
It is a fundamental problem with how gpt works.

observationist 3y ago

Just because it's printed in a book doesn't mean it's any more factual than any other source of information. Many studies and reviews have found Wikipedia to be more accurate overall than Encyclopedia Britannica. In fact, because it's in a book, it's harder to correct, and more insidious because of opinions like the one you just expressed. In both cases, you should not treat the information as canonically or authoritatively accurate or factual. Biases, gaps, outright lies and fabrication exist in any large collection of human writing.

breck 3y ago
> Many studies and reviews have found Wikipedia to be more accurate overall than Encyclopedia Britannica I'm not doubting this, but do you have citations?
- RandallBrown 3y ago
  I realize this isn't the most unbiased source, but it does have lots of citations with studies comparing Wikipedia and Britannica. https://en.wikipedia.org/wiki/Reliability_of_Wikipedia https://en.wikipedia.org/wiki/Reliability_of_Wikipedia
  - wolverine876 3y ago
    That comment is an amazing demonstration of the intellectual failure of Internet users (as you somewhat acknowledge) and the reason Wikipedia is so popular. If you know it's heavily biased, maybe find something else?
    - RandallBrown 3y ago
      The post I'm replying to was asking for a list of citations where Wikipedia was equally accurate to Encyclopedia Britannica. That wiki article has several of those citations (as well as Britannica's attempted refutations). I understand that you can't read every link in a comment, but you certainly don't have to post a comment like this if you haven't at least looked into what I posted.

HankB99 3y ago

That will be interesting. I suspect that the hallucinations come from combining unrelated data in ways that stretch reality. However, using the general Internet as a source of Truth seems like it was never a good idea.

fcsp 3y ago

I took a brief look at their online article about the British empire. While it does briefly mention Jamaica requiring "conquest" in the origins section, it seems mostly oblivious to the consequences of the empire's "commercial ambitions" for the local populations. Not sure that would form a great "factual" source of truth to train an AI on.

HughParry 3y ago

I've got a 20 year old set of their books in the basement. It's got facts which were wrong at the time of writing. At the end of the day it's just a single centralised source

dragonwriter 3y ago

An LLM that only used “factual encyclopedia data” as its training set would probably be very good at producing content that looked like encyclopedia articles, but bad at anything else. Probably wouldn’t be any less prone to confabulation than any other LLM, and given how limited in coverage the sum of any factual encyclopedia is compared to typical LLM training sets, would have even a spottier base of factual information that most LLM’s have to work from. > If they can completely eliminate hallucinations Yes, but that’s orthogonal to the training set.

justinclift 3y ago
> Yes, but that’s orthogonal to the training set. Of course. I didn't say otherwise. > An LLM that ... Note that I also wrote "AI" rather than "LLM". ;) Hopefully there will be better alternatives (to LLMs) at some point that don't hallucinate. My point is that having one of those (something that doesn't hallucinate) trained on factual data sounds like it'd be useful. :)
- dragonwriter 3y ago
  > Hopefully there will be better alternatives (to LLMs) at some point that don’t hallucinate. There are AIs that don’t “hallucinate” (a bad metaphor), e.g., most of the technologies developed in the previous rounds of AI – like expert systems. I doubt that we will find more advanced, equally-or-more general AI techniques than LLMs that don’t “hallucinate”, what we’ll probably do is find better ways to build systems around LLMs (or more future AI architectures) that allow them to resolve “hallucination” better before presenting results. It’s not as if natural intelligences don’t both literally hallucinate and, more relevant to what is described as “hallucination” in AIs, confabulate when pressed to answer without access to facts, and we know with them that grounding in physical reality via sensory data and having access to factual references doesn’t mitigate those effects (and we’ve also been able to demonstrate the latter with LLMs.)
nickpsecurity 3y ago
You’re missing that you can reinforce information by repeating specific, data sets. They could use some repetitions of it to drill in the knowledge, put in a lot of content that’s less reliable in a different style, and re-run high-quality content afterwards to reinforce it. The AI’s already accidentally do that for topics people repeat a lot on the Internet, esp political. ChatGPT used to fight with me over some of those. That it would not relent or stop redirecting me showed that they can be trained to follow authoritative sources, too.