3 ms·

The way we fix this in most of the rest of the world is "providing citations": we should always be a bit suspicious of unsupported assertions about facts, no? [

The way we fix this in most of the rest of the world is "providing citations": we should always be a bit suspicious of unsupported assertions about facts, no? [0] It won't help for those cases where the facts are right, but the reasoning around them is wrong, I suppose; but critical reasoning is a generally essential skill. [0] https://www.pcc.edu/library/research/reasons-for-citing-sources/ https://www.pcc.edu/library/research/reasons-for-citing-sour...

dragonwriter 4y ago
> The way we fix this in most of the rest of the world is “providing citations” Statistical generation of text doesn’t work in a way which makes “providing citations” a straightforward thing; It’s not reading a bunch of text, making notecards on ideas, and then using the notecards to build a written response where ideas directly come from sources that are quoted or paraphrased. “Providing citations” can be a goal for the output, but its not much in the way of an an answer to “how”. And you can ask some of the current tools to provide citations, and when they do it at all, they can hallucinate citations just as well as anything else. (This has been seen in many of the attempts to get ChatGPT, etc., to produce legal arguments.) Now, checking citations, to some degree of utility, is probably tractable if the cited material is available, as AI’s are good at summarizing text, so comparing summary of a passage with a citation against the summary of the cited material should be possible, but I don’t think any AI system yet has found a way to provide good enough citations (even with some occasional hallucinations) that citation checking to raise flags on the bad ones would be enough to get us to useful results. The challenge is how to both recognize when citations are appropriate and generate good ones a high percentage of the time.
- NovemberWhiskey 4y ago
  I suppose I'm indifferent to whether it's hard or not; it doesn't matter to me as a consumer. If your LLM can't provide the sources for the facts it's asserting, then maybe we shouldn't trust it in matters that depend on facts? If your LLM can't tell the difference between factual statements, postulation, and argument, then why would I trust for anything at all? It's a great party trick though. FWIW, I prompted ChatGPT with "Describe how tail recursion works in programming, and why it is important. Include citations." and it actually included two relevant links in its sources, but did not indicates which part of the text depended on them.
  - dragonwriter 4y ago
    > I suppose I’m indifferent to whether it’s hard or not; it doesn’t matter to me as a consumer. Nor should you, as a consumer. I am just saying that, in the discussion of how to get where we want to be, “provide citations” is much more a description of the destination than the route we are trying to map.

dragonwriter 4y ago

> The way we fix this in most of the rest of the world is “providing citations” Statistical generation of text doesn’t work in a way which makes “providing citations” a straightforward thing; It’s not reading a bunch of text, making notecards on ideas, and then using the notecards to build a written response where ideas directly come from sources that are quoted or paraphrased. “Providing citations” can be a goal for the output, but its not much in the way of an an answer to “how”. And you can ask some of the current tools to provide citations, and when they do it at all, they can hallucinate citations just as well as anything else. (This has been seen in many of the attempts to get ChatGPT, etc., to produce legal arguments.) Now, checking citations, to some degree of utility, is probably tractable if the cited material is available, as AI’s are good at summarizing text, so comparing summary of a passage with a citation against the summary of the cited material should be possible, but I don’t think any AI system yet has found a way to provide good enough citations (even with some occasional hallucinations) that citation checking to raise flags on the bad ones would be enough to get us to useful results. The challenge is how to both recognize when citations are appropriate and generate good ones a high percentage of the time.

NovemberWhiskey 4y ago
I suppose I'm indifferent to whether it's hard or not; it doesn't matter to me as a consumer. If your LLM can't provide the sources for the facts it's asserting, then maybe we shouldn't trust it in matters that depend on facts? If your LLM can't tell the difference between factual statements, postulation, and argument, then why would I trust for anything at all? It's a great party trick though. FWIW, I prompted ChatGPT with "Describe how tail recursion works in programming, and why it is important. Include citations." and it actually included two relevant links in its sources, but did not indicates which part of the text depended on them.
- dragonwriter 4y ago
  > I suppose I’m indifferent to whether it’s hard or not; it doesn’t matter to me as a consumer. Nor should you, as a consumer. I am just saying that, in the discussion of how to get where we want to be, “provide citations” is much more a description of the destination than the route we are trying to map.