5 ms·

Even the latest commercial LLMs are happy to confidently bullshit about what they think is in published research even if they provide citations. Often the citat

Even the latest commercial LLMs are happy to confidently bullshit about what they think is in published research even if they provide citations. Often the citations themselves are slightly corrupted. I actually verify each LLM claim so I know this is happening a lot. Occasionally they are complete fabrications. It really varies by research topic. Its really bad in esoteric research areas. They even acknowledge the paper was actually about something else if you call them out on it. What a disaster. LLMs are still useful for information retrieval and exploration as long as you understand you are having a conversation with a habitual liar / expert beginner and adjust your prompts and expectations accordingly.

bongodongobob 3y ago
Unintuitively, I think you'll probably end up with better answers if you don't ask for citations. The vast majority of its training isn't white papers so you're artificially constraining its "imagination" to the cited sources space. I find the more constraints you add, the worse your answers are.
notnullorvoid 3y ago
> They even acknowledge the paper was actually about something else if you call them out on it. For clarity is not really acknowledging it made a mistake. "Calling out" an LLM's mistake just leads to the next most likely text to be something that sounds like an acknowledgement of a mistake, but the same is likely to happen if the LLM generated a correct response and you respond claiming that it's incorrect.
- greenavocado 3y ago
  The LLM generally correctly identifies the topics in the paper after calling out the original mistake.
jiggawatts 3y ago
> What a disaster. Using tool inappropriately leads to suboptimal outcomes -- news at 11. A good mental model is that an LLM is a blurry JPEG of the Internet. You sound like a scientist, right? You reference "published research", after all. What would your opinion be of a researcher measuring the exact values of the pixels of a JPEG image instead of the RAW sensor data?
- greenavocado 3y ago
  Using your flawed analogy: the picture is not blurry. It is a gigantic canvas filled in with false details.
  - jiggawatts 3y ago
    It's false in the same sense that the pixels of lossy-compressed image are false. No, you're not going to get the precise values back out either with an LLM or a JPG image. Yes, the picture will still "look like" the original. You asked the LLM to output a URL or reference that looks like something that would appear in a research paper. It did so. You're complaining the precise byte sequence returned is not what you wanted. The error is with your expectations, not the tool.

bongodongobob 3y ago

Unintuitively, I think you'll probably end up with better answers if you don't ask for citations. The vast majority of its training isn't white papers so you're artificially constraining its "imagination" to the cited sources space. I find the more constraints you add, the worse your answers are.

notnullorvoid 3y ago

> They even acknowledge the paper was actually about something else if you call them out on it. For clarity is not really acknowledging it made a mistake. "Calling out" an LLM's mistake just leads to the next most likely text to be something that sounds like an acknowledgement of a mistake, but the same is likely to happen if the LLM generated a correct response and you respond claiming that it's incorrect.

greenavocado 3y ago
The LLM generally correctly identifies the topics in the paper after calling out the original mistake.

jiggawatts 3y ago

> What a disaster. Using tool inappropriately leads to suboptimal outcomes -- news at 11. A good mental model is that an LLM is a blurry JPEG of the Internet. You sound like a scientist, right? You reference "published research", after all. What would your opinion be of a researcher measuring the exact values of the pixels of a JPEG image instead of the RAW sensor data?

greenavocado 3y ago
Using your flawed analogy: the picture is not blurry. It is a gigantic canvas filled in with false details.
- jiggawatts 3y ago
  It's false in the same sense that the pixels of lossy-compressed image are false. No, you're not going to get the precise values back out either with an LLM or a JPG image. Yes, the picture will still "look like" the original. You asked the LLM to output a URL or reference that looks like something that would appear in a research paper. It did so. You're complaining the precise byte sequence returned is not what you wanted. The error is with your expectations, not the tool.