3 ms·

That sounds mostly like an incentives problem. If OpenAI, Anthropic, etc decide their LLMs need to be accurate they will find some way of better catching halluc

That sounds mostly like an incentives problem. If OpenAI, Anthropic, etc decide their LLMs need to be accurate they will find some way of better catching hallucinations. It probably will end up (already is?) being yet another LLM acting as a control structure trying to fact check responses before they are sent to users though, so who knows if it will work well. Right noe there's no incentive though. People keep paying good money to use these tools despite their hallucinations, aka lies/gas lighting/fake information. As long as users don't stop paying and LLM companies don't have business pressure to lean on accuracy as a market differentiator, no one is going to bother fixing it.

bearjaws 2y ago
Believe me, if they could use another LLM to audit an LLM, they would have done that already. It's inherit to transformers that they predict the next most likely token, its not possible to change that behavior without making them useless at generalizing tasks (overfitting). LLMs run on statistics, not logic. There is no fact checking, period. There is just the next most likely token based on the context provided.
- _heimdall 2y ago
  Yeah its an interesting question, and I'm a little surprised I got down voted here. I wouldn't expect them to add an additional LLM layer unless hallucinations from the underlying LLM aren't acceptable, and in this case that means it is unacceptable enough to cost them users and money. Adding a check/audit layer, even if it would work, is expensive both financially and computationally. I'm not sold that it would actually work, but I just don't think they've had enough reason to really give it a solid effort yet either. Edit: as far as fact checking, I'm not sure why it would be impossible. An LLM wouldn't likely be able to run a check against a pre-trained model of "truth," but that isn't the only option. An LLM should be able to mimic what a human would do, interpret the response and search a live dataset of sources considered believable. Throw a budget of resources at processing the search results and have the LLM decide if the original response isn't backed up, or contradicts the source entirely.
- coliveira 2y ago
  Yes, most people who disagree with this have no clear understanding of how a LLM works. It is just a prediction mechanism for the next token. The implementation is very fancy and takes a lot of training, but it's not doing anything more than next token prediction. That's why it is incapable of doing any reasoning.

bearjaws 2y ago

Believe me, if they could use another LLM to audit an LLM, they would have done that already. It's inherit to transformers that they predict the next most likely token, its not possible to change that behavior without making them useless at generalizing tasks (overfitting). LLMs run on statistics, not logic. There is no fact checking, period. There is just the next most likely token based on the context provided.

_heimdall 2y ago
Yeah its an interesting question, and I'm a little surprised I got down voted here. I wouldn't expect them to add an additional LLM layer unless hallucinations from the underlying LLM aren't acceptable, and in this case that means it is unacceptable enough to cost them users and money. Adding a check/audit layer, even if it would work, is expensive both financially and computationally. I'm not sold that it would actually work, but I just don't think they've had enough reason to really give it a solid effort yet either. Edit: as far as fact checking, I'm not sure why it would be impossible. An LLM wouldn't likely be able to run a check against a pre-trained model of "truth," but that isn't the only option. An LLM should be able to mimic what a human would do, interpret the response and search a live dataset of sources considered believable. Throw a budget of resources at processing the search results and have the LLM decide if the original response isn't backed up, or contradicts the source entirely.
coliveira 2y ago
Yes, most people who disagree with this have no clear understanding of how a LLM works. It is just a prediction mechanism for the next token. The implementation is very fancy and takes a lot of training, but it's not doing anything more than next token prediction. That's why it is incapable of doing any reasoning.