4 ms·

While all these architectures are innovative and have helped improve either accuracy or speed, the same fundamental problem of generating factual information st

While all these architectures are innovative and have helped improve either accuracy or speed, the same fundamental problem of generating factual information still exists. Retrieval Augmented Generation (RAG), Agents and other similar methods help mitigate this. It will be interesting to see if future architectures eventually replace these techniques.

tormeh 1y ago
To me, the issue seems to be that we're training transformers to predict text, which only forces the model to embed limited amounts of logic. We'd have to find something different to train models on in order for them to stop hallucinating.
- lblume 1y ago
  Modern neuroscience suggests that everything the human brain might be doing is basically a kind of predictive processing, i.e. hallucination based on inductive biases. I do not think this is the main bottleneck.
bsenftner 1y ago
I'm still thinking about how RAG being conceptually simple and easy to implement, why the foundational models have not incorporated it into their base functionality? The lack of that strikes me as a negative point about RAG and it's variants, because if any of them worked, it would be in the models directly and not need to be added afterwards.
- bavell 1y ago
  RAG is a prompting technique, how could they possibly incorporate it into the pre training?
  - maleldil 1y ago
    CoT is a prompting technique too, and it's been incorporated.
    - bavell 1y ago
      IIUC, CoT is "incorporated" into training by just providing better quality training data which steers the model towards "thinking" more deeply in its responses. But at the end of the day, it's still just regular pre training. RAG - Retrieval augmented generation - how can the retrieval be done during training? RAG will always remain external to the model. The whole point is that you can augment the model by injecting relevant context into the prompt at inference time, bringing your own proprietary/domain-specific data.
      - bsenftner 1y ago
        Who says "during training"? RAG could be built into the functionality of the LLM directly - give it the documents you want it to incorporate, and it ingests them as a temp mini-fine tune. That would work just fine.
      - impossiblefork 1y ago
        These things with <think> and </think> tokens are actually trained using RL, so it's not like GSM8k or something like that where you just train on some reasoning. It's actually like QuietSTaR but with a focus on a big thought in the beginning and with more sophisticated RL than just REINFORCE (QuietSTaR uses REINFORCE).
  - bsenftner 1y ago
    The same way developers incorporate it now. Why are you thinking "pre-training", this is a feature of the deployed model: it ingests documents and generates a mini-fine tune right then.
- mdp2021 1y ago
  Why would be a proper documents-at-hand based inquiry be «simple». Information is at paragraph #1234 of book B456; that paragraph acquires special meaning in light of its neighbours, its chapter, the book. Further information is in other paragraphs of other books. You can possibly encode with some "strong" compression information (data), but not insight. The information that a query may point to can be a big cloud of fuzzy concepts. What do you input, how? How big should that input be? "How much" of the past reflection does the Doctor use to build a judgement? RAG seems simple because it has simpler cases ("What is the main export of Bolivia").
- rybosome 1y ago
  Well, even if we assume for a moment that we aren’t talking about non-public data… Then RAG which serves up knowledge already in the model’s pretraining data is still useful, because it primes the model for the specific context with which you want to engage it. I maybe can see what you are saying, like why can’t the model just do a good job without being re-reminded? But even in that sense, any intelligence, artificial or otherwise, will do better given more context. And that ignores the reality of data outside the model’s pretraining corpus, like every single business’ internal data.
  - Mars008 1y ago
    It still makes sense to use external data storage for smaller local models. They just can't hold that much.
- Mars008 1y ago
  One problem is in datasets which use RAG. Training foundational model requires a lot of samples, and there aren't many. The only option is to use other models to generate, so called distillation. BTW, RAG is similar to web search. Models can do it. Web server for RAG can be implemented.
esafak 1y ago
The models can't tell when they shouldn't extrapolate and simply need more information. Which rules can be generalized and which ones can't. Why shouldn't a method `doWhizBang()` exist if there methods for all sorts of other things? When I was young, I once beamed that my mother was a good cooker. It made perfect sense based on other verbs, but I did not know that that word was already claimed by machines, and humans were assigned the word cooks. Decades later, I had the pleasure of hearing my child call me a good cooker...
- lblume 1y ago
  This made me think – the fact that the underlying rule that nouns e.g. representing activities such as cooking can be formed from the corresponding verb via the suffix -er breaks in this case is just a historical / cultural artifact of languages, a type of completely unnecessary complication from the machines' standpoint. Maybe LLM hallucination might also partially be caused by this exception-based social modelling forced via our training data into all model architectures?
  - deleted 1y ago
    [deleted]
ethan_smith 1y ago
Some newer architectures like DeepSeek-V2 and Llama 3.1 have actually shown significant factuality improvements through architectural changes alone, including improved attention mechanisms and training objectives specifically targeting hallucination reduction.

tormeh 1y ago

To me, the issue seems to be that we're training transformers to predict text, which only forces the model to embed limited amounts of logic. We'd have to find something different to train models on in order for them to stop hallucinating.

lblume 1y ago
Modern neuroscience suggests that everything the human brain might be doing is basically a kind of predictive processing, i.e. hallucination based on inductive biases. I do not think this is the main bottleneck.

bsenftner 1y ago

I'm still thinking about how RAG being conceptually simple and easy to implement, why the foundational models have not incorporated it into their base functionality? The lack of that strikes me as a negative point about RAG and it's variants, because if any of them worked, it would be in the models directly and not need to be added afterwards.

bavell 1y ago
RAG is a prompting technique, how could they possibly incorporate it into the pre training?
- maleldil 1y ago
  CoT is a prompting technique too, and it's been incorporated.
  - bavell 1y ago
    IIUC, CoT is "incorporated" into training by just providing better quality training data which steers the model towards "thinking" more deeply in its responses. But at the end of the day, it's still just regular pre training. RAG - Retrieval augmented generation - how can the retrieval be done during training? RAG will always remain external to the model. The whole point is that you can augment the model by injecting relevant context into the prompt at inference time, bringing your own proprietary/domain-specific data.
    - bsenftner 1y ago
      Who says "during training"? RAG could be built into the functionality of the LLM directly - give it the documents you want it to incorporate, and it ingests them as a temp mini-fine tune. That would work just fine.
    - impossiblefork 1y ago
      These things with <think> and </think> tokens are actually trained using RL, so it's not like GSM8k or something like that where you just train on some reasoning. It's actually like QuietSTaR but with a focus on a big thought in the beginning and with more sophisticated RL than just REINFORCE (QuietSTaR uses REINFORCE).
- bsenftner 1y ago
  The same way developers incorporate it now. Why are you thinking "pre-training", this is a feature of the deployed model: it ingests documents and generates a mini-fine tune right then.
mdp2021 1y ago
Why would be a proper documents-at-hand based inquiry be «simple». Information is at paragraph #1234 of book B456; that paragraph acquires special meaning in light of its neighbours, its chapter, the book. Further information is in other paragraphs of other books. You can possibly encode with some "strong" compression information (data), but not insight. The information that a query may point to can be a big cloud of fuzzy concepts. What do you input, how? How big should that input be? "How much" of the past reflection does the Doctor use to build a judgement? RAG seems simple because it has simpler cases ("What is the main export of Bolivia").
rybosome 1y ago
Well, even if we assume for a moment that we aren’t talking about non-public data… Then RAG which serves up knowledge already in the model’s pretraining data is still useful, because it primes the model for the specific context with which you want to engage it. I maybe can see what you are saying, like why can’t the model just do a good job without being re-reminded? But even in that sense, any intelligence, artificial or otherwise, will do better given more context. And that ignores the reality of data outside the model’s pretraining corpus, like every single business’ internal data.
- Mars008 1y ago
  It still makes sense to use external data storage for smaller local models. They just can't hold that much.
Mars008 1y ago
One problem is in datasets which use RAG. Training foundational model requires a lot of samples, and there aren't many. The only option is to use other models to generate, so called distillation. BTW, RAG is similar to web search. Models can do it. Web server for RAG can be implemented.

esafak 1y ago

The models can't tell when they shouldn't extrapolate and simply need more information. Which rules can be generalized and which ones can't. Why shouldn't a method `doWhizBang()` exist if there methods for all sorts of other things? When I was young, I once beamed that my mother was a good cooker. It made perfect sense based on other verbs, but I did not know that that word was already claimed by machines, and humans were assigned the word cooks. Decades later, I had the pleasure of hearing my child call me a good cooker...

lblume 1y ago
This made me think – the fact that the underlying rule that nouns e.g. representing activities such as cooking can be formed from the corresponding verb via the suffix -er breaks in this case is just a historical / cultural artifact of languages, a type of completely unnecessary complication from the machines' standpoint. Maybe LLM hallucination might also partially be caused by this exception-based social modelling forced via our training data into all model architectures?
- deleted 1y ago
  [deleted]

ethan_smith 1y ago

Some newer architectures like DeepSeek-V2 and Llama 3.1 have actually shown significant factuality improvements through architectural changes alone, including improved attention mechanisms and training objectives specifically targeting hallucination reduction.