5 ms·

RAG sucks. Microsoft is the force behind it because they don't allow training on their chatgpt models. Fine-tuning even Lora on the open source models is nearl

RAG sucks. Microsoft is the force behind it because they don't allow training on their chatgpt models. Fine-tuning even Lora on the open source models is nearly always better than these other approaches

Kiro 3y ago
What do you mean? https://platform.openai.com/docs/guides/fine-tuning https://platform.openai.com/docs/guides/fine-tuning
- whimsicalism 3y ago
  none of those are chatgpt models fwiw
binarymax 3y ago
RAG is the ONLY way to make sure you models are keeping true to facts and source material. Fine-tuning a model before using RAG helps with shaping the style of the summary, and gravitating towards more important facts presented.
- rolisz 3y ago
  It still doesn't give guarantees, it just makes hallucinations much less likely.
- nostrebored 3y ago
  RAG doesn't give you this -- it gives you a higher probability that you're keeping true to facts and source material, but the model may still give you hallucinated responses.
- treprinum 3y ago
  There are other methods than RAG over vector database; you can use basic TF-IDF or even full-text search to find candidate paragraphs and put them into the context.
  - binarymax 3y ago
    Of course. “Retrieval” in RAG doesn’t require a special kind of retriever. As long as the relevance is tuned for the top documents to seed the prompt context, it doesn’t matter what kind of search backend you use.
    - treprinum 3y ago
      Agreed though R in RAG typically means specifically vector search. Not sure how this was called before vector DBs were popularized by LLaMAIndex, probably just retrieval systems with LLMs.
      - binarymax 3y ago
        That's not true. Information Retrieval needs much more than vector search. I've contributed to a book on the subject: https://aipoweredsearch.com https://aipoweredsearch.com
        QuantumGood 3y ago
        Any more info? The book's not published yet, I see.
- natsucks 3y ago
  not to mention staying true to their benchmarks...
sharemywin 3y ago
Is Lora used on language models?
- Tostino 3y ago
  Yes, quite successfully. And qlora, where things are further quantized while just sitting in VRAM.
- whimsicalism 3y ago
  yes
lukev 3y ago
RAG is fundamentally different and there will always be a place for it. A model's weights are inherently lossy and opaque. If a model asserts some fact, it is impossible to tell whether that fact was true or hallucinated just from the model, because the model has no notion of "fact" or "truth", it's just probabilities. Generating an answer solely from model weights is like asking a random person to answer a question from memory. Sure, you might get the right answer, but there's no guarantee. Using RAG is like handing them a book and asking them what the book says the answer is. With the benefit that LLMs can "read" much faster than a human. My take is that LLMs are actually much better at "reading" than they are at "writing", and RAG plays to that strength.
- cubefox 3y ago
  > My take is that LLMs are actually much better at "reading" than they are at "writing", and RAG plays to that strength. They are certainly much faster at reading than at writing. In fact, they re-read the entire context window for every token they write!
  - rolisz 3y ago
    I think there is some caching of some of the computation
simonw 3y ago
"Fine-tuning even Lora on the open source models is nearly always better than these other approaches" Can you expand on that? I've not seen evidence of that myself yet, but maybe I haven't looked in the right places.
- a_ml 3y ago
  From Aug 2022, but still worth a look "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" https://arxiv.org/abs/2205.05638 https://arxiv.org/abs/2205.05638
intellectronica 3y ago
Fine-tuning is coming soon for GPT-3.5-turbo and GPT-4 from both Open AI and Azure. Still, I don't think many users will need it. Fine-tuning is not a solution for getting fresh data as you would with RAG, unless you are planning to run your entire fine-tuning suite for every new document. It can help improve accuracy a bit when you need to specialise the model for a very specific domain or modality. In practice, this is rare and unnecessary for most use cases building on LLMs. The counter argument to your assertion (my own opinion, not Microsoft's) is that the reason you hear so much about fine-tuning from everyone other than Open AI / MS, is that they offer less capable models that can't reliably produce the same quality of results without fine-tuning.
brigadier132 3y ago
How does RAG not meet your expectations? When I use chatgpt I provide source material for my questions and typically get much higher quality responses.

Kiro 3y ago

What do you mean? https://platform.openai.com/docs/guides/fine-tuning https://platform.openai.com/docs/guides/fine-tuning

whimsicalism 3y ago
none of those are chatgpt models fwiw

binarymax 3y ago

RAG is the ONLY way to make sure you models are keeping true to facts and source material. Fine-tuning a model before using RAG helps with shaping the style of the summary, and gravitating towards more important facts presented.

rolisz 3y ago
It still doesn't give guarantees, it just makes hallucinations much less likely.
nostrebored 3y ago
RAG doesn't give you this -- it gives you a higher probability that you're keeping true to facts and source material, but the model may still give you hallucinated responses.
treprinum 3y ago
There are other methods than RAG over vector database; you can use basic TF-IDF or even full-text search to find candidate paragraphs and put them into the context.
- binarymax 3y ago
  Of course. “Retrieval” in RAG doesn’t require a special kind of retriever. As long as the relevance is tuned for the top documents to seed the prompt context, it doesn’t matter what kind of search backend you use.
  - treprinum 3y ago
    Agreed though R in RAG typically means specifically vector search. Not sure how this was called before vector DBs were popularized by LLaMAIndex, probably just retrieval systems with LLMs.
    - binarymax 3y ago
      That's not true. Information Retrieval needs much more than vector search. I've contributed to a book on the subject: https://aipoweredsearch.com https://aipoweredsearch.com
      - QuantumGood 3y ago
        Any more info? The book's not published yet, I see.
natsucks 3y ago
not to mention staying true to their benchmarks...

sharemywin 3y ago

Is Lora used on language models?

Tostino 3y ago
Yes, quite successfully. And qlora, where things are further quantized while just sitting in VRAM.
whimsicalism 3y ago
yes

lukev 3y ago

RAG is fundamentally different and there will always be a place for it. A model's weights are inherently lossy and opaque. If a model asserts some fact, it is impossible to tell whether that fact was true or hallucinated just from the model, because the model has no notion of "fact" or "truth", it's just probabilities. Generating an answer solely from model weights is like asking a random person to answer a question from memory. Sure, you might get the right answer, but there's no guarantee. Using RAG is like handing them a book and asking them what the book says the answer is. With the benefit that LLMs can "read" much faster than a human. My take is that LLMs are actually much better at "reading" than they are at "writing", and RAG plays to that strength.

cubefox 3y ago
> My take is that LLMs are actually much better at "reading" than they are at "writing", and RAG plays to that strength. They are certainly much faster at reading than at writing. In fact, they re-read the entire context window for every token they write!
- rolisz 3y ago
  I think there is some caching of some of the computation

simonw 3y ago

"Fine-tuning even Lora on the open source models is nearly always better than these other approaches" Can you expand on that? I've not seen evidence of that myself yet, but maybe I haven't looked in the right places.

a_ml 3y ago
From Aug 2022, but still worth a look "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" https://arxiv.org/abs/2205.05638 https://arxiv.org/abs/2205.05638

intellectronica 3y ago

Fine-tuning is coming soon for GPT-3.5-turbo and GPT-4 from both Open AI and Azure. Still, I don't think many users will need it. Fine-tuning is not a solution for getting fresh data as you would with RAG, unless you are planning to run your entire fine-tuning suite for every new document. It can help improve accuracy a bit when you need to specialise the model for a very specific domain or modality. In practice, this is rare and unnecessary for most use cases building on LLMs. The counter argument to your assertion (my own opinion, not Microsoft's) is that the reason you hear so much about fine-tuning from everyone other than Open AI / MS, is that they offer less capable models that can't reliably produce the same quality of results without fine-tuning.

brigadier132 3y ago

How does RAG not meet your expectations? When I use chatgpt I provide source material for my questions and typically get much higher quality responses.