6 ms·

Fine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (id

Fine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)

danielbln 3y ago
Could you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.
- minimaxir 3y ago
  There is a risk of (greater) hallucination if there aren't explicit sample inputs provided by RAG.
  - huac 3y ago
    Finetuning teaches skills. RAG provides 'facts.'
- lukev 3y ago
  Hallucination, but also the ability to cite a source. A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.
- stavros 3y ago
  Speaking of RAG, does anyone know of a Python library that (hopefully) doesn't need a server, that can do embeddings, along with an example? I want to do RAG in Python, and I can wrangle the prompt to where I want it, but I have no idea how to basically search for relevant documents with embeddings.
  - lukev 3y ago
    https://www.trychroma.com https://www.trychroma.com It works as a library, you can require the PIP package and get started immediately.
    - stavros 3y ago
      This looks great, thank you!
  - J_Shelby_J 3y ago
    Here is my implementation used for my discord and slack bots. The logic for rag is the Shelby_agent module. RAG is straightforward but i improve it by generating extra keywords to help the search. I also add extra metadata filtering to return both technical documentation and more fluffy sales and marketing stuff. IMO, semantic search alone might not be the best, and I want to also add keyword search. I think the thing most people will struggle with is prepping the documents. Chunking them to the best token count for your use case for example. https://github.com/shelby-as-a/shelby-as-a-service https://github.com/shelby-as-a/shelby-as-a-service
    - stavros 3y ago
      That's very helpful, thanks!
    - stavros 3y ago
      I know I left another comment, but I'm replying in a new one to increase the chance that you'll see this: It would probably be very useful to a lot of people if you (or someone) made a standalone library out of this. It looks like Shelby requires Pinecone, but using something like ChromaDB for local embeddings and providing a library that I can just import, add documents to, and then query for answers would enable all kinds of applications. I'm currently trying to build a hobby virtual assistant, and it looks like RAG does need some nuance to implement correctly (from what you said). A library that does all of what you mentioned would take a lot of work out of it.
      - 3y ago

danielbln 3y ago

Could you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.

minimaxir 3y ago
There is a risk of (greater) hallucination if there aren't explicit sample inputs provided by RAG.
- huac 3y ago
  Finetuning teaches skills. RAG provides 'facts.'
lukev 3y ago
Hallucination, but also the ability to cite a source. A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.
stavros 3y ago
Speaking of RAG, does anyone know of a Python library that (hopefully) doesn't need a server, that can do embeddings, along with an example? I want to do RAG in Python, and I can wrangle the prompt to where I want it, but I have no idea how to basically search for relevant documents with embeddings.
- lukev 3y ago
  https://www.trychroma.com https://www.trychroma.com It works as a library, you can require the PIP package and get started immediately.
  - stavros 3y ago
    This looks great, thank you!
- J_Shelby_J 3y ago
  Here is my implementation used for my discord and slack bots. The logic for rag is the Shelby_agent module. RAG is straightforward but i improve it by generating extra keywords to help the search. I also add extra metadata filtering to return both technical documentation and more fluffy sales and marketing stuff. IMO, semantic search alone might not be the best, and I want to also add keyword search. I think the thing most people will struggle with is prepping the documents. Chunking them to the best token count for your use case for example. https://github.com/shelby-as-a/shelby-as-a-service https://github.com/shelby-as-a/shelby-as-a-service
  - stavros 3y ago
    That's very helpful, thanks!
  - stavros 3y ago
    I know I left another comment, but I'm replying in a new one to increase the chance that you'll see this: It would probably be very useful to a lot of people if you (or someone) made a standalone library out of this. It looks like Shelby requires Pinecone, but using something like ChromaDB for local embeddings and providing a library that I can just import, add documents to, and then query for answers would enable all kinds of applications. I'm currently trying to build a hobby virtual assistant, and it looks like RAG does need some nuance to implement correctly (from what you said). A library that does all of what you mentioned would take a lot of work out of it.
    - 3y ago