3 ms·
If you do a lot of retrieval augmentation, then the 8x cost might still be cheaper than burning a lot of tokens on injected context.
by danielbln 3y ago
If you do a lot of retrieval augmentation, then the 8x cost might still be cheaper than burning a lot of tokens on injected context.
- BoorishBears 3y agoFine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)
- danielbln 3y agoCould you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.
- minimaxir 3y agoThere is a risk of (greater) hallucination if there aren't explicit sample inputs provided by RAG.
- huac 3y agoFinetuning teaches skills. RAG provides 'facts.'
- lukev 3y agoHallucination, but also the ability to cite a source. A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.
- stavros 3y agoSpeaking of RAG, does anyone know of a Python library that (hopefully) doesn't need a server, that can do embeddings, along with an example? I want to do RAG in Python, and I can wrangle the prompt to where I want it, but I have no idea how to basically search for relevant documents with embeddings.
- lukev 3y agohttps://www.trychroma.com https://www.trychroma.com It works as a library, you can require the PIP package and get started immediately.
- stavros 3y agoThis looks great, thank you!
- J_Shelby_J 3y agoHere is my implementation used for my discord and slack bots. The logic for rag is the Shelby_agent module. RAG is straightforward but i improve it by generating extra keywords to help the search. I also add extra metadata filtering to return both technical documentation and more fluffy sales and marketing stuff. IMO, semantic search alone might not be the best, and I want to also add keyword search. I think the thing most people will struggle with is prepping the documents. Chunking them to the best token count for your use case for example. https://github.com/shelby-as-a/shelby-as-a-service https://github.com/shelby-as-a/shelby-as-a-service
- stavros 3y agoThat's very helpful, thanks!
- stavros 3y agoI know I left another comment, but I'm replying in a new one to increase the chance that you'll see this: It would probably be very useful to a lot of people if you (or someone) made a standalone library out of this. It looks like Shelby requires Pinecone, but using something like ChromaDB for local embeddings and providing a library that I can just import, add documents to, and then query for answers would enable all kinds of applications. I'm currently trying to build a hobby virtual assistant, and it looks like RAG does need some nuance to implement correctly (from what you said). A library that does all of what you mentioned would take a lot of work out of it.
- 3y ago
- davedx 3y agoThat doesn't make sense - if you're doing retrieval augmentation, then every prompt will be different depending on the query, right? So you still need to burn those tokens.
- danielbln 3y agoYou can persist the knowledge you would normally retrieve dynamically and inject into the context at runtime via finetuning directly into the model weights. This means you won't have to pull in a glob of additional data with every prompt, but only need to push the query through, which can be answered by the data persisted in the weights. However, as others have pointed out, you lose the ability to reference sources reliably and the output is much more prone to hallucination.
- davedx 3y agoAhh understood, thank you.