3 ms·
Sure, RAG isn't ideal. I don't know of an alternative. Attempting to constantly train it fine-tune entire new models to update their knowledge doesn't appear to
by simonw 1y ago
Sure, RAG isn't ideal. I don't know of an alternative. Attempting to constantly train it fine-tune entire new models to update their knowledge doesn't appear to be practical - I've not seen anyone demonstrate that working.
I think long context plus tricks with tools is the best solution we have right now.
- simianwords 1y agoThe balance may tip in favour of fine tuning once we have made small breakthroughs in this space. It might be especially useful in enterprise contexts where you can have one model per company trained through all Wiki, code, documentation etc.
- simonw 1y agoThat right there is the thing I'm most skeptical of. It's so very obviously what every company wants: a custom model fine-tuned on their internal documentation and code. And yet stories of it actually working are incredibly rare! The closest I've heard to a success story in that space is Jane's Street who fine-tuned their own model because they use OCaml more than anyone else: https://www.youtube.com/watch?v=0ML7ZLMdcl4 https://www.youtube.com/watch?v=0ML7ZLMdcl4 I am confident that any startup today who could provably demonstrate that "we can fine tune a model on your company's internal code and documentation and have it answer questions about them" would have enormous financial success. I'll believe it works when I see that!