5 ms·
It is trivial to fine tune these days. RAG is already irrelevant with large context windows.
by jnwatson 3y ago
It is trivial to fine tune these days. RAG is already irrelevant with large context windows.
- Xenoamorphous 3y ago> RAG is already irrelevant with large context windows Just last Friday I took the contents of the 2024 folder of one of the teams at the company I work for, for which we use RAG at the moment. I dumped the text index, concatenated it and used Google’s API to return the token count, to see if it would fit in Gemini’s 1M context window; turned out it was 5.7M tokens. And that’s less than 3 months worth of documents for that team. So yeah RAG is not dead yet, although I do question its usefulness, but that’s a separate topic.
- greenavocado 3y agoDid I read this correctly? You uploaded millions of words of your company's internal communications to Google?
- Xenoamorphous 3y agoI did. But this is under an enterprise deal with them that warrants privacy, not the generally available stuff. OpenAI has similar arrangements (Enterprise ChatGPT) and MS Azure before them.
- simonw 3y agoCitation needed on "trivial to fine tune".
- jnwatson 3y agoCheck out Google's AI Studio. It makes it easy to fine tune. Disclaimer: I work for Google.
- danielmarkbruce 3y agoThere is no citation needed. It is indeed trivial to fine-tune. Doing a good job is another matter, but the claim is correct. Google around and find a blog post showing how. The claim that RAG is dead is obviously wrong.
- simonw 3y agoFor "citation needed", read "please link me to a blog post showing how, don't just tell me to Google for one". The internet is full of blog posts about this. That doesn't mean they're actually good - I'd love to be pointed at one that has proven itself useful for someone (and definitely isn't just LLM blog-spam). I don't care if it's trivial to fine-tune and get crap results - I care about fine-tuning where the result was worth the effort. For the record, my favourite guide to fine-tuning is the section of this Jeremy Howard video that shows how to train a text-to-SQL model: https://www.youtube.com/watch?v=jkrNMKz9pWU&t=4850s https://www.youtube.com/watch?v=jkrNMKz9pWU&t=4850s
- danielmarkbruce 3y agoIt's an internet forum, not an academic journal. Water tight arguments are not needed. If one wants to call bs, they can just do it, no need to dance around the topic by asking for a citation.
- simonw 3y agoOK, I call BS. Fine-tuning an LLM is not "trivial" - especially if you want to get useful results, as opposed to just being able to say "look, I fine-tuned an LLM".
- danielmarkbruce 3y agoYup, largely agree.
- wakaru44 3y agoExactly, it's a forum not Twitter/reddit. Without references and citations this is no better than a bunch of random words, and it's hard to make any argument of substance. The person asked for citations, leave it be, stop dudexplaining how Internet works for you please. I call bs on your 2 comments.
- SgtBastard 3y agoA remarkable comment in that it is clear, confident and wrong. Fine-tunes lead to catastrophic forgetting. RAG is only irrelevant if you’re completely disinterested in cost and latency. We also don’t have enough data to gauge performance of models >200k context window size when reasoning over inputs of that size, much of which will be irrelevant to any particular user. Multiple random needles in haystack tests work flawlessly, but rarely applies to real world activity.