5 ms·
Garbage in, garbage out.
by gumballindie 3y ago
Garbage in, garbage out.
- kibibu 3y agoAre you suggesting the 10MB of references were garbage? I don't understand what you mean.
- LoganDark 3y agoIs 10MB enough to fine-tune a model nowadays? Assuming that the base model is fairly well-trained already, of course.
- regularfry 3y agoIt's not fine-tuning, it's RAG.
- mirekrusin 3y agoRAGs are great for glossary like structured information. Arbitrarily chunking prose text feels like making garbage out of otherwise good quality text. Ideally prose would be used to generate glossary like documents (LLM aided). Performance plays role here (you'd need to process a lot of text up front) so maybe smaller models could be used?
- gchamonlive 3y agoI think as long as you are below 50% context window, performance should be ok (https://dev.to/maximsaplin/gpt-4-128k-context-it-is-not-big-enough-1h02 https://dev.to/maximsaplin/gpt-4-128k-context-it-is-not-big-...)
- jcims 3y agoThis is something that has always bothered me about RAG. It seems that it’s fine for first order relevance, like a search engine, but for knowledge there needs to be some kind of rumination stage where it revisits the entire corpus to find information that has a second order relevance to round out its ‘understanding’. You might be able to approximate by chunking and globbing the chunks and searching for those, as well as having the LLM summarize and extract data and search for those items as well.
- meehai 3y agoisn't this what llamaindex is doing though?
- LoganDark 3y agoOh, so it's not fine-tuning the model, it's allowing the model to recall things from a database. (I didn't read the article before commenting)
- B1FF_PSUVM 3y agoRetrieval augmented generation, apparently.
- CSSer 3y agoYeah, and I swear that term appeared out of nowhere. I feel like a year ago we were called it vector embeddings, but I’m no expert.
- regularfry 3y agoSometimes the terminology is around but not well distributed. I'm not sure this is the paper that coined it, but here's an example from 2020: https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401
- Der_Einzige 3y agoThese days it’s enough to completely mindbreak a model via DPO. Seeing how modern open source LLMs are willing to do anything was quite a surprise given how locked down ChatGPT is.
- piuantiderp 3y agoJust like democracy
- gumballindie 3y agoI am not sure how democracy is relevant to this post but yes. See the UK - democratically making one poor choice after the other due a steady but obvious cultural decline. Garbage in, garbage out.