5 ms·
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression the
by andy99 3y ago
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses.
Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
- kristopolous 3y agoIs this what you mean by RAG? https://www.promptingguide.ai/techniques/rag https://www.promptingguide.ai/techniques/rag?
- teaearlgraycold 3y agoYes
- simonw 3y agoI have an explanation of RAG in the context of embeddings here: https://simonwillison.net/2023/Oct/23/embeddings/#answering-questions-with-retrieval-augmented-generation https://simonwillison.net/2023/Oct/23/embeddings/#answering-...
- Grimburger 3y agoYou could just sum it up for us all rather than do a divert to your blog? It's Retrieval Augmented Generation btw. To quote: > The key idea is this: a user asks a question. You search your private documents for content that appears relevant to the question, then paste excerpts of that content into the LLM (respecting its size limit, usually between 3,000 and 6,000 words) along with the original question. > The LLM can then answer the question based on the additional content you provided.
- simonw 3y ago> You could just sum it up for us all rather than do a divert to your blog? Why? Have links gone out of fashion? I even linked directly to the relevant section rather than linking to the top of the page. The paper that coined the term used the hyphen, though I think I prefer it without: https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401
- Grimburger 3y ago> Have links gone out of fashion? Yes. You wrote far more words than needed to answer the comment, I did it for you instead.
- simonw 3y agoOne of the reasons I write so much stuff is so I can provide links to things I've written to answer relevant questions.
- scubbo 3y agoAnd those of us with the sense to value your insight, and the attention-span to read more than tweet-sized content, thank you for it.
- mhog_hn 3y agoThank you, nice blog.
- discordance 3y agoThanks so much for your writings and for posting the link (and also for Datasette!). I've learned in the past few months from your blog.
- monkeydust 3y agoAppreciate it. Your posts in general have been great - accessible to a large audience, quality links to follow up research and catchy analogies even when they don't fully hold true (llm as a calculator for words - which I admit I use with citation!). Keep going.
- deleted 3y ago[deleted]
- teaearlgraycold 3y agoYou could get a facsimile to a summary for a full article or short story. Reducing an 8k token article to a summary using a completions model would cost far more. So if you need to search through collections of contracts, scientific papers, movie scripts, etc. for recommendations/clustering then bigger input sizes can do that in one shot. Think of it like skipping the square root step in Euclidean distance. Perfectly valid as long as you don’t want a distance so much as a way to compare distances. And doing so skips the most computationally expensive operation.
- refulgentis 3y agoI think I'm missing something: like, yeah, it's vector search for bigger text chunks. But arguably vector search with bigger text chunks is _definitively_ worse -- this isn't doing summarization, just turning about 25 pages of text to 1024 floats, which you then can use cosine similarity to measure the semantic similarity to other text I'd much rather know what paragraph to look in than what 25 pages to look in
- simonw 3y agoI imagine it's more useful for finding related articles and clustering things than for semantic search, which will work much better against smaller chunks - especially if you're implementing Retrieval Augmented Generation.
- rolisz 3y agoI think the point is: if you compress 25 pages of text into 1024 floats, you will lose a ton of information, regardless of what the use case is, so you're probably still better of with chunking.
- simonw 3y agoI've been getting great results for related documents by embedding entire blog posts, e.g. here: https://til.simonwillison.net/gis/pmtiles#related https://til.simonwillison.net/gis/pmtiles#related I'm not sure how I would do that after chunking.
- dragonwriter 3y ago> What is the use case for an 8k token embedding? Calculating embeddings on larger documents than smaller-window embedding models. > My (somewhat limited) experience with long context models is they aren't great for RAG. The only reason they wouldn't be great for RAG is that they aren't great at using information in their context window, which is possible (ISTR that some models have a strong recency bias within the window, for instance) but I don't think is a general problem of long context models. > Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? I would say the usual use is for search and semantic similarity comparisons generally. RAG is itself an application of search, but its not the only one.
- 3abiton 3y agoI wonder how the perfomance fair when context size is increased. Intuitively this should be higher, but some quantized models I've tested showed noticeably worst performance.
- Kubuxu 3y agoYour KV cache size is linear with context size which might put you tight on memory. There is also increased cost of recalculating KV cache of context window when the window has to move but this is close to being solved with streaming LLMs.
- woadwarrior01 3y agoBERT style encoder-only models, like the embedding model being discussed here, don't need a KV cache for inference. A KV cache is only needed for efficient inference with encoder-decoder and decoder-only (aka GPT) models.