3 ms·
I have been trying to understand the hype as well. Happy to see all the work happening in this space still. I was pretty curious about the context limit. I am
by infecto 3y ago
I have been trying to understand the hype as well. Happy to see all the work happening in this space still.
I was pretty curious about the context limit. I am not an expert in this area but I always thought the biggest problem was the length of your original text. So typically you might only encode a sentence or a selection of sentences. You could always stuff more in but they you are potentially losing the specificity, I would think that is a function of the dimensionality. This model is 768, are they saying I can stuff 8k tokens worth of text and can utilize it just as well as I have with other models on a per 1-3 sentence level?
- infecto 3y agoThinking about it some more as I read through more comments. I guess in the stated case of research papers it can make sense if your task is looking for the common themes and not specific details. If you are embedding a sentence or a paragraph you miss out on the connection between those sentences across the whole paper...or at least its harder to manage that. By encoding a large number of pages from the paper (or the entire paper) you can hopefully do a better job of capturing the theme of that paper. This also opens up another question though, how would that compare to using a LLM to summarize that paper and then just embed on top of that summary.
- stormfather 3y agoI would guess that the embedded summary is better, but for many tasks where you use embeddings (like document search), summarizing every document with an LLM is too expensive and slow.