4 ms·
Compression in order for the input data to fit context, maybe? For example if context is 4092 and input size is 6000, figuring out the appropriate way to run an
by jerrygenser 3y ago
Compression in order for the input data to fit context, maybe? For example if context is 4092 and input size is 6000, figuring out the appropriate way to run an operation on all 6000 where context over all 6000 might be relevant to the operation.
- newswasboring 3y agoMaybe I'm not getting it, but I see this as an indexing problem. The goal shouldn't be to fit the entire document in the prompt, we should include relevant part of the doc when we query it. Edit: I'm thinking of something like LlamaIndex
- jerrygenser 3y agoEmbedding chunks and finding chunks based on similarity is definitely in use now. But if you can increase context size cheaply then the model can figure out what's relevant.
- newswasboring 3y agoYeah I get that, after all attension is all you need. But unless you want to spend a bunch of money on the 32k context version I don't think there are other options than embeddings and index.
- cced 3y agoWhat is the state of the art for running models locally in terms of context size?
- newswasboring 3y agoI think local models SOTA is llama which has 2048 context[1]. [1] https://github.com/facebookresearch/llama/issues/16 https://github.com/facebookresearch/llama/issues/16