4 ms·

Are there open models that support these extremely long context lengths? It seems that for a RAG like functionality, this is much easier than retraining the mod

by xrd 3y ago

Are there open models that support these extremely long context lengths? It seems that for a RAG like functionality, this is much easier than retraining the model, even if the quality isn't always perfect. For personal usage I'm very interested in this application.

xrd 3y ago
Later on down it says that Mixtral has a context length of 32k which isn't shabby. That's interesting and probably would work for my needs.
sp332 3y ago
Abacus AI fine-tuned some llama models to 32k context lengths.