4 ms·
It’s been interesting watching the open source community get ROPE working well. I find it interesting that the method appears to be as good as retraining a larg
by lettergram 3y ago
It’s been interesting watching the open source community get ROPE working well. I find it interesting that the method appears to be as good as retraining a large model.
To me, that reads like models can be much smaller, understanding of human language at a fraction the size. But we then need a technique (such as this) to expand the context window to interpret larger inputs.
This could dramatically democratize model development.
- soulofmischief 3y agoThe thing is, GPT-4 isn't useful to me because it understands language. It's useful to me because it understands a wide variety of advanced, complex phenomena and is able to synthesize these insights into novel solutions to all sorts of problems. A lot of knowledge could be taken out and put in a vector search system which dynamically loads context into the input, but some knowledge seems worth embedding into the model. The question becomes what knowledge is critical in order to create a useful, large-context language model which can run on the edge.