3 ms·
It’s called in-context learning. I wouldn’t call it a trick so much as the most exciting and useful feature of LLMs. A lot of research is being done to expand t
by valine 3y ago
It’s called in-context learning. I wouldn’t call it a trick so much as the most exciting and useful feature of LLMs. A lot of research is being done to expand the context window which would allow you to prime the model with entire books or codebases. GPT4 has a max context window of 32k tokens up from 2k with GPT3.5. Whose to say iteratively shifting the output distribution isn’t a form of learning.
- danielmarkbruce 3y agoIf you want to call in-context learning learning, that's fine. I thought the presented paper was about the model being updated so that it could be used in "fresh contexts", but I might have misunderstood what they meant by "fresh context". I thought they meant next prompt. I think everyone is aware that if you say "I use the word jimbo to mean dog. Write a story about dogs using my way" it will work. https://chat.openai.com/share/617ef2c3-ad3e-4d1f-8e6b-3940f28b1d2c https://chat.openai.com/share/617ef2c3-ad3e-4d1f-8e6b-3940f2...
- valine 3y agoThe model wasn’t update after the initial training, they’re definitely using in-context learning. The advancement here, as I understand it, is that they’ve achieved in-context learning with a very small model. From the paper: >> Few-shot instruction-learning task. MLC was evaluated on this task in several ways; in each case, MLC responded to this novel task through learned memory-based strategies, as its weights were frozen and not updated further.
- danielmarkbruce 3y agoYup, my mistake