3 ms·
Ok but what about transfer learning the LLM, are there practical ways of building more static knowledge into the LLM? Long term memory and all that?
by CMCDragonkai 3y ago
Ok but what about transfer learning the LLM, are there practical ways of building more static knowledge into the LLM? Long term memory and all that?
- BoorishBears 3y agoPutting knowledge in the context window enables the same thing For example, letting the LLM return a result to the user and return a list of entities it wants to save details on for later, then letting it query past entries allows long term memory for a class of entities. Over all it will be a very long time before approaches like that are beat by fine tuning, because as the models improve, this strategy gets stronger: the model gets better are crafting queries, you can provide more context in the context window. Meanwhile fine tuning in knowledge will get more difficult as MOE and other unforseen developments reset what works as far as directly tinkering with weights.