3 ms·
One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them.
by freediver 3y ago
One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them.
Future LLMs are likely to have much smaller size and have outside long term memory/training knowledge as well as work memory (a'la RAG approach).
- adamgordonbell 3y agoBut in compressing all that data, they seem to build up internal representations that let them generalize on data they haven't seem. That reasoning ability, if not the actual data, is what gives them their power.
- eldenring 3y agoIsn't this just ignoring all the lessons we learned from pretraining large models? Seems like a different flavor of the bitter lesson.