3 ms·
The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks. MoE notw
by Balinares 8mo ago
The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks.
MoE notwithstanding, a model trained on the whole Internet and a few hundred thousands stolen books carries way more knowledge than is actually needed for any given workflow. It would be great if we could ship slimmed down models into which we'd plug the knowledge banks useful for today's work, and only those.
It would also mean that you could keep a model's knowledge fresh without retraining the whole of it.
- gitpusher 8mo ago> pluggable knowledge banks. plugs in knowledge bank LLM: ... I know kung fu.
- dormento 8mo agoThis is interesting. Would this mean less space for hallucination as well (depending on the breadth of knowledge applied to a specific task)?
- oliver_dr 8mo ago[dead]
- pennomi 8mo agoAgreed, I suspect that LLMs in the future will have separate (possibly standardized) decoding/encoding layers that plug into logic layers.
- ay 8mo agoIsn’t that what LoRA does ?
- CuriouslyC 8mo agoLoRAs are better at steering models to produce correct answers from their data set than imparting new knowledge.
- kgeist 7mo agohttps://arxiv.org/abs/2603.01097 https://arxiv.org/abs/2603.01097 >Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.