4 ms·
Not an expert but surely it's only a matter of time until there's a way to update with the latest information without having to retrain on the entire corpus?
by rcarr 7mo ago
Not an expert but surely it's only a matter of time until there's a way to update with the latest information without having to retrain on the entire corpus?
- Filligree 7mo agoIt’s an extremely difficult problem, and if you know how to do that you could be a billionaire. It’s not impossible, obviously—humans do it—but it’s not yet certain that it’s possible with an LLM-sized architecture.
- Wowfunhappy 7mo ago> It’s not impossible, obviously—humans do it It's still not at all obvious to me that LLMs work in the same way as the human brain, beyond a surface level. Obviously the "neurons" in neural nets resemble our brains in a sense, but is the resemblance metaphorical or literal?
- Yiin 7mo agohttps://www.youtube.com/watch?v=l-OLgbdZ3kk https://www.youtube.com/watch?v=l-OLgbdZ3kk
- jdub 7mo agoDigital neural networks and "neurons" were already vastly simpler than biological neural networks and neurons... and getting to transformers involved optimisations that took us even further away from biomimicry.
- Filligree 7mo agoI didn’t mean “possible for LLMs”; this is clearly an open question. In fact, I didn’t even mean “possible for a neural network the size of an LLM”. I just meant “possible”.
- Wowfunhappy 7mo agoI'm not actually convinced that computers can replicate what our brains do. I don't know that a turing machine is sufficient for that.
- computably 7mo agoOn a technical level, sure, you could say it's a matter of time, but that could mean tomorrow, or in 20 years. And even after that, it still doesn't really solve the intrinsic problem of encoding truth. An LLM just models its training data, so new findings will be buried by virtue of being underrepresented. If you brute force the data/training somehow, maybe you can get it to sound like it's incorporating new facts, but in actuality it'll be broken and inconsistent.