3 ms·
Interesting how two camps are emerging for LLMs. This one is about how GPT actually learns something, the other one represented by Chomsky and Gary Marcus repre
by imranq 4y ago
Interesting how two camps are emerging for LLMs. This one is about how GPT actually learns something, the other one represented by Chomsky and Gary Marcus represent how GPT has learned nothing (https://news.ycombinator.com/item?id=34278243 https://news.ycombinator.com/item?id=34278243)
I think the difference is what "learned" means. This paper basically says that any finite amount of knowledge compression is learning, whereas the other camp defines learning as some kind of infinite information compression like being able to add any two numbers no matter how large, which is something no language model will ever be able to do.
Personally, I think both sides are right, GPT3 has compressed language patterns across the internet, but its undeniable that GPT3 makes up stuff frequently. Overall, the point still stands that these language models are valuable in some specific contexts, but its not clear how far they can go.
- TimTheTinker 4y agoPerhaps learning is best defined as the acquisition of knowledge in the philosophical sense - an increasing collection of justified true beliefs.
- voiper1 4y ago>the other camp defines learning as some kind of infinite information compression like being able to add any two numbers no matter how large, which is something no language model will ever be able to do. I know this isn't strictly an LLM, but can't there be an "extension" that the LLM learns a formula and how to plug values into it (it already seems pretty good at explaining code, so it can already do this) and then new: the ability to actually perform the calculation - execute the code/formula or at least "use a calculator" with the values.
- simonh 4y agoThe paper addresses the same idea, using the same example. Sure, that can be done. I suppose it’s a case of evaluating the prompt and then selecting from various different mechanisms for generating a response, one or several of which might be an LLM. That’s interesting but really out of open for a discussion if LLMs themselves though.
- lostmsu 4y agoHumans can not add any two numbers because of eventual mistake, which makes the second definition trivially useless in regards to comparing human intelligence vs AI.