6 ms·
After a year the entire argument you make boils down to “so far”.
by gmerc 3y ago
After a year the entire argument you make boils down to “so far”.
- ra 3y agoIndeed. LLM is an application on a transformer trained with backpropagation. What stops you from adding a logic/mathematic "application" on the same transformer?
- seanhunter 3y agoNothing, and there are methods which allow these types of models to learn to use special purpose tools of this kind[1]. [1] https://arxiv.org/abs/2302.04761 https://arxiv.org/abs/2302.04761 Toolformer: Language Models Can Teach Themselves to Use Tools
- Terr_ 3y agoWhereas your post sounds like "Just give the approach more time, it shall continue to incrementally improve until it finally works someday, cuz reasons." Early attempts at human flight approached it by strapping wings to people's arms and flapping: Do you think that would have eventually worked too, if only we had just given it a bit more time and faith?
- xcv123 3y ago> Just give the approach more time, it shall continue to incrementally improve until it finally works someday, cuz reasons Yes. Because we haven't yet reached the limit of deep learning models. GPT-3.5 has 175 billion parameters. GPT-4 has an estimated 1.8 trillion parameters. That was nearly a year ago. Wait until you see what's next.
- meheleventyone 3y agoWhy would adding more parameters suddenly make it better at this sort of reasoning? It feels a bit of a “god of the gaps” where it’ll just stop being a stochastic parrot in just a few more million parameters.
- Al-Khwarizmi 3y agoI don't think it's guaranteed, but I do think it's very plausible because we've seen these models gain emerging abilities at every iteration, just from sheer scaling. So extrapolation tells us that they may keep gaining more capabilities (we don't know how exactly it does it, though, so of course it's all speculation). I don't think many people would describe GPT-4 as a stochastic parrot already... when the paper that coined (or at least popularized) the term came up in early 2021, the term made a lot of sense. In late 2023, with models that at the very least show clear signs of creativity (I'm sticking to that because "reasoning" or not is more controversial), it's relegated to reductionistic philosophical arguments, but not really a practical description anymore.
- meheleventyone 3y agoI don’t think we should throw out the stochastic parrot so easily. As you say there are “clear signs of creativity” but that could be it getting significantly better as a stochastic parrot. We have no real test to tell mimicry apart from reasoning and as you note we also can only speculate about how any of it works. I don’t think it’s reductionist in light of that, maybe cautious or pessimistic.
- Al-Khwarizmi 3y agoThey can write original stories in a setting deliberately designed to not be found in the training set (https://arxiv.org/abs/2310.08433 https://arxiv.org/abs/2310.08433). To me that's rather strong evidence of being beyond stochastic parrots by now, although I must concede that we know so little about how everything works, that who knows.
- GTP 3y agoI didn't look at the paper but... How do you design a setting in a way that you're sure there isn't a similar one in the training set, when we don't even precisely know what the training set for the various GPT models was?
- tarsinge 3y agoYou are missing the point that it can be a model limit. LLMs were a breakthrough but that doesn’t mean they are a good model for some other problems, no matter the number of parameters. Language contains more than we thought, as GPT has impressively showed (ie semantics embedded in the syntax emerging from text compression), but still not every intellectual process is language based.
- xcv123 3y agoI know that, but deep learning is more than LLMs. Transformers aren't the final ultimate stage of deep learning. We haven't found the limit yet.
- tarsinge 3y agoYou were talking about the number of parameters on existing models. Like the history of Deep Learning has shown, simply throwing more computing power at an existing approach will plateau and not result in a fundamental breakthrough. Maybe we'll find new architectures, but the point was that the current ones might be showing their limits, and we shouldn't expect the model suddenly become good at something they are currently unable to handle because "more parameters".
- xcv123 3y agoYes you're right I only mentioned the size of the model. The rate of progress has been astonishing and we haven't reached the end, in terms of both of size and algorithmic sophistication of the models. There is no evidence that we have reached a fundamental limit of AI in the context of deep learning.
- lelanthran 3y agoEver heard of something called diminishingly returns? The value improvement between 17.5b parameters and 175b parameters is much greater than the value improvement between 175b parameters and 18t parameters. IOW, each time we throw 100 times more processing power at the problem, we get a measly 2 time increase in value.
- xcv123 3y agoYes that's a good point. But the algorithms are improving too.
- londons_explore 3y ago> Early attempts at human flight approached it by strapping wings to people's arms and flapping: Do you think that would have eventually worked too, if only we had just given it a bit more time and faith? Interestingly, we how have human powered aircraft... We have flown ~60km with human leg power alone. We've also got human powered ornithopters (flapping wing designs) which can fly but only for very short times before the pilot is exhausted. I expect that another 100 years from now, both records will be exceeded, altough probably for scientific curiosity more than because human powered flight is actually useful.
- ben_w 3y agoI knew about the legs (there was a model in the London Science Museum when I was a kid), but I didn't know about the ornithopter. https://en.wikipedia.org/wiki/UTIAS_Snowbird https://en.wikipedia.org/wiki/UTIAS_Snowbird 13 years ago! Wow, how did I miss that?