3 ms·
I can't say it's totally surprising you need a machine (store, stack, instructions) to handle higher language issues that GPT targets Not to crit the paper: fo
by ggm 4y ago
I can't say it's totally surprising you need a machine (store, stack, instructions) to handle higher language issues that GPT targets
Not to crit the paper: formally stating things is good. But it feels like the outcome was highly predictable. Inferences being drawn from the language model where the language is a real world language and not a restricted subset were always going to hit up against the complexities of real languages.
And I'd be mindful that conforming to Chomsky Hierarchies says nothing about the nature of universal language models, or {{GPT or turing machines} and emergent intelligence}. Which again, isn't to lay things on the authors. People have a habit (here, in general) of taking things out of context.
I kind of feel thats about it: If the analysis of the classifier architecture shows constructs which map to a machine capable of being the higher Chomsky class, then its likely that reflects the complexity of the languages being analysed.
- cma 4y agoMatching braces is part of written natural language and would be beyond the power of regular languages wouldn't it? I thought there was something special with transformers allowing to learn that at least within its context window size in a generalizable way. It also sounds like from the paper transformers are formally proven to capable of it, but the training can't get the weights to make it capable of it, even though there should exist such weights, if I am reading them right.
- ben_w 4y agoI've seen humans mess up matching braces. I've even been that human already this year, only finding out because the compiler told me.
- randorand0 4y agothis may well be the only real skill of programmers who learned C style syntaxes, andor LISPs on the ohter hand, thinking with the parenthesis takes language up to the level of functional-thinking. computable stuff is all about substitution of all things inside a parenthesis for a single thing. and this is the essence of all classic computation
- cma 4y agoBut we still generalize it to bigger and bigger without seeing matching brace problems of all sizes we can handle. In the paper I don't think they are looking for perfect, but they show which models can't seem to learn significantly beyond the size of examples they explicitly saw.
- ben_w 4y agoThe {specific (number I (messed up was a depth of 3)}. I missed it because it was spaced over a few lines and I was tired.
- cma 4y agoBut not because you only trained up to examples of two.
- ben_w 4y agoI don't know what point you're making, GPT-3 almost certainly has training examples for greater depth. Pattern recognition can fail. Mine does, GPT's does. I count further than I subitize, but from experience my mind may wander and I may lose track after 700 of a thing even if there's literally nothing else to focus on. But I still didn't notice an error at depth 3.
- cma 4y agoIt seems transformers don't generalize in this class and neural turing machines and neural stack machines do. You get the general rule (in linguistics known as competency) but can't flawlessly do it (performance). Transformers can't seem to get competence here. https://en.m.wikipedia.org/wiki/Linguistic_competence https://en.m.wikipedia.org/wiki/Linguistic_competence
- ben_w 4y agoAh, gotcha. Thanks for the clear explanation and the link! :)
- charcircuit 4y agoIf there is a finite input it is regular.
- smadge 4y agoAs in you construct a FSM that can recognize all strings of a context free language less than a certain length?
- deleted 4y ago[deleted]
- Kranar 4y agoYou might mean something like if you restrict the set of all valid inputs to N characters then the language is regular, but that's a much stronger condition than requiring the input to be finite. Furthermore, such a restriction makes the language non-recursive; it's effectively a trivial language. In standard automata theory, all inputs to automatons are finite.
- charcircuit 4y agoYes, I was hoping people would understand that from the context.
- cma 4y agoThe paper covers that. They only test to a finite size, but the models that fail can't generalize significantly beyond the size of explicit examples they were trained on. Formally transformers should be capable of modeling it with the right weights, but they show good evidence those weights can't be learned with gradient descent. Whereas a neural turing machine or stack machine can learn them with gradient descent.
- ashupadhi01 4y agoCan you tell some resources which helped you gain a sound understanding of this domain ?
- ggm 4y agoNo, because I regard my understanding as unsound.
- randorand0 4y agoeven better, please share. I'm thinkin if your understanding were sound I would have to listen to it, but I rather read about these kinds of things, so whis is why I say unsound is best on the other hand, putting things down into writing is difficult. So while you diminish this paper's accomplishments, you couldn't have done it yourself in spite of easily understanding it.