4 ms·
Regarding the "failed paradigm" (i.e: of logic- and knowledge-based AI) and how neural nets scale, specifically on language modelling tasks. I study a class of
by YeGoblynQueenne 7y ago
Regarding the "failed paradigm" (i.e: of logic- and knowledge-based AI) and
how neural nets scale, specifically on language modelling tasks.
I study a class of machine learning algorithms that learn logic programs (as
in Prolog, or ASP) from examples. The field is Inductive Logic Programming.
ILP algorithms are characterised by their ability to incorporate background
knowledge and impose strong inductive biases on the hypothesis space to learn
accurately from a handful of examples.
My favourite example of this is learning a grammar of the aⁿbⁿ language. My
implementation of one of the algorithms that my research group studies (I'm a
Phd research student) is Thelma:
https://github.com/stassa/thelma https://github.com/stassa/thelma
In Thelma's README on github you will find an example of Thelma learning a
grammar of the aⁿbⁿ language from three positive examples and no negative
examples. The learned grammar generalises to any n. In typical CFG notation
it's this grammar:
S → AB
S → AS₁
S₁ → B
A → a
B → b
(The nonterminal S₁ is invented, i.e. it was not given in the original
learning problem defintion. The preterminals A and B are background
knowledge.)
By contrast, neural networks can only learn fragments of this grammar up to
some limited n and that only if they're given tens of thousands of training
examples.
For instance, a classic paper by Gers and Schmidhüber [1] claims LSTM
"generalisation" from training sets of 22 to 42 thousand aⁿbⁿ strings, but
their average generalisaion is, e.g. from n in [1,50] (i.e. that's the value
of n in training strings) to n in [1,430] (n in testing strings) and their
"best" generalisation is at most to n in [1,1000].
On Thelma's README on github I have a small testing query that tests how the
aⁿbⁿ grammar it learned from 3 positive examples with n in [1,3] generalises
to an aⁿbⁿ string of e.g. n = 100,000:
?- _N = 100_000, findall(a, between(1,_N,_), _As), findall(b, between(1,_N,_),_Bs), append(_As,_Bs,_AsBs), anbn:'S'(_AsBs,[]).
true .
Well, it's a correct grammar so it generalises perfectly. You can bind _N to
any number your computer memory will allow and it will still parse.
So, besides the plug of my work (sorry) I would say that the earlier paradigm
that you say has "failed" with respect to recent connectionist success is
alive and well and it still gets many things right that the connectionist
paradigm struggles with mightily, in particular robust generalisation from
very few examples (without expensive pre-training etc), and of course any task
that requires reasoning [2].
______________
[1] "LSTM recurrent networks learn simple context free and context sensitive
languages": https://ieeexplore.ieee.org/document/963769 https://ieeexplore.ieee.org/document/963769 See table 2 on page 7
for the results I quote above.
[2] The examples of good performance on reasoning tasks you bring up have been
strongly criticised, e.g. in https://arxiv.org/abs/1907.07355 https://arxiv.org/abs/1907.07355 (Probing Neural
Network Comprehension of Natural Language Arguments) or Gary Marcus' recent
paper on a critical appraisal of deep learning etc.
P.S. I'm very sorry to see you're being downvoted.
- dooglius 7y agoThe fact that logic programming is still looking at "languages" like a^nb^n while neural nets can generate at least somewhat sensible English only serves to underscore how utterly the field has failed. If you think that such simple toy problems are impressive, you really need to get out of your bubble.
- deleted 7y ago[deleted]
- YeGoblynQueenne 7y agoApologies if this sounds rude, but that sounds like sour grapes to me. It's not that learning anbn is not a toy problem. It is. But it's a toy problem for ILP. Neural nets can't solve it. And neither can any other statistical machine learning algorithms. I'd say that it's rather statistical machine learning research that is in a bubble, satisfied to solve the same old classification problems again and again, apparently oblivious to the fact that there are many problems that their algorithms can't touch because they are neither differentiable, nor reducible to classification. Unfortunately, since statistical machine learning is the current dominant paradigm, success is measured in terms of what it can do- and anything it can't do is either completely ignored or discounted. Or, like I say: sour grapes.
- dooglius 7y ago> It's not that learning anbn is not a toy problem. It is. But it's a toy problem for ILP. I don't see why the distinction you are making matters. > Neural nets can't solve it. They can, it takes additional training to do so from scratch. Also, the work you cited is also nearly two decades old, I have no idea how SOTA would perform. > solve the same old classification problems again and again I truly have no idea what you're talking about, please point me to where, for instance, [0] has been solved "again and again"? The claim I am making is not that Deep Learning can solve all problems, it cannot, the claim is that logic-based AI has failed to come up with anything of value whatsoever, whereas statistical ML has had sweeping results across many real problems -- for instance: AlphaZero solving Go, Chess, and Shogi; GPT-2 creating readable stories given a prompt; cars that can drive themselves under a wide range of circumstances. If you think a^nb^n is even remotely comparable to those feats, I don't think we can find any common ground. [0] https://www.youtube.com/watch?v=Wwwyr7cOBlU https://www.youtube.com/watch?v=Wwwyr7cOBlU