3 ms·
Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures, hence for
by YeGoblynQueenne 6y ago
Deep neural nets do not learn only from examples! They encode strong inductive
biases in their carefully hand-engineered and hand-tuned architectures, hence
for example CNNs are used for image recognition and LSTMs for sequence
learning etc. Without these biases deep neural nets would not be able to
generalise as well as they do (in the sense of local generalisation but not
global generalisation as meant by François Chollet [1]).
The biggest advances in deep neural nets have come from the discovery and use
of good inductive biases: training with gradient descent, backpropagation,
more hidden layers, the "constant error carousel", convolutional layers, ReLu
over sigmoid, attention, etc, etc. One could say that deep neural nets are all
about good inductive bias.
It's interesting that you bring up the ARC dataset. The paper that introduced
it (also from Chollet) [2] makes a strong claim about the necessity of
"knowledge priors" for a system to be considered intelligent. These are
described at length in section III.1.2 "Core knowledge priors" and are exactly
a set of strong inductive biases that the author of the paper considers
necessary for a machine learning system to solve the ARC tasks and that
consist of such problem-specific biases as object cohesion, object
persistence, object influence via contact, etc. It is exactly such "knowledge
priors" that are encoded as background knowledge in ILP systems.
Indeed, in the ARC challenge on Kaggle, the best-performing systems (i.e. the
ones that solved the most tasks) were crude approximations of the ILP
approach: a library of hand-crafted functions and a brute-force search
procedure to combine them. I note also that attempts to use deep learning to
solve the challenge didn't go anywhere.
Humans also have strong inductive biases that help us solve such problems. But
I'm not the best placed to discuss all this - I'm not a cognitive scientist.
In the end, what you are asking for is magick: a learner that learns only from
examples, without any preconceived notions about how to learn from those
examples, or what to learn from them. There is no such machine learning
system.
>> Doesn't matter, since the programmer gives that information. As you admit,
Louise cannot check for arbitrary programs.
I don't understand what you mean "check for arbitrary programs". I can give
Louise zero BK and metarules and ask it to generate all Prolog programs, say.
Prolog is a Turing complete language so that would give me the set of all
programs computable by a Universal Turing Machine (it would take a while). But
what would that achieve?
At this point I'm not sure I understand what your remaining objections are
against the approach I showed you. For the purpose of learning arbitrary
programs it works better than anything else. Of course it's not magick.
Perhaps you should take my suggestion to think about the problem a bit more
carefully, if you're really intersted in it. Or are you? I mean, if you
consider AI solved, e.g. by GPT-3, then I can see how you wouldn't be
interested in thinking any further about the issue.
_________________
[1] https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html
[2] https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547
P.S. To clarify, I'm keeping this discussion up for your sake, albeit eagerly.
You have expressed some strongly held, but incorrect opinions that it seems to
me you have acquired by consulting inexpert sources, probably because you have
a day job that has nothing to do with AI and doesn't leave you enough time to
study the matter properly. My day job is to study AI and I feel that such a
privilege is only justified if I spend time and effort to help others improve
their knowledge on the subject. I'm guessing that on your part, you're more
interested in "winning" the conversation, but please try to gain something
from our interaction, otherwise all the time we both spent at it would be to
waste. When this is over, try to dig out and read some authoritative sources.
I would advise you on which ones - but you'd probably resist my recommendation
anyway, so you're on your own there.
- Veedrac 6y agoMy initial response was a fairly kneejerk reaction to the snark. The following is a rewrite. Please don't; if you really think so little of me, rather don't reply than reply unpleasantly. > Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures “Solomonoff Induction does not learn only from evidence! It encodes strong inductive biases in its construction and choice of Turing machine...” but it doesn't matter. Our universe is not a random soup of maximal entropy. The tasks I am talking about solving are overtly not impossible. You talk about ML methods like the success of, say, image recognition comes from image-recognition-specific architectures. You mention ‘hand-engineered’ or ‘hand-tuned’. And yet, to throw your snark back at you, if you were up to date with the literature, you would know this is not true. Consider ViT as an example. The same Transformer, the same minimal inductive biases, work as well for language modelling as for image segmentation as for proof search—the only difference perhaps that ViT works on patches for efficiency, though the paper shows that probably hurts performance in the limit. All it takes is an appropriate quantity of data to learn the appropriate task-specific adaptations the network needs. Heck, even cross-domain works; it's all one architecture, so it's all one inductive bias. To my mind, this is what it means to learn from examples. There is no way that an architecture designed for language translation could also encode task-specific priors for these different tasks. For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example. ML people call that data leakage. > I don't understand what you mean. Yes, Louise can check for arbitrary programs. I can give it zero BK and metarules and ask it to generate all Prolog programs, say. Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs.
- deleted 6y ago[deleted]
- YeGoblynQueenne 6y ago