5 ms·

My initial response was a fairly kneejerk reaction to the snark. The following is a rewrite. Please don't; if you really think so little of me, rather don't rep

My initial response was a fairly kneejerk reaction to the snark. The following is a rewrite. Please don't; if you really think so little of me, rather don't reply than reply unpleasantly. > Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures “Solomonoff Induction does not learn only from evidence! It encodes strong inductive biases in its construction and choice of Turing machine...” but it doesn't matter. Our universe is not a random soup of maximal entropy. The tasks I am talking about solving are overtly not impossible. You talk about ML methods like the success of, say, image recognition comes from image-recognition-specific architectures. You mention ‘hand-engineered’ or ‘hand-tuned’. And yet, to throw your snark back at you, if you were up to date with the literature, you would know this is not true. Consider ViT as an example. The same Transformer, the same minimal inductive biases, work as well for language modelling as for image segmentation as for proof search—the only difference perhaps that ViT works on patches for efficiency, though the paper shows that probably hurts performance in the limit. All it takes is an appropriate quantity of data to learn the appropriate task-specific adaptations the network needs. Heck, even cross-domain works; it's all one architecture, so it's all one inductive bias. To my mind, this is what it means to learn from examples. There is no way that an architecture designed for language translation could also encode task-specific priors for these different tasks. For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example. ML people call that data leakage. > I don't understand what you mean. Yes, Louise can check for arbitrary programs. I can give it zero BK and metarules and ask it to generate all Prolog programs, say. Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs.

deleted 6y ago
[deleted]
YeGoblynQueenne 6y ago
I see I made you feel bad with my advice to read up. I'm sorry, because that was not my intention. However, you really do need to take my advise seriously. You've insisted throughout our conversation that you don't need to read older machine learning or AI papers because they're not relevant anymore. And yet, they are. And you do need to read them because without them you will not be able to understand the recent developments you seem to be intersted in. Take for instance your example of ViT. This is a transformer, so it's clearly not an unbiased generaliser that learns only from examples. You say so yourself: "it's all one inductive bias". Yes, that's how machine learning works and deep neural nets don't do anything different, neither do they learn only from examples, as you seemed to suggest in your previous comment (you replied "That's literally what DL is" to my comment that "you can't learn only from examples"). But I think you misunderstood my comment about how the biggest advances in deep neural nets have come from purpose-built architectures. That is not to say that the same architectures cannot be applied to different domains- but the state of the art systems are always fine-tuned for specific tasks or datasets. This hasn't changed recently and it hasn't changed in the last 30 years. >> For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example. A truly bias-free algorithm is not "random bytes". It's a learner that memorises its training examples and can only recognise its training examples. Hence why it can't generalise. This is in Mitchell's paper which I suggested you read. Louise's biases are not problem-specific in the short example I showed you. I defined BK predicates with wide applicability in programs processing lists and numbers. There is no such limitation, theoretical or practical, in the general sense, either. You can give Louise a million irrelevant BK predicates, if you like, and it will still find the ones it needs to complete the learning task assuming they're in there somewhere. In fact, it will find all of the relevant ones - and return the superset of all programs that solve the task (so you can use it for example to identify interesting relations in your dataset). Like I say in a previous comment, Louise's learning algorithm was originally designed to select relevant BK. Additionally, like I said in an earlier comment, Louise can learn its own bias, both the BK and the metarules, so it is not only not limited to task-specific bias, it is not even limited to user-provided bias. Under some circumstances it can even invent new examples. And then use them to learn a hypothesis that generalises better to unseen examples. * >> Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs. I don't follow. What do you mean?

deleted 6y ago

[deleted]

YeGoblynQueenne 6y ago

I see I made you feel bad with my advice to read up. I'm sorry, because that was not my intention. However, you really do need to take my advise seriously. You've insisted throughout our conversation that you don't need to read older machine learning or AI papers because they're not relevant anymore. And yet, they are. And you do need to read them because without them you will not be able to understand the recent developments you seem to be intersted in. Take for instance your example of ViT. This is a transformer, so it's clearly not an unbiased generaliser that learns only from examples. You say so yourself: "it's all one inductive bias". Yes, that's how machine learning works and deep neural nets don't do anything different, neither do they learn only from examples, as you seemed to suggest in your previous comment (you replied "That's literally what DL is" to my comment that "you can't learn only from examples"). But I think you misunderstood my comment about how the biggest advances in deep neural nets have come from purpose-built architectures. That is not to say that the same architectures cannot be applied to different domains- but the state of the art systems are always fine-tuned for specific tasks or datasets. This hasn't changed recently and it hasn't changed in the last 30 years. >> For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example. A truly bias-free algorithm is not "random bytes". It's a learner that memorises its training examples and can only recognise its training examples. Hence why it can't generalise. This is in Mitchell's paper which I suggested you read. Louise's biases are not problem-specific in the short example I showed you. I defined BK predicates with wide applicability in programs processing lists and numbers. There is no such limitation, theoretical or practical, in the general sense, either. You can give Louise a million irrelevant BK predicates, if you like, and it will still find the ones it needs to complete the learning task assuming they're in there somewhere. In fact, it will find all of the relevant ones - and return the superset of all programs that solve the task (so you can use it for example to identify interesting relations in your dataset). Like I say in a previous comment, Louise's learning algorithm was originally designed to select relevant BK. Additionally, like I said in an earlier comment, Louise can learn its own bias, both the BK and the metarules, so it is not only not limited to task-specific bias, it is not even limited to user-provided bias. Under some circumstances it can even invent new examples. And then use them to learn a hypothesis that generalises better to unseen examples. * >> Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs. I don't follow. What do you mean?