3 ms·

In ML, they might say you were overfitting. Predicting is all well and good, but predicting too well can indicate the machine hasn't really learned anything. It

by ggggtez 10y ago

In ML, they might say you were overfitting. Predicting is all well and good, but predicting too well can indicate the machine hasn't really learned anything. It's just spitting back nearly identical information as the original. It sounds like that is what this black box is supposed to do. All you need for a perfect black box is every possible data point...

sgt101 10y ago
So we have instance based learning which basically is about approximating this, and support vector machines which abstract the idea into representing the space of decisions represented by all the data points via a kernel which is a mapping function. The problem I see which deep learning seems to be "winning on" is that the space of instances that you have does not either abstract a meaningful theory of the distribution of these points (therefor predicting outside of this space) or describe things exterior to that space - but that exist, or might exist. A scientific theory is considered valid if it predicts things that have not been seen yet (and you can then test that, hence the post today about "string theory : enough"), I think IBL and SVMs can't represent the unseen, I used to work using something called inductive logic programming (I used a tool called Progol that Steven Muggleton made, it was good!) and I thought that that did produce such insights, sometimes, but it turned out that often these were actually me due to fiddling. Some play projects I did recently made me think that conv. nets were doing the same sort of things, but it was /is harder for me to catch and show this (and as I say, I ended up thinking that I'd often fooled myself with ILP, even though it was very useful). The old skool difficulty I have with deep nets is that I used to think in terms of structural risk minimization vs empirical risk minimisation when dealing with overfitting. The idea was that if you had the right size of information store in your learned system and it was experimentally producing results that showed it was an effective predictor you could say that it was generalizing properly. Deep nets seem to me to have all the information storing capability of the domains they address and I worry.... But I am shocked, shocked, by how well they seem to work.

by ggggtez 10y ago

sgt101 10y ago

So we have instance based learning which basically is about approximating this, and support vector machines which abstract the idea into representing the space of decisions represented by all the data points via a kernel which is a mapping function. The problem I see which deep learning seems to be "winning on" is that the space of instances that you have does not either abstract a meaningful theory of the distribution of these points (therefor predicting outside of this space) or describe things exterior to that space - but that exist, or might exist. A scientific theory is considered valid if it predicts things that have not been seen yet (and you can then test that, hence the post today about "string theory : enough"), I think IBL and SVMs can't represent the unseen, I used to work using something called inductive logic programming (I used a tool called Progol that Steven Muggleton made, it was good!) and I thought that that did produce such insights, sometimes, but it turned out that often these were actually me due to fiddling. Some play projects I did recently made me think that conv. nets were doing the same sort of things, but it was /is harder for me to catch and show this (and as I say, I ended up thinking that I'd often fooled myself with ILP, even though it was very useful). The old skool difficulty I have with deep nets is that I used to think in terms of structural risk minimization vs empirical risk minimisation when dealing with overfitting. The idea was that if you had the right size of information store in your learned system and it was experimentally producing results that showed it was an effective predictor you could say that it was generalizing properly. Deep nets seem to me to have all the information storing capability of the domains they address and I worry.... But I am shocked, shocked, by how well they seem to work.