4 ms·

Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible - but if small data c

Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible - but if small data cannot give it confidence to confirm the existence of some subtile pattern, on what grounds is even an ideal learner supposed to believe in it? For example, no smarts would ever confirm the Higgs in the tevatron in its 10 year run even though the machine produced thousands of Higges; its a subtle signal and you just don't have the sample size to be more than 3 sigma sure there's something there (nor where exactly). What is key, however, is being able to leverage more than just explicit training sets, like with transfer learning and unsupervised learning, as well as making algorithms scale well with more data. The latter seems promising with deep learning, the former still a research thing. "more data beats better algorithm" is obviously false when the algorithms don't scale with data as well, as deep learning has convincingly demonstrated, being just as mediocre as anything else on small datasets, but being leaps above everything else on sufficiently large ones.

Xcelerate 10y ago
> Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
- jandrewrogers 10y ago
  The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.
- eli_gottlieb 10y ago
  >I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning. >We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information. Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.

Xcelerate 10y ago

> Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.

jandrewrogers 10y ago
The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.
eli_gottlieb 10y ago
>I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning. >We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information. Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.