4 ms·

> Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible This is interestin

> Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.

jandrewrogers 10y ago
The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.
eli_gottlieb 10y ago
>I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning. >We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information. Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.

jandrewrogers 10y ago

The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.

eli_gottlieb 10y ago

>I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning. >We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information. Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.