5 ms·

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff lik

It seems profound, but it's not really saying anything different than "compression is the same thing as forecasting." FWIIW unsupervised learning and stuff like topological data analysis is almost entirely about discovering the actual manifolds (or some hand wavey topology). Doesn't always work; the data often doesn't cooperate and live on a metric space.

throwawaymath 8y ago
That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially incompressible strings. This plays nicely with the recent (2018) Lei-Luo-Yau-Gu result that there exist manifolds which cannot be learned. As I harp on every chance I can get, I have a pet hypothesis that there's a very deep corollary here waiting to be proven rigorously. Namely, that we can show there exist adversarial inputs that exploit neural networks because they're incompressible. Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds).
- whatshisface 8y ago
  >Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds). I get the image of a technique that, when applied to humans, allows you to see through political speeches and reveals the eldritch horrors scurrying around us continually.
angel_j 8y ago
That's if you go about it from low-dimensional embedding to feature classification, and predictions based on probabilities for those features. As you mention, that works if you have metric data, or can juice metrics from abstraction. Modern ML is all about solving for this case; adding manifold hypothesis, as the author noted, may only improve them slightly, or not at all. That isn't what the "manifold hypothesis" is good for. Manifold space is more interesting to think of, not in the deconstruction of metric space, but in the formation of it; i.e. what higher-dimensional, non-metric (intensive), manifold-space, through some complex function (eg physics), results in this lower-dimensional, feature-rich entity; and what is that function (or more interestingly, how does it change the resulting entity-state, given various changes in the manifolds). Which may not be useful for classification, but that's old hat anyway. Personally, I feel that feature dependence for everything NN is brute force. How features come-to-be, even abstractly modeled, is the thing.
srean 8y ago
"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.
- scottlocklin 8y ago
  Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There are interesting problems where you can't do anything with metrics or quasi-metric distance measures, but you can forecast things. Geometry is easier to reason about than Kolmogorov complexity though, hence papers like the above.
  - srean 8y ago
    Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?
    - scottlocklin 8y ago
      Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.

throwawaymath 8y ago

That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially incompressible strings. This plays nicely with the recent (2018) Lei-Luo-Yau-Gu result that there exist manifolds which cannot be learned. As I harp on every chance I can get, I have a pet hypothesis that there's a very deep corollary here waiting to be proven rigorously. Namely, that we can show there exist adversarial inputs that exploit neural networks because they're incompressible. Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds).

whatshisface 8y ago
>Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds). I get the image of a technique that, when applied to humans, allows you to see through political speeches and reveals the eldritch horrors scurrying around us continually.

angel_j 8y ago

That's if you go about it from low-dimensional embedding to feature classification, and predictions based on probabilities for those features. As you mention, that works if you have metric data, or can juice metrics from abstraction. Modern ML is all about solving for this case; adding manifold hypothesis, as the author noted, may only improve them slightly, or not at all. That isn't what the "manifold hypothesis" is good for. Manifold space is more interesting to think of, not in the deconstruction of metric space, but in the formation of it; i.e. what higher-dimensional, non-metric (intensive), manifold-space, through some complex function (eg physics), results in this lower-dimensional, feature-rich entity; and what is that function (or more interestingly, how does it change the resulting entity-state, given various changes in the manifolds). Which may not be useful for classification, but that's old hat anyway. Personally, I feel that feature dependence for everything NN is brute force. How features come-to-be, even abstractly modeled, is the thing.

srean 8y ago

"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.

scottlocklin 8y ago
Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There are interesting problems where you can't do anything with metrics or quasi-metric distance measures, but you can forecast things. Geometry is easier to reason about than Kolmogorov complexity though, hence papers like the above.
- srean 8y ago
  Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?
  - scottlocklin 8y ago
    Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.