4 ms·

"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a no

by srean 8y ago

"compression is the same thing as forecasting" This is true by definition regardless of manifold hypothesis. In order to define compression you dont need a nontrivial metric or a topological space. You need things to even talk about the manifold hypothesis, at least in any interesting way.

scottlocklin 8y ago
Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There are interesting problems where you can't do anything with metrics or quasi-metric distance measures, but you can forecast things. Geometry is easier to reason about than Kolmogorov complexity though, hence papers like the above.
- srean 8y ago
  Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?
  - scottlocklin 8y ago
    Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.
    - srean 8y ago
      Oh the post is a good one. What I was drawing attention to is that manifold hypothesis and the compression hypothesis are different concepts. The compression hypothesis is more general to the point of being a definition. It holds true regardless of whether the low dimensional manifold property holds or not.

by srean 8y ago

scottlocklin 8y ago

Well I'll say it a different way: the manifold hypothesis is basically a geometric restatement of the fact that compression is the same thing as forecasting. If you couldn't map a metric space (or other distance measure) onto a lower dimensional manifold, you're basically saying you can't forecast using a geometric representation, or you have already extracted all the signal from the noise in some other way. There are interesting problems where you can't do anything with metrics or quasi-metric distance measures, but you can forecast things. Geometry is easier to reason about than Kolmogorov complexity though, hence papers like the above.

srean 8y ago
Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology. One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?
- scottlocklin 8y ago
  Hey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.
  - srean 8y ago
    Oh the post is a good one. What I was drawing attention to is that manifold hypothesis and the compression hypothesis are different concepts. The compression hypothesis is more general to the point of being a definition. It holds true regardless of whether the low dimensional manifold property holds or not.