3 ms·
Not quite buying that yet. Why would one need a metric ? Let me give a concrete example, in compression and statistical estimation you can get by fine with KL
by srean 8y ago
Not quite buying that yet. Why would one need a metric ?
Let me give a concrete example, in compression and statistical estimation you can get by fine with KL divergence alone. KL divergence is not a metric and does not define a topology.
One can of course define metrics on the probability space or the space of parameters, but I dont see why that would be necessary ?
- scottlocklin 8y agoHey man, I didn't post the paper on neural nets, manifolds and topology; I'm not going to defend it. I already said you don't need a metric. But in the presence of an interesting metric or other distance measure; of course the signal fits on a lower dimensional manifold. Metric spaces and other distance measures give people a lot of tools to reason about data in general, hence virtually all common unsupervised learning algorithms and stuff like topological data analysis.
- srean 8y agoOh the post is a good one. What I was drawing attention to is that manifold hypothesis and the compression hypothesis are different concepts. The compression hypothesis is more general to the point of being a definition. It holds true regardless of whether the low dimensional manifold property holds or not.