4 ms·
If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry, with a metric. Just as in real life, we want to be
by esafak 1y ago
If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry, with a metric. Just as in real life, we want to be able to compare things.
Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement. Here are some relevant papers:
* Topology and geometry of data manifold in deep learning (https://arxiv.org/abs/2204.08624 https://arxiv.org/abs/2204.08624)
* Topology of Deep Neural Networks (https://jmlr.org/papers/v21/20-345.html https://jmlr.org/papers/v21/20-345.html)
* Persistent Topological Features in Large Language Models (https://arxiv.org/abs/2410.11042 https://arxiv.org/abs/2410.11042)
* Deep learning as Ricci flow (https://www.nature.com/articles/s41598-024-74045-9 https://www.nature.com/articles/s41598-024-74045-9)
- profchemai 1y agoAgree, if anything it's Applied Linear Algebra...but that sounds less exotic.
- lostmsu 1y agoWell, we know it is non-linear. More like differential equations.
- theahura 1y ago> Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? If you've ever played with GANs or VAEs, you can actually answer this question! And the answer is more or less 'yes'. You can look at GANs at various checkpoints during training and see how different points in the high dimensional space move around (using tools like UMAP / TSNE). > I imagine it violently changing at first before stabilizing, followed by geometric refinement Also correct, though the violent changing at the beginning is also influenced the learning rate and the choice of optimizer.
- esafak 1y agoAnd crucially, the initialization algorithm.