3 ms·
preserving topology during dimension reduction might affect this? something something, fractal dimensionality, erm tropical geometry and amoebas and the, here i
by tudorw 3y ago
preserving topology during dimension reduction might affect this? something something, fractal dimensionality, erm tropical geometry and amoebas and the, here it is, https://proceedings.mlr.press/v80/zhang18i.html https://proceedings.mlr.press/v80/zhang18i.html
Edit, obviously I don't know my zonotope from my tropical hypersurface, I do however like the pretty pictures ;)
Here's another good piece of how topology might help work out why some models do better than others; https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011792 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
"By computing geometric descriptors of DNNs and performing large-scale model comparisons, we discovered a geometric phenomenon that has been overlooked in previous work: DNN models of high-level visual cortex benefit from high-dimensional latent representations. This finding runs counter to the view that both DNNs and neural systems benefit by compressing representations down to low-dimensional subspaces [20–39, 78]. "
- benreesman 3y agohttps://en.wikipedia.org/wiki/Parallel_transport https://en.wikipedia.org/wiki/Parallel_transport ?
- tudorw 3y agoMaybe! I'm lost in the maths, "Our results suggest that learned optimizers can benefit from considering the (symmetry) structure of the weight space they optimize. " this from 7th Feb came out of DeepMind; https://arxiv.org/abs/2402.05232 https://arxiv.org/abs/2402.05232 When it comes to the math underneath an LLM, https://medium.com/autonomous-agents/part-8-mathematical-explanation-of-why-its-hard-for-llms-to-memorize-86ee1d58eeb9 https://medium.com/autonomous-agents/part-8-mathematical-exp... is about the most accessible explanation I have found so far.