Modern neural networks make heavy use of linear algebra, in particular the transformer[1] architecture that powers modern LLMs. Since linear algebra is closely

Modern neural networks make heavy use of linear algebra, in particular the transformer[1] architecture that powers modern LLMs. Since linear algebra is closely related to geometry[2], it seems quite reasonable that there are some geometric aspects that define their capabilities and performance. Specifically, in this paper they're considering the intrinsic dimension[3] of the attention layers, and seeing how it correlates with the performance of LLMs. [1]: https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Scaled_dot-product_attention https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2]: https://en.wikipedia.org/wiki/Linear_algebra#Relationship_with_geometry https://en.wikipedia.org/wiki/Linear_algebra#Relationship_wi... [3]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension