The training objective does not have much to do with it, the bigger reason is that these neural networks themselves use linear algebra (dot products/projections

The training objective does not have much to do with it, the bigger reason is that these neural networks themselves use linear algebra (dot products/projections and whatnot) to manipulate embeddings.