Like you said, language models have started "working" to write text since (and not before) they used attention, that is, adjacency matrices (being GNNs). GloVe

Like you said, language models have started "working" to write text since (and not before) they used attention, that is, adjacency matrices (being GNNs). GloVe and Word2Vec did well for aggregate word representations over an entire corpus, but those representations were not sufficient to generate text. In other words, they were an incomplete representation of language. Turns out, you need non-euclidean operations to actually represent language in sufficient detail to generate text. An embedding into a Euclidean space with a distance is a compression and this compression can be coded back into a graph which is by necessity symmetric. As such, it representative power for complex graphs is limited. That is why in a LLM, "embeddings" are not Euclidean. They are a combination of operations on adjacency matrices. This goes for any interesting graph: For a triplet of nodes, relations need not follow the triangle inequality. As such, embedding a triplet into a 2D space and applying a (e.g. cosine) distance will not work exactly. This generalizes to larger networks. For tree structures, you need a hyperbolic embedding. For non-tree graphs, there is no general euclidean node embedding that works, and you have to rely on high dimensionality of the embedding vector. This becomes icky when you have more difficult problems, such as embedding a graph onto a geometric structure, manifold, or more complex graph structures. LLM embeddings were inferior to even Word2Vec for 2-3 years even when the models were obviously very good at understanding language. The reason is simple: The models do not create any single number for a token that is associated with a single Euclidean space. There are, in fact, many such spaces within an LLM. Early BERT embeddings tried to find that single space by some linear combination of token activations and failed. Point being: Graph embeddings are important. There ARE reasons to embed a graph, since graphs can become huge (recommender systems, social graphs, material, 3D modeling, even certain knowledge graph structures). The moving parts are both the underlying data, and the desired space in which to embed. In many cases not all paths lengths need to be considered and a local factorization is sufficient is another matter. But that's still a "GNN". We can show that transformer like GNNs or factorizations are also not sufficient to represent any graph structure. Yet, any modern LLM is literally a GNN and that's just for text. Now if you add a huge knowledge or process graph on top of that, it stands to reason that you might profit from considering how to represent this additional layer of relationships between text tokens.