3 ms·

Graph Language Models

123 points by Anon84 2y ago

fisian 2y ago
In case you haven't heard about Graph NNs yet, pyyorch geometric is a library to build GNNs and has some info how they work [1]. [1]: https://pytorch-geometric.readthedocs.io/ https://pytorch-geometric.readthedocs.io/
kelsey98765431 2y ago
Graphing is not only the future, it's the present that is happening behind closed doors that you don't get to hear about. The things that can be done already are insane and the capabilities are growing quickly. Agentic graph based systems are the future for ASI.
- freilanzer 2y ago
  And how do you hear about the things happening behind closed doors? Source: just trust me.
  - exe34 2y ago
    people don't trust the official spokesperson but will immediately trust "an anonymous source".
- awestroke 2y ago
  The future is looking bright indeed for Addiction Severity Index
- esafak 2y ago
  https://thegradient.pub/transformers-are-graph-neural-networks/ https://thegradient.pub/transformers-are-graph-neural-networ...
- OutOfHere 2y ago
  Anyone using the word "agentic" has jumped on the loony train. These people use it to discriminate against AIs that they personally dislike, claiming it's not sufficiently "agentic". In reality there is a wide spectrum of how a system uses AI, and there is no hard line separating agentic from non-agentic.
  - kordlessagain 2y ago
    I would argue that agentic means writing and running code at will, by the LLM. Eric Schmidt recently commented that he thought three things would bring profound change: (a) an infinite context window, (b) chain of thought reasoning in agents and (c) the text-to-action capacity for programming. In that view, agentic is mostly about (b) and (c). Having a way to chain together steps (which is just looping with the right context) and the ability to write and run code allows LLMs to take action on the things it is given to "do".
    - OutOfHere 2y ago
      > Having a way to chain together steps (which is just looping with the right context) and the ability to write and run code allows LLMs to take action on the things it is given to "do". What you describe is basically just a standard LLM workflow. It's not as if the LLM is evolving the workflow itself as it goes.
VHRanger 2y ago
I've talked about this before on my blog: https://singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-neural-networks/ https://singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-... Basically, language models are already graph neural networks. To understand why, go back to Word2Vec/GLoVe: word embedding distance represent co-occurrence frequency of words in a sentence. Note how this is the same as a graph embedding problem: words are nodes, and the edge weight is co-occurence frequency. You embed the graph nodes. In fact, this is stated in formal math in the GLoVe paper. The LLM architecture is basically doing the same thing, except the graph is conditional occurence based on the previous contextual words. This setup makes for a graph with a truly astronomic number of nodes (word|context) and edges. This huge graph exists only in the land of abstract math, but it also shows why LLMs require so many parameters to perform well. In any case, 4 years on, I'm still pretty lukewarm on the current gen of graph neural network architectures. Case in point: the OP paper is pretty much the classic ML paper mill setup of "take some existing algorithm, add some stuff over it, spend a ton hyperparameter searching on your algo and show it beats some 2 year old baseline".
- deleted 2y ago
  [deleted]
- zwaps 2y ago
  Like you said, language models have started "working" to write text since (and not before) they used attention, that is, adjacency matrices (being GNNs). GloVe and Word2Vec did well for aggregate word representations over an entire corpus, but those representations were not sufficient to generate text. In other words, they were an incomplete representation of language. Turns out, you need non-euclidean operations to actually represent language in sufficient detail to generate text. An embedding into a Euclidean space with a distance is a compression and this compression can be coded back into a graph which is by necessity symmetric. As such, it representative power for complex graphs is limited. That is why in a LLM, "embeddings" are not Euclidean. They are a combination of operations on adjacency matrices. This goes for any interesting graph: For a triplet of nodes, relations need not follow the triangle inequality. As such, embedding a triplet into a 2D space and applying a (e.g. cosine) distance will not work exactly. This generalizes to larger networks. For tree structures, you need a hyperbolic embedding. For non-tree graphs, there is no general euclidean node embedding that works, and you have to rely on high dimensionality of the embedding vector. This becomes icky when you have more difficult problems, such as embedding a graph onto a geometric structure, manifold, or more complex graph structures. LLM embeddings were inferior to even Word2Vec for 2-3 years even when the models were obviously very good at understanding language. The reason is simple: The models do not create any single number for a token that is associated with a single Euclidean space. There are, in fact, many such spaces within an LLM. Early BERT embeddings tried to find that single space by some linear combination of token activations and failed. Point being: Graph embeddings are important. There ARE reasons to embed a graph, since graphs can become huge (recommender systems, social graphs, material, 3D modeling, even certain knowledge graph structures). The moving parts are both the underlying data, and the desired space in which to embed. In many cases not all paths lengths need to be considered and a local factorization is sufficient is another matter. But that's still a "GNN". We can show that transformer like GNNs or factorizations are also not sufficient to represent any graph structure. Yet, any modern LLM is literally a GNN and that's just for text. Now if you add a huge knowledge or process graph on top of that, it stands to reason that you might profit from considering how to represent this additional layer of relationships between text tokens.
intalentive 2y ago
Yes transformers are a class of graph NN but there can still be value in dealing directly with both text and knowledge graphs (for RAG-style applications, presumably) because the graph enforces additional constraints. It is like the difference between concrete and abstract syntax. LLMs frequently generate code that won't compile, since they predict tokens not AST nodes. They are underconstrained for the task. How to address? You can train a single model to handle both, as the authors did, or you can manually enforce constraints while decoding.

123 points by Anon84 2y ago

In case you haven't heard about Graph NNs yet, pyyorch geometric is a library to build GNNs and has some info how they work [1]. [1]: https://pytorch-geometric.readthedocs.io/ https://pytorch-geometric.readthedocs.io/

kelsey98765431 2y ago

Graphing is not only the future, it's the present that is happening behind closed doors that you don't get to hear about. The things that can be done already are insane and the capabilities are growing quickly. Agentic graph based systems are the future for ASI.

freilanzer 2y ago
And how do you hear about the things happening behind closed doors? Source: just trust me.
- exe34 2y ago
  people don't trust the official spokesperson but will immediately trust "an anonymous source".
awestroke 2y ago
The future is looking bright indeed for Addiction Severity Index
esafak 2y ago
https://thegradient.pub/transformers-are-graph-neural-networks/ https://thegradient.pub/transformers-are-graph-neural-networ...
OutOfHere 2y ago
Anyone using the word "agentic" has jumped on the loony train. These people use it to discriminate against AIs that they personally dislike, claiming it's not sufficiently "agentic". In reality there is a wide spectrum of how a system uses AI, and there is no hard line separating agentic from non-agentic.
- kordlessagain 2y ago
  I would argue that agentic means writing and running code at will, by the LLM. Eric Schmidt recently commented that he thought three things would bring profound change: (a) an infinite context window, (b) chain of thought reasoning in agents and (c) the text-to-action capacity for programming. In that view, agentic is mostly about (b) and (c). Having a way to chain together steps (which is just looping with the right context) and the ability to write and run code allows LLMs to take action on the things it is given to "do".
  - OutOfHere 2y ago
    > Having a way to chain together steps (which is just looping with the right context) and the ability to write and run code allows LLMs to take action on the things it is given to "do". What you describe is basically just a standard LLM workflow. It's not as if the LLM is evolving the workflow itself as it goes.

VHRanger 2y ago

I've talked about this before on my blog: https://singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-neural-networks/ https://singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-... Basically, language models are already graph neural networks. To understand why, go back to Word2Vec/GLoVe: word embedding distance represent co-occurrence frequency of words in a sentence. Note how this is the same as a graph embedding problem: words are nodes, and the edge weight is co-occurence frequency. You embed the graph nodes. In fact, this is stated in formal math in the GLoVe paper. The LLM architecture is basically doing the same thing, except the graph is conditional occurence based on the previous contextual words. This setup makes for a graph with a truly astronomic number of nodes (word|context) and edges. This huge graph exists only in the land of abstract math, but it also shows why LLMs require so many parameters to perform well. In any case, 4 years on, I'm still pretty lukewarm on the current gen of graph neural network architectures. Case in point: the OP paper is pretty much the classic ML paper mill setup of "take some existing algorithm, add some stuff over it, spend a ton hyperparameter searching on your algo and show it beats some 2 year old baseline".

deleted 2y ago
[deleted]
zwaps 2y ago
Like you said, language models have started "working" to write text since (and not before) they used attention, that is, adjacency matrices (being GNNs). GloVe and Word2Vec did well for aggregate word representations over an entire corpus, but those representations were not sufficient to generate text. In other words, they were an incomplete representation of language. Turns out, you need non-euclidean operations to actually represent language in sufficient detail to generate text. An embedding into a Euclidean space with a distance is a compression and this compression can be coded back into a graph which is by necessity symmetric. As such, it representative power for complex graphs is limited. That is why in a LLM, "embeddings" are not Euclidean. They are a combination of operations on adjacency matrices. This goes for any interesting graph: For a triplet of nodes, relations need not follow the triangle inequality. As such, embedding a triplet into a 2D space and applying a (e.g. cosine) distance will not work exactly. This generalizes to larger networks. For tree structures, you need a hyperbolic embedding. For non-tree graphs, there is no general euclidean node embedding that works, and you have to rely on high dimensionality of the embedding vector. This becomes icky when you have more difficult problems, such as embedding a graph onto a geometric structure, manifold, or more complex graph structures. LLM embeddings were inferior to even Word2Vec for 2-3 years even when the models were obviously very good at understanding language. The reason is simple: The models do not create any single number for a token that is associated with a single Euclidean space. There are, in fact, many such spaces within an LLM. Early BERT embeddings tried to find that single space by some linear combination of token activations and failed. Point being: Graph embeddings are important. There ARE reasons to embed a graph, since graphs can become huge (recommender systems, social graphs, material, 3D modeling, even certain knowledge graph structures). The moving parts are both the underlying data, and the desired space in which to embed. In many cases not all paths lengths need to be considered and a local factorization is sufficient is another matter. But that's still a "GNN". We can show that transformer like GNNs or factorizations are also not sufficient to represent any graph structure. Yet, any modern LLM is literally a GNN and that's just for text. Now if you add a huge knowledge or process graph on top of that, it stands to reason that you might profit from considering how to represent this additional layer of relationships between text tokens.

intalentive 2y ago

Yes transformers are a class of graph NN but there can still be value in dealing directly with both text and knowledge graphs (for RAG-style applications, presumably) because the graph enforces additional constraints. It is like the difference between concrete and abstract syntax. LLMs frequently generate code that won't compile, since they predict tokens not AST nodes. They are underconstrained for the task. How to address? You can train a single model to handle both, as the authors did, or you can manually enforce constraints while decoding.