4 ms·
Graph attention networks were inspired by transformers. I am pretty sure they explicitly say it in the paper.
by justlexi93 7y ago
Graph attention networks were inspired by transformers. I am pretty sure they explicitly say it in the paper.
- sqrt17 7y agoMaybe a better way to put it would be "Graph Neural Networks are a Generalization of Transformers". The formulation of "X are Y" indeed suggests that you could just have stuck with Y instead of using the special case X, whereas in reality they took Y and added something to make it more general.
- huffmsa 7y agoYeah but even without attention, they're pretty similar. Sentences can be reasonable modeled as a mostly 1 directional graph.
- chaitjo 7y agoIndeed, the two papers came out within months of each other iirc. The GAT paper discusses Transformers in the context of stabilizing the learning of attention mechanisms. Of course, this connection may be trivial to most people, but I hadn't seen a post on this before. So I decided to write one for myself as I studied these architectures.