3 ms·

Each attention block in the Transformer models a fully connected graph (with the attention heads being learned edge attributes). A graph is the most general dat

Each attention block in the Transformer models a fully connected graph (with the attention heads being learned edge attributes). A graph is the most general data structure possible, so yeah, I don't think there's really a fundamental limitation to them, just a computational one. Latest papers from ICLR explore how they fully model CNNs and RNNs, for example, and I'm sure papers on their theoretical equivalence with GNNs are coming.

contravariant 6y ago
> I'm sure papers on their theoretical equivalence with GNNs are coming. Not exactly a paper, but you mean something like this article: https://graphdeeplearning.github.io/post/transformers-are-gnns/ https://graphdeeplearning.github.io/post/transformers-are-gn... ?
- sooheon 6y ago
  Pretty much that, yeah. Thanks for the link. That kind of article sets the intuition, and papers like these give it a better theoretical treatment: - https://openreview.net/pdf?id=HJlnC1rKPB https://openreview.net/pdf?id=HJlnC1rKPB - https://openreview.net/pdf?id=ByxRM0Ntvr https://openreview.net/pdf?id=ByxRM0Ntvr
p1esk 6y ago
I'm not sure we want to work with "the most general data structure possible". We can use a Hopfield-like network where every neuron is connected to every other neuron and to itself. It probably won't be very useful. NN design have been moving from more general to less general architectures.
- Erlich_Bachman 6y ago
  It is implied that at least in theory, the promise is to use them with the similar total complexity (like numberr of parameters and amount of required calculation), in which case yes we do want the most general data structure possible. If we can have a more general data structure that provides similar performance characteristics, it is easier to apply, to debug, to understand, and it likely means that we have found something more fundamental about the underlying world in general.
- sooheon 6y ago
  I think you're conflating "generality" of NN architecture and "generality" of the inductive bias the NN models.

contravariant 6y ago

> I'm sure papers on their theoretical equivalence with GNNs are coming. Not exactly a paper, but you mean something like this article: https://graphdeeplearning.github.io/post/transformers-are-gnns/ https://graphdeeplearning.github.io/post/transformers-are-gn... ?

sooheon 6y ago
Pretty much that, yeah. Thanks for the link. That kind of article sets the intuition, and papers like these give it a better theoretical treatment: - https://openreview.net/pdf?id=HJlnC1rKPB https://openreview.net/pdf?id=HJlnC1rKPB - https://openreview.net/pdf?id=ByxRM0Ntvr https://openreview.net/pdf?id=ByxRM0Ntvr

p1esk 6y ago

I'm not sure we want to work with "the most general data structure possible". We can use a Hopfield-like network where every neuron is connected to every other neuron and to itself. It probably won't be very useful. NN design have been moving from more general to less general architectures.

Erlich_Bachman 6y ago
It is implied that at least in theory, the promise is to use them with the similar total complexity (like numberr of parameters and amount of required calculation), in which case yes we do want the most general data structure possible. If we can have a more general data structure that provides similar performance characteristics, it is easier to apply, to debug, to understand, and it likely means that we have found something more fundamental about the underlying world in general.
sooheon 6y ago
I think you're conflating "generality" of NN architecture and "generality" of the inductive bias the NN models.