4 ms·

It's kind of LHC-like, isn't it? Same physics "but larger".. "will it work if we scale it up? Apparently: yes." I think the big point, and what scares me a bit

It's kind of LHC-like, isn't it? Same physics "but larger".. "will it work if we scale it up? Apparently: yes." I think the big point, and what scares me a bit, is that we have yet to discover any sort of fundamental conceptual limit to the Transformer architecture.

sooheon 6y ago
Each attention block in the Transformer models a fully connected graph (with the attention heads being learned edge attributes). A graph is the most general data structure possible, so yeah, I don't think there's really a fundamental limitation to them, just a computational one. Latest papers from ICLR explore how they fully model CNNs and RNNs, for example, and I'm sure papers on their theoretical equivalence with GNNs are coming.
- contravariant 6y ago
  > I'm sure papers on their theoretical equivalence with GNNs are coming. Not exactly a paper, but you mean something like this article: https://graphdeeplearning.github.io/post/transformers-are-gnns/ https://graphdeeplearning.github.io/post/transformers-are-gn... ?
  - sooheon 6y ago
    Pretty much that, yeah. Thanks for the link. That kind of article sets the intuition, and papers like these give it a better theoretical treatment: - https://openreview.net/pdf?id=HJlnC1rKPB https://openreview.net/pdf?id=HJlnC1rKPB - https://openreview.net/pdf?id=ByxRM0Ntvr https://openreview.net/pdf?id=ByxRM0Ntvr
- p1esk 6y ago
  I'm not sure we want to work with "the most general data structure possible". We can use a Hopfield-like network where every neuron is connected to every other neuron and to itself. It probably won't be very useful. NN design have been moving from more general to less general architectures.
  - Erlich_Bachman 6y ago
    It is implied that at least in theory, the promise is to use them with the similar total complexity (like numberr of parameters and amount of required calculation), in which case yes we do want the most general data structure possible. If we can have a more general data structure that provides similar performance characteristics, it is easier to apply, to debug, to understand, and it likely means that we have found something more fundamental about the underlying world in general.
  - sooheon 6y ago
    I think you're conflating "generality" of NN architecture and "generality" of the inductive bias the NN models.
krick 6y ago
Only very remotely "kind of". LHC isn't mindless "will it work if we scale it up?" type of experiment at all. The idea behind it is very simple: there are fundamental properties we know we are after, we have some predictions about how particles behave, and we know we want to smash the particles together hard enough to verify some of these predictions. We know how to accelerate charged particles, so here we go, now we need powerful particle accelerator. Building or even running LHC isn't the experiment by itself, it's just an "easy" way to find out how theoretically predicted particle states look in the real life (hopefully disproving some possible theories). Many of ML experiments, though, are exactly what you said: we ultimately have no slightest idea, what we are doing, but maybe if we throw enough compute-power into gradient-descent over some generic enough network, it will learn to do something interesting. Well, I tell you what, theoretically you need only 2 wide layers of neurons to compute anything there is to compute, there just isn't enough computational power in the entire world to possibly come up with the "correct" neuron weights entirely by chance, so what the research mostly really is about is coming up with various tricks to make do with less computational resources, which is essentially making sense of some computational problems and making neural networks "less random", to direct the learning process. It is either about new neural architectures, new ways of posing questions for NN to answer, ways to make more training data out of the same amounts of available external data, or the alternatives for the gradient-descent learning approach altogether. So, to wrap it up. It may be kinda interesting to know that GPT-2 type of network didn't reach its full capacity, and if we scale it up even more, it still learns something new. Inspecting how it behaves also might lead to some insights or new possible applications. But ultimately, if there's no novelty of any kind in an experiment, training an NN nobody can realistically reproduce is a great way to show off (or to achieve some practical goal for the owner of this NN, if it was used to find something practical), but doesn't really contribute much (or anything) to ML research.
- FeepingCreature 6y ago
  "Yes, but." 2-wide layers can compute any computable function, but they do not scale in any manageable fashion with problem complexity. (Because you have to model problems as increasingly large lookup tables, whose size explodes rapidly.) The big thing about GPT is that so far, improvement in GPT size has corresponded to a .. I think logarithmic?- improvement in both quality of response and complexity of the problem domain that GPT can model. GPT scales, in exactly the way that a 2-layer network doesn't, and we don't yet have a handle on at what size it stops scaling.

sooheon 6y ago

Each attention block in the Transformer models a fully connected graph (with the attention heads being learned edge attributes). A graph is the most general data structure possible, so yeah, I don't think there's really a fundamental limitation to them, just a computational one. Latest papers from ICLR explore how they fully model CNNs and RNNs, for example, and I'm sure papers on their theoretical equivalence with GNNs are coming.

contravariant 6y ago
> I'm sure papers on their theoretical equivalence with GNNs are coming. Not exactly a paper, but you mean something like this article: https://graphdeeplearning.github.io/post/transformers-are-gnns/ https://graphdeeplearning.github.io/post/transformers-are-gn... ?
- sooheon 6y ago
  Pretty much that, yeah. Thanks for the link. That kind of article sets the intuition, and papers like these give it a better theoretical treatment: - https://openreview.net/pdf?id=HJlnC1rKPB https://openreview.net/pdf?id=HJlnC1rKPB - https://openreview.net/pdf?id=ByxRM0Ntvr https://openreview.net/pdf?id=ByxRM0Ntvr
p1esk 6y ago
I'm not sure we want to work with "the most general data structure possible". We can use a Hopfield-like network where every neuron is connected to every other neuron and to itself. It probably won't be very useful. NN design have been moving from more general to less general architectures.
- Erlich_Bachman 6y ago
  It is implied that at least in theory, the promise is to use them with the similar total complexity (like numberr of parameters and amount of required calculation), in which case yes we do want the most general data structure possible. If we can have a more general data structure that provides similar performance characteristics, it is easier to apply, to debug, to understand, and it likely means that we have found something more fundamental about the underlying world in general.
- sooheon 6y ago
  I think you're conflating "generality" of NN architecture and "generality" of the inductive bias the NN models.

krick 6y ago

Only very remotely "kind of". LHC isn't mindless "will it work if we scale it up?" type of experiment at all. The idea behind it is very simple: there are fundamental properties we know we are after, we have some predictions about how particles behave, and we know we want to smash the particles together hard enough to verify some of these predictions. We know how to accelerate charged particles, so here we go, now we need powerful particle accelerator. Building or even running LHC isn't the experiment by itself, it's just an "easy" way to find out how theoretically predicted particle states look in the real life (hopefully disproving some possible theories). Many of ML experiments, though, are exactly what you said: we ultimately have no slightest idea, what we are doing, but maybe if we throw enough compute-power into gradient-descent over some generic enough network, it will learn to do something interesting. Well, I tell you what, theoretically you need only 2 wide layers of neurons to compute anything there is to compute, there just isn't enough computational power in the entire world to possibly come up with the "correct" neuron weights entirely by chance, so what the research mostly really is about is coming up with various tricks to make do with less computational resources, which is essentially making sense of some computational problems and making neural networks "less random", to direct the learning process. It is either about new neural architectures, new ways of posing questions for NN to answer, ways to make more training data out of the same amounts of available external data, or the alternatives for the gradient-descent learning approach altogether. So, to wrap it up. It may be kinda interesting to know that GPT-2 type of network didn't reach its full capacity, and if we scale it up even more, it still learns something new. Inspecting how it behaves also might lead to some insights or new possible applications. But ultimately, if there's no novelty of any kind in an experiment, training an NN nobody can realistically reproduce is a great way to show off (or to achieve some practical goal for the owner of this NN, if it was used to find something practical), but doesn't really contribute much (or anything) to ML research.

FeepingCreature 6y ago
"Yes, but." 2-wide layers can compute any computable function, but they do not scale in any manageable fashion with problem complexity. (Because you have to model problems as increasingly large lookup tables, whose size explodes rapidly.) The big thing about GPT is that so far, improvement in GPT size has corresponded to a .. I think logarithmic?- improvement in both quality of response and complexity of the problem domain that GPT can model. GPT scales, in exactly the way that a 2-layer network doesn't, and we don't yet have a handle on at what size it stops scaling.