8 ms·
> GPT's model architecture was invented at Google anyway No it wasn't. Transformers were invented at Google, but "architecture" when talking about neural netwo
by nl 4y ago
> GPT's model architecture was invented at Google anyway
No it wasn't. Transformers were invented at Google, but "architecture" when talking about neural networks means how they are arranged (and to some extent the training objective function) rather than the building blocks used.
- l33t233372 4y agoI’m not sure what you mean. The words “transformer architecture” are standard across the literature.
- nl 4y agoThat's like saying "LSTM architecture". It says it uses a transformer but no description of how it is used. For example, the GPT architecture (which GPT 2 & 3 are slight modifications of) comprises of an embedding layer followed by 12x(self-attention/layer norm/feed forward/layer norm. That's what that GPT "transformer architecture" is, not just the transformer block itself. Strictly the architecture really also includes things like the embedding size and number of heads (which are in the GPT paper).