3 ms·

I liked this article a lot, but > One thought that occurred to me after this investigation was the premise that the immense bleeding-edge models of today with

I liked this article a lot, but > One thought that occurred to me after this investigation was the premise that the immense bleeding-edge models of today with billions of parameters might be able to be built using orders of magnitude fewer network resources by using more efficient or custom-designed architectures. Transformer units themselves are already specialized things. Wikipedia says that GPT-3 is a standard transformer network, so I'm sure there is additional room for specialization. But that's not a new idea either, and it's often the case that a after a model is released, smaller versions tend to follow.

sebzim4500 4y ago
Transformers are specialized to run on our hardware, whereas the article is suggesting architectures which are specialized for a specific task.
- nerdponx 4y ago
  Are transformers not already very specialized to the task of learning from sequences of word vectors? I'm sure there is more that can be done with them other than making the input sequences really long, but my point was that LLMs are hardly lacking in design specialized to their purpose.
  - sebzim4500 4y ago
    > Are transformers not already very specialized to the task of learning from sequences of word vectors? No, you can use transformers for vision, image generation, audio generation/recognition, etc. They are 'specialized' in that they are for working with sequences of data, but almost everything can be nicely encoded as a sequence. In order to input images, for example, you typically split the image into blocks and then use a CNN to produce a token for each block. Then you concatenate the tokens and feed them into a transformer.
    - nerdponx 4y ago
      That's fair, and I didn't realize people were using them for images that way. I would still argue that they are at least somewhat more specialized than plain fully-connected layers, much like convolutional layers. It is definitely interesting that we can do so much with a relatively small number of generic "primitive" components in these big models. but I suppose that's part of the point.
deleted 4y ago
[deleted]

sebzim4500 4y ago

Transformers are specialized to run on our hardware, whereas the article is suggesting architectures which are specialized for a specific task.

nerdponx 4y ago
Are transformers not already very specialized to the task of learning from sequences of word vectors? I'm sure there is more that can be done with them other than making the input sequences really long, but my point was that LLMs are hardly lacking in design specialized to their purpose.
- sebzim4500 4y ago
  > Are transformers not already very specialized to the task of learning from sequences of word vectors? No, you can use transformers for vision, image generation, audio generation/recognition, etc. They are 'specialized' in that they are for working with sequences of data, but almost everything can be nicely encoded as a sequence. In order to input images, for example, you typically split the image into blocks and then use a CNN to produce a token for each block. Then you concatenate the tokens and feed them into a transformer.
  - nerdponx 4y ago
    That's fair, and I didn't realize people were using them for images that way. I would still argue that they are at least somewhat more specialized than plain fully-connected layers, much like convolutional layers. It is definitely interesting that we can do so much with a relatively small number of generic "primitive" components in these big models. but I suppose that's part of the point.

deleted 4y ago

[deleted]