That's fair, and I didn't realize people were using them for images that way. I would still argue that they are at least somewhat more specialized than plain fu

That's fair, and I didn't realize people were using them for images that way. I would still argue that they are at least somewhat more specialized than plain fully-connected layers, much like convolutional layers. It is definitely interesting that we can do so much with a relatively small number of generic "primitive" components in these big models. but I suppose that's part of the point.