9 ms·
The convolution empire strikes back
- adamnemecek 3y agoAll machine learning is just convolution in the context of Hopf algebra convolution.
- mensetmanusman 3y agoIs this an intellectual leap aiming to make the field more cohesive, like the quest for unifying theories in physics?
- adamnemecek 3y agoThat is one of the goals, yes. In addition, it seems like you get neural architecture search (architecture is optimized), faster training, inference and interpretability. I'm working it out as we speak. Ironically, convolution provides some unification in physics too, e.g. renormalization is a convolution.
- dpflan 3y agoInteresting, please do elaborate...
- cwillu 3y agoI read it as a riff on “monads are just monoids in the category of endofunctors”, but maybe that wasn't intended.
- uoaei 3y agoIt kind of is. The commenter has been working on this formalism for a year or more. I'm sure he will come by with his link for the Discord channel where he discusses with and finds collaborators soon.
- adamnemecek 3y agoI has been less than 9 months. But yeah there is a discord if you want to follow progress https://discord.cofunctional.ai https://discord.cofunctional.ai.
- visarga 3y agoMy theory is that architecture doesn't matter - convolutional, transformer or recurrent, as long as you can efficiently train models of the same size, what counts is the dataset. Similarly, humans achieve about the same results when they have the same training. Small variations. What matters is not the brain but the education they get. Of course I am exaggerating a bit, just saying there are a multitude of architectures of brain and neural nets with similar abilities, and the differentiating factor is the data not the model. For years we have seen hundreds of papers trying to propose sub-quadratic attention. They all failed to get traction, big labs still use almost vanilla transformer. At some point a paper declared "mixing is all you need" (MLP-Mixers) to replace "attention is all you need". Just mixing, the optimiser adapts to what it gets. If you think about it, maybe language creates a virtual layer where language operations are performed. And this works similarly in humans and AIs. That's why the architecture doesn't matter, because it is running the language-OS on top. Similarly for vision. I place 90% the merits of AI on language and 10% on the model architecture. Finding intelligence was inevitable, it was hiding in language, that's how we get to be intelligent as well. A human raised without language is even worse than a primitive. Intelligence is encoded in software, not hardware. Our language software has more breadth and depth than any one of us can create or contain.
- kookamamie 3y agoDataset counts, but also the number of total parameters in the network, i.e. capacity.
- visarga 3y agoAgreed, it's in my first phrase "as long as you can efficiently train models of the same size, what counts is the dataset". But useful sizes are just a few. 7, 13, 35, 70, 120B - because they are targeted to various families of GPUs. A 2T model I can't run or too expensive to use on APIs is of no use. Not just dataset size, but data quality matters just as much, and diversity. I believe LLMs will train mostly on synthetic data engineered to have extreme diversity and very high quality. This kind of data confers 5x gains in efficiency as demonstrated by Microsoft in the Phi-1.5 paper.
- 3y ago
- gradascent 3y agoThis is great, but what is a possible use-case of these massive classifier models? I'm guessing they won't be running at the edge, which precludes them from real-time applications like self-driving cars, smartphones, or military. So then what? Facial recognition for police/governments or targeted advertisement based on your Instagram/Google photos? I'm genuinely curious.
- deleted 3y ago[deleted]
- currymj 3y ago1) it's basic research, 2) you can always chop off the last layer and use the embeddings, which I guess might be useful for something
- pjs_ 3y agohttps://external-preview.redd.it/du7KQXLvBmVqc5G0T3tIEbWsYn8-qvtKTaMaZi7WaQ0.png?width=960&crop=smart&auto=webp&s=fc212d5b696c4d8ad3e7bdeae271b92255c29ee4 https://external-preview.redd.it/du7KQXLvBmVqc5G0T3tIEbWsYn8...
- dontreact 3y agoThis is nice because convolutional models seem better for some vision tasks like segmentation which are less obvious how to do with ViTs. Convolution seems like something you fundamentally want to do in order to model translation invariance in vision.
- famouswaffles 3y agoSegment Anything is a transformer https://segment-anything.com/ https://segment-anything.com/
- lostmsu 3y agoBut what about rotational invariance? I feel like tilts are about as common as slides.
- matrix2596 3y agoI haven't fully read the paper yet. Isn't the strength of Vision Transformers in unsupervised learning, meaning that the data doesn't need labels? And don't ResNets require labeled data?
- janalsncm 3y agoIn the vision transformer paper they trained on cfar 10 and Imagenet which are supervised learning problems.
- Herring 3y agoIt's possible to adapt ResNets for unsupervised learning. A popular approach is to use self-supervised learning techniques, where the model is initially trained to predict some aspect of the data (e.g., predicting a missing part of an image) without explicit labels. This is similar to masked language modeling.