4 ms·
fuck. I have an idea just like this one. I guess it's true that ideas are a dime a dozen. Diffusions bear a remarkable similarity to backpropagation to me.
by justanotherjoe 3y ago
fuck. I have an idea just like this one. I guess it's true that ideas are a dime a dozen.
Diffusions bear a remarkable similarity to backpropagation to me. I thought that it could be used in place of it for some parts of a model.
Furthermore, I posit that resnet especially in transformers allows the model into a more exploratory behavior that is really powerful, and is a necessary component of the power of transformers. Transformers is just such a great architecture the more i think about it. It's doing so many things so right. Although this is not really related to the topic.
- crotchfire 3y agoActually it is related. Transformers are just networks that learn to program the weights of other networks [1]. In the successful cases the programmed network has been quite primitive -- merely a key-value store -- in order to ensure that you can backpropagate errors from the programmed network's outputs all the way to the programmer network's inputs. The present work extends this idea to a different kind of programmed network: a convolutional image-processing network. There are many more breakthroughs to be achieved along this line of research -- it is a rich vein to mine. I believe our best shot at getting neural networks to do discrete math and symbolic logic, and to write nontrivial computer programs, will result from this line of research. [1] https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174