3 ms·

I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling

I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that is slowly combined through the layers, then relative positioning can be decoded when the representation is transformed into a dense 1d-vector aka flattening. In contrast, attention mechanisms in transformers can be seen as taking into consideration dense graphs of the whole input (at least in text, I haven't really worked with vision transformers but if an attention mechanism exists then it should be similar), along with some positional encoding and a neighborhood summary. If they indeed can be thought as stacking neighborhood summaries along with attention mechanisms, then they shouldn't fall for the same tricks since they have access to "global" information instead of disconnected components. But take this reply with a grain of salt as I am still learning about Geometric DL. If I misunderstood something, please correct me.

deleted 5y ago
[deleted]
dpflan 5y ago
Thank you for bringing up GDL. I've been following its developments, and a really great resource is this site: https://geometricdeeplearning.com/ https://geometricdeeplearning.com/ It contains links to the paper and lectures, and the keynote by M. Bronstein is illuminating and discusses the operations on graphs that lead to equivalence to other network topologies and designs: transformer equivalence, and more. > Keynote: https://www.youtube.com/watch?v=9cxhvQK9ALQ https://www.youtube.com/watch?v=9cxhvQK9ALQ How are you learning/studying GDL? Would you like someone else to discuss/learn it with?
- PartiallyTyped 5y ago
  I believe Bronstein is onto something huge here, with massive implications. Illuminating is the best adjective to describe it, as I watched this keynote: > https://www.youtube.com/watch?v=w6Pw4MOzMuo https://www.youtube.com/watch?v=w6Pw4MOzMuo Everything clicked into place and I was given a new language to see the world that combined everything together well beyond the way standard DL is taught: > we do feature extraction using this function that resembles the receptive fields of the visual cortex and then we project the dense feature representation onto multiple other vectors and pass that through stacked non-linearities, and oh by the way we have myriad of different, seemingly disconnected, architectures that we are not sure why they work, but we call it inductive bias. > https://geometricdeeplearning.com/ https://geometricdeeplearning.com/ That's my main source, along with the papers that lead up to the proto-book, so pretty much Bronstein's work along with related papers found using `connectedpapers.com`. I don't have an appropriate background so I am grinding through abstract algebra, geometric algebra, will then go into geometry and whatever my supervisor suggests I should read. Sure, I would like to have other people to discuss it, but don't expect much just yet.
  - dpflan 5y ago
    I agree, this perspective is very interesting and tames the zoo of architectures through mathematical unification. It is indeed exciting! Good luck with your studies/learning!
kaibee 5y ago
> I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that is slowly combined through the layers, then relative positioning can be decoded when the representation is transformed into a dense 1d-vector aka flattening. Hey just trying to check my understanding, this is what Taco-tron does right? It outputs an image that can be Fourier transformed into a soundwave, which is the flatting to a dense 1d vector? And the construction of that image works because the network was able to learn from examples of existing sounds transformed into images, because the transformation into an image encodes some invariance that biases the learning network to generalize better or something? I didn't do really great in math in college, but always found deep learning interesting. Not sure if anything I said above makes any sense.

deleted 5y ago

[deleted]

dpflan 5y ago

Thank you for bringing up GDL. I've been following its developments, and a really great resource is this site: https://geometricdeeplearning.com/ https://geometricdeeplearning.com/ It contains links to the paper and lectures, and the keynote by M. Bronstein is illuminating and discusses the operations on graphs that lead to equivalence to other network topologies and designs: transformer equivalence, and more. > Keynote: https://www.youtube.com/watch?v=9cxhvQK9ALQ https://www.youtube.com/watch?v=9cxhvQK9ALQ How are you learning/studying GDL? Would you like someone else to discuss/learn it with?

PartiallyTyped 5y ago
I believe Bronstein is onto something huge here, with massive implications. Illuminating is the best adjective to describe it, as I watched this keynote: > https://www.youtube.com/watch?v=w6Pw4MOzMuo https://www.youtube.com/watch?v=w6Pw4MOzMuo Everything clicked into place and I was given a new language to see the world that combined everything together well beyond the way standard DL is taught: > we do feature extraction using this function that resembles the receptive fields of the visual cortex and then we project the dense feature representation onto multiple other vectors and pass that through stacked non-linearities, and oh by the way we have myriad of different, seemingly disconnected, architectures that we are not sure why they work, but we call it inductive bias. > https://geometricdeeplearning.com/ https://geometricdeeplearning.com/ That's my main source, along with the papers that lead up to the proto-book, so pretty much Bronstein's work along with related papers found using `connectedpapers.com`. I don't have an appropriate background so I am grinding through abstract algebra, geometric algebra, will then go into geometry and whatever my supervisor suggests I should read. Sure, I would like to have other people to discuss it, but don't expect much just yet.
- dpflan 5y ago
  I agree, this perspective is very interesting and tames the zoo of architectures through mathematical unification. It is indeed exciting! Good luck with your studies/learning!

kaibee 5y ago

> I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that is slowly combined through the layers, then relative positioning can be decoded when the representation is transformed into a dense 1d-vector aka flattening. Hey just trying to check my understanding, this is what Taco-tron does right? It outputs an image that can be Fourier transformed into a soundwave, which is the flatting to a dense 1d vector? And the construction of that image works because the network was able to learn from examples of existing sounds transformed into images, because the transformation into an image encodes some invariance that biases the learning network to generalize better or something? I didn't do really great in math in college, but always found deep learning interesting. Not sure if anything I said above makes any sense.