4 ms·
I'm not a fan of the database lookup analogy either. The analogy I prefer when teaching attention is celestial mechanics. Tokens are like planets in (latent) s
by ianand 9mo ago
I'm not a fan of the database lookup analogy either.
The analogy I prefer when teaching attention is celestial mechanics. Tokens are like planets in (latent) space. The attention mechanism is like a kind of "gravity" where each token is influencing each other, pushing and pulling each other around in (latent) space to refine their meaning. But instead of "distance" and "mass", this gravity is proportional to semantic inter-relatedness and instead of physical space this is occurring in a latent space.
https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=3569s https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=3569s
- art_mach 9mo agoThis is an excellent analogy! Thank you!
- skinner_ 9mo agoThen I think you’ll like our project which aims to find the missing link between transformers and swarm simulations: https://github.com/danielvarga/transformer-as-swarm https://github.com/danielvarga/transformer-as-swarm Basically a boid simulation where a swarm of birds can collectively solve MNIST. The goal is not some new SOTA architecture, it is to find the right trade-off where the system already exhibits complex emergent behavior while the swarming rules are still simple. It is currently abandoned due to a serious lack of free time (*), but I would consider collaborating with anyone willing to put in some effort. (*) In my defense, I’m not slacking meanwhile: https://arxiv.org/abs/2510.26543 https://arxiv.org/abs/2510.26543 https://arxiv.org/abs/2510.16522 https://arxiv.org/abs/2510.16522 https://www.youtube.com/watch?v=U5p3VEOWza8 https://www.youtube.com/watch?v=U5p3VEOWza8