3 ms·

Big-picture, the idea is that different modalities of sensory data (visual, olfactory, etc.) are processed by different minicolumns in the brain, i.e., differen

by cs702 2y ago

Big-picture, the idea is that different modalities of sensory data (visual, olfactory, etc.) are processed by different minicolumns in the brain, i.e., different subnetworks, each outputting a different firing pattern. These firing patterns propagate across the surface area of the brain, competing with conflicting messages. And then, to quote the OP, "after some period of time a winner is chosen, likely the message that controls the greatest surface area, the greatest number of minicolumns. When this happens, the winning minicolumns are rewarded, likely prompting them to encode a tendency for that firing pattern into their structure." And this happens in multiple layers of the brain. In other words, there's some kind of iterative mechanism for higher-level layers to find which lower-level subnetworks are most in agreement about the input data, inducing learning. Capsule-routing algorithms, proposed by Hinton and others, seek to implement precisely this idea, typically with some kind of expectation-maximization (EM) process. There are quite a few implementations available on github: https://github.com/topics/capsules https://github.com/topics/capsules https://github.com/topics/em-routing https://github.com/topics/em-routing https://github.com/topics/routing-algorithm https://github.com/topics/routing-algorithm

calepayson 2y ago
Great summary. Thanks for the links. These are awesome.
abeppu 2y ago
I haven't heard of anyone talk about Hinton's capsule network concepts for some time. In 2017-18 it seemed exciting both because of Hinton but also because the pose/transformation sounded pretty reasonable. I don't know what would count as "explanation", but I'd be curious to hear any thoughts about why it seems they didn't really pan out. (Are there any tasks for which capsule methods are the best?)
- cs702 2y ago
  The short answer as to why capsule networks have "fallen out of fashion" is... Transformers. Transformers came out at roughly the same time[a] and have proven to be great at... pretty much everything. They just work. Since then, most AI research money, effort, and compute has been invested to study and improve Transformers and related models, at the expense of almost everything else. Many promising ideas, including routing, won't be seriously re-explored until and unless progress towards AGI seems to stall. --- [a] https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
  - abeppu 2y ago
    I think this is a non-answer in some sense. Yes, transformers have been clearly very successful across a very wide range of tasks. But what about the approach taken in capsules is comparatively deficient? Some kinds of explanations which I think are at least plausible (but IDK if any evidence exists for them): - The attention structure in transformers allows any chunk to be learned to be important for any other chunk. And pretty quickly people tended towards these being pretty deep. By comparison, the capsule + routing structure (IIUC) came with a built-in kind of sparsity (from capsules at a level in the hierarchy being independent), and because the hierarchy was meant to align with composition, it often (I think) didn't have a huge number of levels? Maybe this flexibility + depth are key? - Related to capsules being independent, an initial design feature in capsule networks seems to have been smaller model sizes. Perhaps this was at some level just a bad thing to reach for? I think at the time, "smaller models means optimization searches over a smaller space, which is faster to converge and requires less data" was still sort of in people's heads, and I think this view is pretty much dead. - I've heard some people argue that one of the core strengths of transformers is that they support training in a way allows for maxing out available GPUs. I think this is mostly in comparison to previous language models which were explicitly sequential. But are capsule networks less conducive to efficient training?
    - cs702 2y ago
      It's hard to make a fair comparison, because there hasn't been anywhere near as much money, effort, or compute invested in trying to scale up routing methods. Promising approaches are often ignored for reasons that have little to do with their merits. For example, Hinton, Bengio, and Lecun spent much of the 1990's and all of the 2000's in the fringes of academia, unable to get much funding, because few others were interested in or saw any promise in deep learning! Similarly, Katalin Karikó lost her job and spent almost two decades in obscurity because few others were interested or saw any promise in RNA vaccines! Now, I'm not saying routing methods will become more popular in the future. I mean, who the heck knows? What I'm saying is that promising approaches can fall out of favor for reasons that are not intrinsic to them.
- MAXPOOL 2y ago
  If you take a birds eye view, fundamental breakthroughs don't happen that often. "Attention Is All You Need" paper also came out in 2017. It has now been 7 years without breakthrough at the same level as transformers. Breakthrough ideas can take decades before they are ready. There are many false starts and dead ends. Money and popularity are orthogonal to pathfinding that leads to breakthroughs.
  - calepayson 2y ago
    Well said

by cs702 2y ago

calepayson 2y ago

Great summary. Thanks for the links. These are awesome.

abeppu 2y ago

I haven't heard of anyone talk about Hinton's capsule network concepts for some time. In 2017-18 it seemed exciting both because of Hinton but also because the pose/transformation sounded pretty reasonable. I don't know what would count as "explanation", but I'd be curious to hear any thoughts about why it seems they didn't really pan out. (Are there any tasks for which capsule methods are the best?)

cs702 2y ago
The short answer as to why capsule networks have "fallen out of fashion" is... Transformers. Transformers came out at roughly the same time[a] and have proven to be great at... pretty much everything. They just work. Since then, most AI research money, effort, and compute has been invested to study and improve Transformers and related models, at the expense of almost everything else. Many promising ideas, including routing, won't be seriously re-explored until and unless progress towards AGI seems to stall. --- [a] https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
- abeppu 2y ago
  I think this is a non-answer in some sense. Yes, transformers have been clearly very successful across a very wide range of tasks. But what about the approach taken in capsules is comparatively deficient? Some kinds of explanations which I think are at least plausible (but IDK if any evidence exists for them): - The attention structure in transformers allows any chunk to be learned to be important for any other chunk. And pretty quickly people tended towards these being pretty deep. By comparison, the capsule + routing structure (IIUC) came with a built-in kind of sparsity (from capsules at a level in the hierarchy being independent), and because the hierarchy was meant to align with composition, it often (I think) didn't have a huge number of levels? Maybe this flexibility + depth are key? - Related to capsules being independent, an initial design feature in capsule networks seems to have been smaller model sizes. Perhaps this was at some level just a bad thing to reach for? I think at the time, "smaller models means optimization searches over a smaller space, which is faster to converge and requires less data" was still sort of in people's heads, and I think this view is pretty much dead. - I've heard some people argue that one of the core strengths of transformers is that they support training in a way allows for maxing out available GPUs. I think this is mostly in comparison to previous language models which were explicitly sequential. But are capsule networks less conducive to efficient training?
  - cs702 2y ago
    It's hard to make a fair comparison, because there hasn't been anywhere near as much money, effort, or compute invested in trying to scale up routing methods. Promising approaches are often ignored for reasons that have little to do with their merits. For example, Hinton, Bengio, and Lecun spent much of the 1990's and all of the 2000's in the fringes of academia, unable to get much funding, because few others were interested in or saw any promise in deep learning! Similarly, Katalin Karikó lost her job and spent almost two decades in obscurity because few others were interested or saw any promise in RNA vaccines! Now, I'm not saying routing methods will become more popular in the future. I mean, who the heck knows? What I'm saying is that promising approaches can fall out of favor for reasons that are not intrinsic to them.
MAXPOOL 2y ago
If you take a birds eye view, fundamental breakthroughs don't happen that often. "Attention Is All You Need" paper also came out in 2017. It has now been 7 years without breakthrough at the same level as transformers. Breakthrough ideas can take decades before they are ready. There are many false starts and dead ends. Money and popularity are orthogonal to pathfinding that leads to breakthroughs.
- calepayson 2y ago
  Well said