3 ms·
The feed forward layer is essentially a differentiable key-value store. Similar to the attention layer, actually. So it just uses an attention mechanism like pr
by namibj 3y ago
The feed forward layer is essentially a differentiable key-value store.
Similar to the attention layer, actually.
So it just uses an attention mechanism like pre-selector to attend to only some experts.
During inference, this cutoff is made a hard cutoff.
- mercat 3y agoThis is a very interesting approach. I know it may be too much to ask, but would you suggest any actual practical and hands-on workshops, playgrounds, or courses where I could practice using NN layers for stuff like that? For example, conditional/weighted selection of previous inputs, etc. It feels like I'm looking at ML programming from another angle.