3 ms·
mixture of experts gates on the feed forward network only. the shared weights are the KQV projections for the attention mechanism of each layer.
by brrrrrm 3y ago
mixture of experts gates on the feed forward network only. the shared weights are the KQV projections for the attention mechanism of each layer.