4 ms·
Does anyone know of any existing research into the effects of scaling the number of experts up in models like this? I assume there's diminishing returns and an
by drusepth 3y ago
Does anyone know of any existing research into the effects of scaling the number of experts up in models like this? I assume there's diminishing returns and an eventual inversion of quality (maybe?), but it'd be interesting to see the curve.
- ajtejankar 3y agoThere is an 'old' paper https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538 but I believe there needs to be renewed effort in this direction.