4 ms·
There's Wu Dao 2.0 and Google has 2 models with 1T+ params.
by machiaweliczny 5y ago
There's Wu Dao 2.0 and Google has 2 models with 1T+ params.
- atty 5y agoFor clarity, i believe these are all mixture of expert models, where each input only sparsely activates some subset subset of the full model. This is why they were able to make such a big jump over the “dense” GPT3. Not really an apples-to-apples comparison.