3 ms·
Any plans on training other (non nlp) huge models using ZeRO? Specifically for Transformers - any plans to train a big model with a bigger context window? Not
by Tenoke 7y ago
Any plans on training other (non nlp) huge models using ZeRO?
Specifically for Transformers - any plans to train a big model with a bigger context window?
Not that this one isn't very impressive, of course.
- saurkt 7y agoThanks for your kind words. Yes, we would like to next train a language representation model. And our hunch is that probably something which is a mixture of language representation and language generation would be able to get the best of both worlds.