4 ms·
Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model wa
by byyoung3 2y ago
Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model was trained on a separate objective. It's a brilliant idea that works amazingly.
- psb217 2y agoIt's a classic "Will it work? IDK, maybe. Let's try it and find out..." paper.
- byyoung3 2y agohaha yeah I mean I think they are all like that to a certain extent
- fxtentacle 2y agoTo me, it seemed that the technique presented here was just a logical continuation of methods that OpenAI used when they trained the Dota agents: https://arxiv.org/pdf/1912.06719v1 https://arxiv.org/pdf/1912.06719v1 And, arguably, Facebook's unsupervised pre-training for their multi-modal speech-to-text models is kind of the same idea as unsupervised pre-training for a multi-modal text-to-image diffuser. https://ai.meta.com/research/publications/wav2vec-2.0-a-framework-for-self-supervised-learning-of-speech-representations/ https://ai.meta.com/research/publications/wav2vec-2.0-a-fram...