A couple of months ago OpenAI created a model called Whisper that can transcribe speech efficiently and with high quality in 97 languages. It might be used to t

A couple of months ago OpenAI created a model called Whisper that can transcribe speech efficiently and with high quality in 97 languages. It might be used to transcribe the whole YouTube, someone estimated about 12T tokens. GPT-3 used about 0.5T tokens for training. https://twitter.com/HenriLemoine13/status/1572846452895875073 https://twitter.com/HenriLemoine13/status/157284645289587507... If you add the visual data you get another order of magnitude increase in size. I think this training will teach the model procedural knowledge - how to do things step by step, and that will have a huge impact in robotics and software agents.