Text is the most useful data we have since it is so informationally dense; it's basically audio/visual data that has already been converted by a wet neural net

Text is the most useful data we have since it is so informationally dense; it's basically audio/visual data that has already been converted by a wet neural net into useful information. Video/audio is not going to be as significant a contribution as people think when it comes to training a language model on expressing new knowledge. It's best to think of video and audio as text encoded in a very complex layer of noise.