3 ms·

I don’t think we have significantly more data, and useful data won’t increase just on a whim. And it’s just one of the very necessary parts.

• CamperBob2 3y ago
  And it’s just one of the very necessary parts. That remains to be seen. You wouldn't say a human lacked human-level intelligence just because they had only read every book in The Pile, rather than every book in the Library of Congress.
• theptip 3y ago
  We have loads more data available. For text, as LLMs start to converse constantly with users and receive direct feedback, this unlocks more and better data. Multimodal (image/video, audio) is barely being scratched. You likely could 100x training compute and not run out of video content. Not to mention synthetic data from simulators, compilers, and other sources of infinite ground truth data. If you listen to the big labs (OpenAI, FAIR, etc) they have started to say they are not worried about data anymore. Quite obviously, at some point you put the agents out into the real world and they can collect new training data there too; this is what all the ChatGPT plugins are allowing OpenAI to do. The plugins that give good responses which are valuable will be disproportionately popular and therefore overrepresented in the new training set of user interactions.
  • Salgat 3y ago
    Text is the most useful data we have since it is so informationally dense; it's basically audio/visual data that has already been converted by a wet neural net into useful information. Video/audio is not going to be as significant a contribution as people think when it comes to training a language model on expressing new knowledge. It's best to think of video and audio as text encoded in a very complex layer of noise.