7 ms·
two thoughts People love to say "it's early" and "it will improve" about ChatGPT. but amount of training data IS the dominant factor in determining the quality
by hooande 4y ago
two thoughts
People love to say "it's early" and "it will improve" about ChatGPT. but amount of training data IS the dominant factor in determining the quality of the output, usually in logarithmic terms. it's already trained on the entire internet. it's hard to see how they'll be able to significantly increase that.
And having models build models is drastically overrated. again, the accuracy/quality improvements are largely driven by the scale and diversity of the dataset. that's like 90% of the solution to any ml problem. choosing the right model and parameters is often a minor relative improvement
- tlarkworthy 4y agoAll the books. I think books might be better
- recuter 4y agoThere's only about 100m books (in English). About the same volume of text as the web all total.
- MonkeyClub 4y agoGenerally a book has deeper thinking than a webpage, though, I think that’s the crux of the GP’s clarification.
- disgruntledphd2 4y agoAnd the distribution of text will be very different for all books vs the internet.
- albert_e 4y agoYes. And (with scary privacy implications) maybe the next frontier is capturing all spoken language uttered by people in real-time and streaming it into a model that is being updated in near-real-time. And finally brain implants extracting unspoken thoughts and neural activity and combining it all. Extend it to non human life forms as well. (universal consciousness?)
- akomtu 4y agoIt's the future where everyone is a universal captcha solver for the needs of the Great AI. No freedom even in your thoughts.
- deleted 4y ago[deleted]
- theGnuMe 4y ago> it's hard to see how they'll be able to significantly increase that. With feedback.
- DenisM 4y agoYes. Which is why ChatGPT is open to public.
- not2b 4y agoPerhaps some sort of adversarial network approach could work better; models that learn to generate text and other models that try to distinguish AIs from humans, competing against each other. Also, children learning language benefit from constant feedback from people who have their best interest at heart ... that last part is important because of episodes like Microsoft's Tay where 4chan folks thought it would be fun to turn the chatbot into a fascist.
- ronsor 4y agoI think OpenAI has already published some research showing humans preferred smaller/fewer parameter models that were "better trained" through the use of human feedback. If there were a model that could adequately replace the role of the human, then that approach would probably work well.
- charcircuit 4y agoPassing the Turing test isn't the goal. You can have a useful model that isn't human like and can have a useless model that you can't tell isn't a human.
- not2b 4y agoCertainly. Approaches that have done better on the Turing test have used various tricks to avoid their lack of understanding, like playing a paranoid person. But some of the best chatbots give themselves away by getting stuck in a loop or demonstrating lack of basic intuition about the world that a three year old would have. Those things perhaps could be caught.
- geysersam 4y agoThe data is the bottleneck for the current generation of models. Better models/training strategies could very well change that in the next couple of decades.
- dr_dshiv 4y agoWell, diffusion networks seemed to work because they increased the amount of data by training so many examples of added noise and it’s removal. Some similar approach might be possible with text, too. Or, i guess they will talk to itself, like alpha go played itself.
- rhacker 4y agoThe next phase has got to be more and more private data sets that don't exist on the internet: Homes with Alexa, Ok Google, etc.. Beyond that, linkages into a human brain.