3 ms·
>since they trained on a huge chunk of web text What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?
by yreg 4y ago
>since they trained on a huge chunk of web text
What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?
- sarchertech 4y agoBecause they trained it on a huge percentage of the existing web. There's isn't a (much) bigger and better data set available.
- yreg 4y agoWhat percentage? What about books, newspapers, documents, etc.?
- JW_00000 4y agoThe LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources. [1] https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971
- m_ke 4y agoOpenAI is working with Microsoft so they definitely had access to the full Bing index and data from their other platforms like Github and Linkedin. They also paid for private datasets, from what I heard they might have gotten a copy of Quora and I'm sure they got a dump of all digitized books from someone. Their best bet now is getting more supervised conversational data, which they should be getting a ton of from Bing and ChatGPT usage (they can use it as is with RLHF dataset which they had to pay people to generate by having fake conversations). I wouldn't be surprised if they partner with Microsoft and hire a large team of doctors to tune it to handle specific medical conditions like diabetes.