3 ms·
Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it.
by wongarsu 2y ago
Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it.
I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means.
Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes).
Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
- gala8y 2y ago> Training data is the competitive edge, so everyone refuses to elaborate about what they use. I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
- dns_snek 2y agoRegarding their ethical concerns or lack thereof, OpenAI Whisper will often hallucinate transcriptions such as "Subtitles by <random nickname>" which strongly indicates that it's trained on ripped movies and crowdsourced subtitles found online.