3 ms·

How much of existing data did they devour? - internet (incl. blogosphere, all www, forums, social networks, dark net,...) - all of digitized books - all stat

by gala8y 2y ago

How much of existing data did they devour? - internet (incl. blogosphere, all www, forums, social networks, dark net,...) - all of digitized books - all statistical data, governments data, military data - all scientific papers and books - all commercial companies data - all private data/personal data - ADDED - all video/audio data - all new data... If someone has some links on what the current state of affairs is (what they really trained on), I would be really interested.

wongarsu 2y ago
Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it. I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means. Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes). Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
- gala8y 2y ago
  > Training data is the competitive edge, so everyone refuses to elaborate about what they use. I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
- dns_snek 2y ago
  Regarding their ethical concerns or lack thereof, OpenAI Whisper will often hallucinate transcriptions such as "Subtitles by <random nickname>" which strongly indicates that it's trained on ripped movies and crowdsourced subtitles found online.

by gala8y 2y ago

wongarsu 2y ago

Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it. I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means. Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes). Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget

gala8y 2y ago
> Training data is the competitive edge, so everyone refuses to elaborate about what they use. I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
dns_snek 2y ago
Regarding their ethical concerns or lack thereof, OpenAI Whisper will often hallucinate transcriptions such as "Subtitles by <random nickname>" which strongly indicates that it's trained on ripped movies and crowdsourced subtitles found online.