4 ms·

Is all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up

Is all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up to the present day? Do we really need that to keep improving AI? GPT is already capable of incredible generalized language understanding and I'd wager we've long since hit diminishing returns from raw internet data. RLHF, fine tuning, and better (and more data efficient) architectures are what we need now.

radarsat1 3y ago
People have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well. I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.
- photonerd 3y ago
  The “somehow” being “it was explicitly marketed that way”
rhdunn 3y ago
Language shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification". If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.
- cj 3y ago
  I always imagined a great source of training would be subtitles of live news or other live programming. Would help to stay up to date on current events, and up to date on its general understanding of the world. With the risk being adopting the bias of news media.
barrysteve 3y ago
People involved in AI have incentive to say 'yes' to everything. Nobody will make anything value in public, that they didn't want to be released for free anyway. ChatGPT's vacuum has brought back a desire for privacy and will probably contribute to destroying piracy too. ChatGPT has destroyed the 'study hard and get reward loop' for collaborative effort on the internet. If you use chatGPT, it absorbs all your question data and gives you nothing in return. You can't commit to random people, as they are expected to leak your IP onto gpt. Isaac Newton using chatGPT would upload the core of calculus to GPT in research questions, and see no personal benefit for doing so. There is no greater thief in history of academic work, than electronics.

radarsat1 3y ago

People have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well. I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.

photonerd 3y ago
The “somehow” being “it was explicitly marketed that way”

rhdunn 3y ago

Language shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification". If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.

cj 3y ago
I always imagined a great source of training would be subtitles of live news or other live programming. Would help to stay up to date on current events, and up to date on its general understanding of the world. With the risk being adopting the bias of news media.

barrysteve 3y ago

People involved in AI have incentive to say 'yes' to everything. Nobody will make anything value in public, that they didn't want to be released for free anyway. ChatGPT's vacuum has brought back a desire for privacy and will probably contribute to destroying piracy too. ChatGPT has destroyed the 'study hard and get reward loop' for collaborative effort on the internet. If you use chatGPT, it absorbs all your question data and gives you nothing in return. You can't commit to random people, as they are expected to leak your IP onto gpt. Isaac Newton using chatGPT would upload the core of calculus to GPT in research questions, and see no personal benefit for doing so. There is no greater thief in history of academic work, than electronics.