5 ms·

Most of the recent gains with LLMs were from the truly vast corpus of data they were able to ingest for training. And at this point, there may not be much more

Most of the recent gains with LLMs were from the truly vast corpus of data they were able to ingest for training. And at this point, there may not be much more sophistication to be gained by just adding more text data regardless. Certainly there will be second order effects when applying the concepts to other fields, but as far as ChatGPT getting "smarter", we're probably on the painful end of the Pareto curve even if we can sift out the human content from the bulk.

gibolt 3y ago
More "quality" data equals more better. The argument here is the LLM generated text is now going to enter the corpus, muddying the waters and reducing the quality.
- chaxor 3y ago
  Perhaps Meta will release their dataset for Galactica sometime, and we will have incredibly good training data quality.
- dontupvoteme 3y ago
  Current LLM outputs already kind of reek of reddit-style responses, at least to me. Is a hivemind really any different than an AI making its own data?
crote 3y ago
They might not even need more sophistication, but even something as simple as updating their data might become increasingly difficult. The initial data set was essentially created by undiscriminatingly crawling the internet. This worked reasonably well because up until now most of the internet was - in one way or another - created by humans. This is no longer the case, as LLMs are incredibly attractive when you want to create spam. Anyone who wants to get any general dataset past 2022 will have to deal with the reality that a significant amount of crawled content will have been written by a LLM and is therefore essentially unusable for training. Facts are useless when they have been hallucinated!
- JustBreath 3y ago
  > updating their data might become increasingly difficult. Very true, I suspect part of the changes at Reddit are being driven by them wanting to hoard their data from AI's et. al or at least make them pay for it.
  - bugglebeetle 3y ago
    This will not solve any of Reddit’s problems or preserve their data value. While more complicated than an API, it’s trivial to run accounts flooding Reddit with AI generated content through browser automation. Even if they switch to app only access, this is still not meaningfully challenging and app-only would also destroy the service.
    - jprete 3y ago
      Reddit doesn’t care about hosting AI content as long as it engages people. The idea (probably wrong) was that Reddit didn’t want to give free unlimited access to the data to anyone training an AI.
      - TeMPOraL 3y ago
        That idea makes no sense though. The whole Reddit corpus from before ChatGPT is already out there on the Internet, packaged, mirrored and ready for download. The content that was submitted after ChatGPT became publicly available is "adulterated" by LLM-generated text, to an unknown but growing degree. I.e. all the valuable data is already out, and whatever new data Reddit would want to gatekeep is losing its value with each passing minute.
  - dontupvoteme 3y ago
    Is post 2023-05 reddit data really that valuable? the website's become quite cliched at this point, it's apparently full of bot spam, and you already have a ton of data to work with. What would be valuable to sell is the real upvote/downvote information. I would be more interested to see if all of per-Eternal September Usenet is included or not. Just reprocess the data so they're having friendly chats about vi vs emacs and you don't have to worry about toxicity.
Hendrikto 3y ago
> And at this point, there may not be much more sophistication to be gained by just adding more text data regardless. There is consensus that almost all contemporary LLMs are undertrained. See, for example, the Gopher, Chinchilla, and LLaMA papers. Larger models are easier to train, and there are diminishing returns when you keep training. Thus, to claim SotA performance, researchers tried to optimize for the best performance, given a certain training budget. The best performance within a certain training budget is achieved by training HUGE models on VAST datasets, for a relatively short amount of time. Almost all models could profit from simply training for longer, but the cost/benefit isn‘t there, if your goal is to achieve the best performance. This is also why distillation and quantization work so well. Models with more and larger weights are easier to train, but ultimately don‘t utilize all that capacity. Recently, researchers have begun focussing on inference – rather than training – budgets, in order to make using these models actually viable and profitable in practice. I.e., what is the best performance we can achieve within a certain computational limit at inference time? This is mostly done by simply training with more data, for longer.
- JustBreath 3y ago
  Are you saying actually contemporary LLMs would benefit from more data?
  - Hendrikto 3y ago
    Yes, and longer training as well.

gibolt 3y ago

More "quality" data equals more better. The argument here is the LLM generated text is now going to enter the corpus, muddying the waters and reducing the quality.

chaxor 3y ago
Perhaps Meta will release their dataset for Galactica sometime, and we will have incredibly good training data quality.
dontupvoteme 3y ago
Current LLM outputs already kind of reek of reddit-style responses, at least to me. Is a hivemind really any different than an AI making its own data?

crote 3y ago

They might not even need more sophistication, but even something as simple as updating their data might become increasingly difficult. The initial data set was essentially created by undiscriminatingly crawling the internet. This worked reasonably well because up until now most of the internet was - in one way or another - created by humans. This is no longer the case, as LLMs are incredibly attractive when you want to create spam. Anyone who wants to get any general dataset past 2022 will have to deal with the reality that a significant amount of crawled content will have been written by a LLM and is therefore essentially unusable for training. Facts are useless when they have been hallucinated!

JustBreath 3y ago
> updating their data might become increasingly difficult. Very true, I suspect part of the changes at Reddit are being driven by them wanting to hoard their data from AI's et. al or at least make them pay for it.
- bugglebeetle 3y ago
  This will not solve any of Reddit’s problems or preserve their data value. While more complicated than an API, it’s trivial to run accounts flooding Reddit with AI generated content through browser automation. Even if they switch to app only access, this is still not meaningfully challenging and app-only would also destroy the service.
  - jprete 3y ago
    Reddit doesn’t care about hosting AI content as long as it engages people. The idea (probably wrong) was that Reddit didn’t want to give free unlimited access to the data to anyone training an AI.
    - TeMPOraL 3y ago
      That idea makes no sense though. The whole Reddit corpus from before ChatGPT is already out there on the Internet, packaged, mirrored and ready for download. The content that was submitted after ChatGPT became publicly available is "adulterated" by LLM-generated text, to an unknown but growing degree. I.e. all the valuable data is already out, and whatever new data Reddit would want to gatekeep is losing its value with each passing minute.
- dontupvoteme 3y ago
  Is post 2023-05 reddit data really that valuable? the website's become quite cliched at this point, it's apparently full of bot spam, and you already have a ton of data to work with. What would be valuable to sell is the real upvote/downvote information. I would be more interested to see if all of per-Eternal September Usenet is included or not. Just reprocess the data so they're having friendly chats about vi vs emacs and you don't have to worry about toxicity.

Hendrikto 3y ago

> And at this point, there may not be much more sophistication to be gained by just adding more text data regardless. There is consensus that almost all contemporary LLMs are undertrained. See, for example, the Gopher, Chinchilla, and LLaMA papers. Larger models are easier to train, and there are diminishing returns when you keep training. Thus, to claim SotA performance, researchers tried to optimize for the best performance, given a certain training budget. The best performance within a certain training budget is achieved by training HUGE models on VAST datasets, for a relatively short amount of time. Almost all models could profit from simply training for longer, but the cost/benefit isn‘t there, if your goal is to achieve the best performance. This is also why distillation and quantization work so well. Models with more and larger weights are easier to train, but ultimately don‘t utilize all that capacity. Recently, researchers have begun focussing on inference – rather than training – budgets, in order to make using these models actually viable and profitable in practice. I.e., what is the best performance we can achieve within a certain computational limit at inference time? This is mostly done by simply training with more data, for longer.

JustBreath 3y ago
Are you saying actually contemporary LLMs would benefit from more data?
- Hendrikto 3y ago
  Yes, and longer training as well.