9 ms·

The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that ca

The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that case will become "when it's right" and some amount of "when it's any amount of wrong" cases will become "when it's a little wrong". A few more cycles of this and we could be looking at GPT-10 which is a complete replacement for most tasks.

ChatGTP 3y ago
Hmm, wouldn’t this only be useful if you were solving the same problems over and over again, which does happen but say in the case of programming, I’d put the reusable pieces into a library, not a language model? Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?
- letitgo12345 3y ago
  the idea is it wld learn how to do reasoning better based on this
croes 3y ago
Better result from less data? I doubt that.
- nah93 3y ago
  [dead]
- ftxbro 3y ago
  I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help. A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.
  - hooande 3y ago
    Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.
    - ftxbro 3y ago
      > Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
    - jasmer 3y ago
      [dead]
    - imtringued 3y ago
      Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.
  - ilaksh 3y ago
    I think TinyStories is a promising direction. I just wish we had an alternative to GPT-4 because supposedly you can't use it to train other models. Or maybe clarification of people publicly saying they are going to ignore that restriction.
- adg33 3y ago
  This is actually possible - if you have a biased dataset, then more data is bad. More data will fix variance problems, but not bias.
  - dTal 3y ago
    This is also why it's a bad idea to read propaganda outlets to "stay more informed".
- letitgo12345 3y ago
  ? adding on ChatGPT data into existing data is not lessening the amount of data...

ChatGTP 3y ago

Hmm, wouldn’t this only be useful if you were solving the same problems over and over again, which does happen but say in the case of programming, I’d put the reusable pieces into a library, not a language model? Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?

letitgo12345 3y ago
the idea is it wld learn how to do reasoning better based on this

croes 3y ago

Better result from less data? I doubt that.

nah93 3y ago
[dead]
ftxbro 3y ago
I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help. A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.
- hooande 3y ago
  Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.
  - ftxbro 3y ago
    > Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
  - jasmer 3y ago
    [dead]
  - imtringued 3y ago
    Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.
- ilaksh 3y ago
  I think TinyStories is a promising direction. I just wish we had an alternative to GPT-4 because supposedly you can't use it to train other models. Or maybe clarification of people publicly saying they are going to ignore that restriction.
adg33 3y ago
This is actually possible - if you have a biased dataset, then more data is bad. More data will fix variance problems, but not bias.
- dTal 3y ago
  This is also why it's a bad idea to read propaganda outlets to "stay more informed".
letitgo12345 3y ago
? adding on ChatGPT data into existing data is not lessening the amount of data...