4 ms·

No, it would be a gross misunderstanding to think ChatGPT has anywhere close to all the data possible. Not even close to all the data on the internet. Not even

by wilg 3y ago

No, it would be a gross misunderstanding to think ChatGPT has anywhere close to all the data possible. Not even close to all the data on the internet. Not even close to all text. Let alone data available by directly interacting with the world.

lelanthran 3y ago
> Not even close to all the data on the internet I agree with your other points, but why would you think ChatGPT was not given all the data on the internet? If you aren't storing the text, the only thing that stops you retrieving all the pages that can possibly be found on the internet is a small amount of money. I'm pretty certain that OpenAI has a lot more than a small amount of money.
- wilg 3y ago
  In addition to what others have said, there is a significant amount of data on the internet that is not in text form.
- revertmean 3y ago
  Because if it was, it would mostly talk about porn? :)
- yardstick 3y ago
  There’s lots of paywalled content, and other content hidden behind logins and group memberships (Eg Facebook posts, University ex-alumni portals, University course portals). Even the paywall issue alone, I can’t see how they could scale doing paywall signups automatically. Each paywall form is different, may require a local phone number in a different country to receive a text, etc.
  - hosh 3y ago
    LLMs might be good enough to sign up for sites, though maybe not yet fool “I am a human” test.
- namaria 3y ago
  You're severely underestimating how much content is on the internet and how hard it would be to see and index it all. Chat OpenAI used common crawl dataset, which is already pretty unwieldy and represents an amalgamation data gathered over several years by many crawlers.
woeirua 3y ago
It’s a bit of an open question as to how much of that data is: high quality, unique, and available. It could be that OpenAI used most of what satisfies those constraints. Training on low quality data won’t help improve its accuracy on queries, nor will duplicative data.

by wilg 3y ago

lelanthran 3y ago

> Not even close to all the data on the internet I agree with your other points, but why would you think ChatGPT was not given all the data on the internet? If you aren't storing the text, the only thing that stops you retrieving all the pages that can possibly be found on the internet is a small amount of money. I'm pretty certain that OpenAI has a lot more than a small amount of money.

wilg 3y ago
In addition to what others have said, there is a significant amount of data on the internet that is not in text form.
revertmean 3y ago
Because if it was, it would mostly talk about porn? :)
yardstick 3y ago
There’s lots of paywalled content, and other content hidden behind logins and group memberships (Eg Facebook posts, University ex-alumni portals, University course portals). Even the paywall issue alone, I can’t see how they could scale doing paywall signups automatically. Each paywall form is different, may require a local phone number in a different country to receive a text, etc.
- hosh 3y ago
  LLMs might be good enough to sign up for sites, though maybe not yet fool “I am a human” test.
namaria 3y ago
You're severely underestimating how much content is on the internet and how hard it would be to see and index it all. Chat OpenAI used common crawl dataset, which is already pretty unwieldy and represents an amalgamation data gathered over several years by many crawlers.

woeirua 3y ago

It’s a bit of an open question as to how much of that data is: high quality, unique, and available. It could be that OpenAI used most of what satisfies those constraints. Training on low quality data won’t help improve its accuracy on queries, nor will duplicative data.