4 ms·

This should be a step in cleaning your data to begin with. If you don't know the providence of your data then you shouldn't be even training with it. Getting h

This should be a step in cleaning your data to begin with. If you don't know the providence of your data then you shouldn't be even training with it. Getting humans to refine your data is the best solution right now and many companies and researches go with this approach.

nharada 4y ago
But how would you know? A random string of text or an image with the watermark removed is going to be very hard to distinguish generated from human written.
FrenchDevRemote 4y ago
You can't use humans to manually refine a dataset on the scale of GPT-3 or DALL-E Clip was trained on 400,000,000 images, GPT is roughly 180B tokens, at 1-2 tokens per word, that's 120,000,000,000 words.
- pshc 4y ago
  At least cleaning it up is an embarrassingly parallel problem, so if you had the resources to throw incentives at millions of casual gamers, you might make a nice dent on Clip.
  - zanderwohl 4y ago
    Alternatively, making a captcha where half the data is unlabeled, and half is labeled, forcing users to categorize data for you as they log into accounts.
jazzyjackson 4y ago
s/ide/ena
sailingparrot 4y ago
> Getting humans to refine your data is the best solution right now Source ? All those big models are trained with data for which the source is not known or vetted. The amount of data needed is not human-refinable. For example for language models we train mostly on subsets of CommonCrawl + other things. CommonCrawl data is “cleaned” by filtering out known bad sources and with some heuristics such as ratio of text to other content, length of sentences etc. The final result is a not too dirty but not clean huge pile of data that comes from millions of sources that no human as vetted and that no one in the team using the data knows about. The same applies to large images dataset, e.g. Laon 400m that also comes from CommonCrawl and is not curated.

nharada 4y ago

But how would you know? A random string of text or an image with the watermark removed is going to be very hard to distinguish generated from human written.

FrenchDevRemote 4y ago

You can't use humans to manually refine a dataset on the scale of GPT-3 or DALL-E Clip was trained on 400,000,000 images, GPT is roughly 180B tokens, at 1-2 tokens per word, that's 120,000,000,000 words.

pshc 4y ago
At least cleaning it up is an embarrassingly parallel problem, so if you had the resources to throw incentives at millions of casual gamers, you might make a nice dent on Clip.
- zanderwohl 4y ago
  Alternatively, making a captcha where half the data is unlabeled, and half is labeled, forcing users to categorize data for you as they log into accounts.

jazzyjackson 4y ago

s/ide/ena

sailingparrot 4y ago

> Getting humans to refine your data is the best solution right now Source ? All those big models are trained with data for which the source is not known or vetted. The amount of data needed is not human-refinable. For example for language models we train mostly on subsets of CommonCrawl + other things. CommonCrawl data is “cleaned” by filtering out known bad sources and with some heuristics such as ratio of text to other content, length of sentences etc. The final result is a not too dirty but not clean huge pile of data that comes from millions of sources that no human as vetted and that no one in the team using the data knows about. The same applies to large images dataset, e.g. Laon 400m that also comes from CommonCrawl and is not curated.