Well, it scraped the entire web, and more. It's hard to avoid overfitting with a dataset that large, so it can develop a fixation on certain data sometimes, ins

Well, it scraped the entire web, and more. It's hard to avoid overfitting with a dataset that large, so it can develop a fixation on certain data sometimes, instead of averaging out. I remember someone trying it with smaller GPT-3 models. Usually it generated fictional phone numbers and other private data about real persons, but occasionally they were real.