4 ms·
If I recall correctly, GPT-3 is trained from Common Crawl archives. Like Internet Archive, Common Crawl does allow websites to "opt out" of being crawled. Bot
by 1vuio0pswjnm7 4y ago
If I recall correctly, GPT-3 is trained from Common Crawl archives.
Like Internet Archive, Common Crawl does allow websites to "opt out" of being crawled.
Both Internet Archive and Google Cache store the Common Crawl's archives.
It is not difficult for anyone to search through Common Crawl archives. The data is not only useful for constructing something like "GPT-3". Even if one manages to get OpenAI to "remove" their PII from GPT-3, it is still accessible to anyone through the Common Crawl archives.
Common Crawl only crawls a "representative sample" of the www (cf. internet). Moreover, this excludes "walled gardens" full of "user-generated content". AFAIK, it is not disclosed how Common Crawl decides what www sites to crawl. It stands to reason that it is based on subjective (and probably commercially-oriented) criteria.
Common Crawl's index is tiny compared to Google's. Google has claimed there is much of the www it does not bother to crawl because it is "garbage". That is obviously a subjective determination and, assuming Google operates as it describes in its SEC disclosures, almost certainly must be connected to a commercial strategy based on advertising. Advertising as a determinant for "intelligence". Brilliant.
OpenAI myopically presumes that the discussions people have and the "content" they submit via the websites in Common Crawl's index somehow represent "intelligence". If pattern-matching against the text of websites is "intelligence", then pigs can fly. Such data may be useful as a representation of www use and could be used to construct a convincing simulation of today's "interactive www"^1 but that is hardly an accurate representation of human intelligence, e.g., original thought. Arguably it is more a representation of human contagion.
There are millions of intelligent people who do not share their thoughts with www sites, let alone have serious discussions via public websites operated by "tech" company intermediaries surveilling www users with the intent to profit from the data they collect. IME, the vast majority of intelligent people are disinterested in computers and the www. As such, any discussion of this phenomena _via the www_ will be inherently biased. It is like having discussions about the world over Twitter. The views expressed will be limited to those of Twitter users.
1. This could be solution to the problem faced by new www sites that intend to rely on UGC but must launch with no users. Users of these sites can be simulated. It is a low bar to simulate such users. Simulating "intelligence" is generally not a requireemnt.