4 ms·
As far as I understand Codex is a fine-tuned GPT-3. GPT-3 was trained on a corpus derived from "the internet" (WikiPedia, links from Reddit with enough votes,
by fpgaminer 5y ago
As far as I understand Codex is a fine-tuned GPT-3.
GPT-3 was trained on a corpus derived from "the internet" (WikiPedia, links from Reddit with enough votes, and a filtered Common Crawl). So not only would GPT-3 had been exposed to code with comments, it would likely have read code examples on WikiPedia, tutorials online, API documentation, and even answers to questions on sites like StackOverflow.
The fine tuning itself is, as far as I know, from code only. So it would lean heavily on comments there. But it has a basis of understanding from the aforementioned sources.