> This is not trivial at all, note that this is not actually the original general purpose GPT-3. Codex has 12-billion parameters compared to GPT-3’s 175 billion

> This is not trivial at all, note that this is not actually the original general purpose GPT-3. Codex has 12-billion parameters compared to GPT-3’s 175 billion and it was trained only on open source code. The fact that it does so well is insane to me! It's not that surprising. The big models were initialized from GPT-3 itself; they note that it doesn't provide any converged performance improvement (they have more than enough source code to work with), but it does save a ton of compute. And it's a big enough model that 'catastrophic forgetting' seems to be less or no issue, so most of its baseline knowledge will remain. (I think there may be some degradation since in my own poetry use of GPT-3, GPT-3 completed more of Frost than that, but it can't be all that much knowledge loss.)