You have millions of tutorials that are about a few pages long, I'm pretty sure it got its programming skills from those. But we are already at model size where

You have millions of tutorials that are about a few pages long, I'm pretty sure it got its programming skills from those. But we are already at model size where the model can keep an entire tutorial page in its context, there is no extra benefit here from expanding the context size, it would need similar tutorials but much much longer and expansive to get better than it is now, and there just aren't enough humans to write all of those.