I had a similar thought. I think in this instance, they mean something like a labeled/supervised training approach where the model is given not just the tokeniz

I had a similar thought. I think in this instance, they mean something like a labeled/supervised training approach where the model is given not just the tokenized code, but also perhaps is grounded in the possible structure (indentation vs. semicolons, function-scope, etc.). My understanding is that copilot is largely a self-supervised approach. They feed a massive body of (somewhat noisy) code into the model. The model really does learn a lot of structure on its own and this is a testament to deep learning on noisy datasets. I'm guessing the "hooks" that they have already from IDE's, language-servers, etc. _are_ quite "structure-aware" - so they want the predicted structure as well as the code, so they can improve the typing experience beyond line-completion. I think the estimation of 100 million for such a task is maybe too high? I don't know - it feels like you could actually get quite close to such a system by simply using thousands of custom prompt engineering tricks that prepend structure examples to the prompt?