Not sure if there is a technical definition of the term. I am using it to refer to the transformer-based neural networks that are used to build models like Open

Not sure if there is a technical definition of the term. I am using it to refer to the transformer-based neural networks that are used to build models like Open AI's Codex (which GitHub Copilot uses IIRC). I believe they are called large because they contain orders of magnitude more parameters than the models that came before them. They do not fit in memory on a single training device (GPU or TPU usually) and require serious infrastructure work to train effectively. The models are also trained on large datasets. These datasets are usually text streams. In the case of GitHub Copilot, it was the textual source code of a large percentage of the git repositories on GitHub. (Maybe supplemented with other code?) I am curious if anyone is parsing source code into syntax trees and then training large language models to operate on the resulting graph structures. Edit: On a related note - GitHub Copilot is extremely good at generating syntactically correct code. Has anyone studied which parameters in the language model are related to learning programming language syntax? Are the parameters in the model related to javascript syntax "close to" the parameters in the model which learn Go syntax? Edit 2: For GitHub Copilot, did they use the content of git diffs to learn the kinds of edits people make to existing code? Edit 3: I realize that a random HN post about some blog is probably not the best place to be asking these questions. If anyone could point me to a better place, I would appreciate it.