3 ms·
I'll admit I haven't seen the training data but some basic googling shows a subset of the labeling is syntax annotations. I am not claiming LLMs parse code in t
by beardedwizard 2y ago
I'll admit I haven't seen the training data but some basic googling shows a subset of the labeling is syntax annotations. I am not claiming LLMs parse code in the way you are suggesting, but they certainly have token level awareness of syntax and probable relations which are the roots of any programming language.
- corimaith 2y agoLast time I researched AST parsing was quite rudimentary for LMs. The problem was preserving their structural properties, whereby a flattening approach via node traversal tended to remove. But you needed to do to put it into a format that language models could parse.