I have not read the paper beyond the abstract so this is something of a throwaway comment. The setup of the study raises questions about construct validity. Th

I have not read the paper beyond the abstract so this is something of a throwaway comment. The setup of the study raises questions about construct validity. The main purpose of these coding assistants is to write new code - i.e. code with a function that is not directly expressed in their training set. The choice of such a simple (and traditional teaching task) for the study means that the function will be heavily represented in the training set. Furthermore, correct approaches to the problem will be represented more heavily than incorrect approaches. As I said I did not read beyond the abstract so I would be happy to be corrected if the authors have accounted for this in their study.