This seems like a silly objection to me, although I am one of the authors :) Yes, Codex has been trained on lots of linked lists. It has also been trained on lo

This seems like a silly objection to me, although I am one of the authors :) Yes, Codex has been trained on lots of linked lists. It has also been trained on lots of bad and insecure versions of linked lists! And when we did our earlier study ("Asleep at the Keyboard"), which looked at the security of code written by Copilot without human intervention, we found that it produced vulnerable code at high rates (40%) even for common scenarios like SQL injection and C string manipulation. So your prior that LLMs should be good at linked lists is not at all obvious; that's why we ran the study. Other complaints in this thread about studying students - sure, perhaps experienced developers will do better. These were upper level undergraduate and masters students though, so they're the kind of developer you will have writing code with Codex in industry within a year or so, and so they are still worth studying. It's not the whole picture, but we never claim it is.