I’ve had a side job as an external examiner for CS students for almost a decade now. While LLMs are generally terrible at programming (in my experience) they ex

I’ve had a side job as an external examiner for CS students for almost a decade now. While LLMs are generally terrible at programming (in my experience) they excel at passing finals. If I were to guess it’s likely a combination of the relatively simple (or perhaps isolated is a better word) tasks coupled with how many times similar problems have been solved before in the available training data. Somewhat ironically the easiest way to spot students who “cheat” is when the code is great. Being an external examiner, meaning that I have a full time job in software development I personally find it sort of silly when students aren’t allowed to use a calculator. I guess changing the way you teach and test is a massive undertaking though, so right now we just pretend LLMs aren’t being used by basically every students. Luckily I’m not part of the “spot the cheater” process, so I can just judge them based on how well they can explain their code. Anyway, I’m not at all surprised that they can handle AoC. If anything I would worry that AoC will still be a fun project to author when many people solve it with AI. It sure won’t be fun to curate any form of leaderboard.