3 ms·
Half the time I try to use gemini questions about the c++ std library, it fabricates non-existent types and functions. I'm honestly impressed it was able to sol
by unclad5968 2y ago
Half the time I try to use gemini questions about the c++ std library, it fabricates non-existent types and functions. I'm honestly impressed it was able to solve any of the AoC problems.
- devjab 2y agoI’ve had a side job as an external examiner for CS students for almost a decade now. While LLMs are generally terrible at programming (in my experience) they excel at passing finals. If I were to guess it’s likely a combination of the relatively simple (or perhaps isolated is a better word) tasks coupled with how many times similar problems have been solved before in the available training data. Somewhat ironically the easiest way to spot students who “cheat” is when the code is great. Being an external examiner, meaning that I have a full time job in software development I personally find it sort of silly when students aren’t allowed to use a calculator. I guess changing the way you teach and test is a massive undertaking though, so right now we just pretend LLMs aren’t being used by basically every students. Luckily I’m not part of the “spot the cheater” process, so I can just judge them based on how well they can explain their code. Anyway, I’m not at all surprised that they can handle AoC. If anything I would worry that AoC will still be a fun project to author when many people solve it with AI. It sure won’t be fun to curate any form of leaderboard.
- uludag 2y agoleetoced/AoC-like problems are probably the easiest class of software related tasks LLMs can do. Using the correct library, the correct way, at the correct version, especially if the library has some level of churn and isn't extremely common, can be a harder task for LLMs than the hardest AoC problems.