3 ms·

My impression of the 2023 AoC was that Wastl has spent considerable effort on making it less LLM-friendly this year. Some days, this seems to have been done by

My impression of the 2023 AoC was that Wastl has spent considerable effort on making it less LLM-friendly this year. Some days, this seems to have been done by adding extra conditions and complications which make the task more difficult for LLMs to parse. Other tasks required studying the input data which is difficult to achieve with an unsupervised LLM. Finally, the first couple of days seemed a lot more difficult this year than previous years, possibly to deter chatgpt users from filling up the leaderboard right away (though this year December started with a weekend which could also be a contributing factor).

qsort 3y ago
> considerable effort on making it less LLM-friendly this year Wastl himself denied that this is the case[1]. This is a lie. > Other tasks required studying the input data Always been the case for AoC, how is that different from other years? We have a clear example of a set of tasks that state-of-the-art LLMs cannot perform. We are doing science, for once. Why do we need to get into full conspiracy mode? [1] https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
- lancebeet 3y ago
  I apologize. It was simply my subjective impression of the AoC this year, not a statement of objective fact. I didn't intend to insinuate that Wastl is a liar.
bnprks 3y ago
> Other tasks required studying the input data I also felt there were more problems than usual this year that could not easily be solved without looking at the input for special cases not alluded to in the problem descriptions. (As someone who has solved all 25 for the past 3 years). An extreme example was this year's day 20 circuit-simulating problem, which was made far easier by having the given circuit split up into a few independent chunks that are only connected at the start + end. (I suspect it might be NP-complete without this feature) It's a slightly different kind of problem solving to think "what makes this particular input easier than the general version of this problem", and one that I'd naively assume LLMs are less skilled at.
- smokel 3y ago
  There have been quite a few of these in the past. Some of the 2018 problems (e.g. day 21 [1]) required quite a lot of reverse engineering of programs in a custom instruction set. [1] https://adventofcode.com/2018/day/21 https://adventofcode.com/2018/day/21

qsort 3y ago

> considerable effort on making it less LLM-friendly this year Wastl himself denied that this is the case[1]. This is a lie. > Other tasks required studying the input data Always been the case for AoC, how is that different from other years? We have a clear example of a set of tasks that state-of-the-art LLMs cannot perform. We are doing science, for once. Why do we need to get into full conspiracy mode? [1] https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...

lancebeet 3y ago
I apologize. It was simply my subjective impression of the AoC this year, not a statement of objective fact. I didn't intend to insinuate that Wastl is a liar.

bnprks 3y ago

> Other tasks required studying the input data I also felt there were more problems than usual this year that could not easily be solved without looking at the input for special cases not alluded to in the problem descriptions. (As someone who has solved all 25 for the past 3 years). An extreme example was this year's day 20 circuit-simulating problem, which was made far easier by having the given circuit split up into a few independent chunks that are only connected at the start + end. (I suspect it might be NP-complete without this feature) It's a slightly different kind of problem solving to think "what makes this particular input easier than the general version of this problem", and one that I'd naively assume LLMs are less skilled at.

smokel 3y ago
There have been quite a few of these in the past. Some of the 2018 problems (e.g. day 21 [1]) required quite a lot of reverse engineering of programs in a custom instruction set. [1] https://adventofcode.com/2018/day/21 https://adventofcode.com/2018/day/21