5 ms·

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it fe

by lsy 3y ago

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-world stakes — taking a motorcycle is better than running for many transport scenarios. But the whole point of this is to have fun! Is it fun to hit <tab> over and over?

frankfrank13 3y ago
amazing analogy
mistrial9 3y ago
Lance Armstrong Foundation on the line for you...
- lgeorget 3y ago
  The incentives to cheat in a professional competition are more obvious since the money you earn is related to your performance. In the case of AoC, the only objective incentive I can think about is impressing a recruiter. Maybe it's enough if you're motivated enough.
  - HKH2 3y ago
    Why do people cheat in video games?
    - lgeorget 3y ago
      Good point. I play chess and some players end up thinking that cheating allows them to reach their real level and that it's just bad luck or conspiracy that prevent them to get to that level without resorting to cheating. Talk about cognitive dissonance.
    - datadrivenangel 3y ago
      Money. Being really good at a game can be a source of income for E-sports or streaming.
      - HKH2 3y ago
        Online games have always been rife with cheats.
HPsquared 3y ago
People can tell if you use a motorbike though. More like, performance enhancing drugs... or designing an anthropomorphic running robot that completes the race for you. Designing such a robot would be an achievement in its own right, but not really in the spirit of the competition.
- KeplerBoy 3y ago
  People can also tell if you submit within seconds.
- dghlsakjg 3y ago
  If you designed and built the robot, its one thing (still not right, but interesting). If you just rented some time on someone else's robot, its just plain cheating. I would love to see teams compete, in a separate venue/ranking, based on their own model's ability to write code with minimal prompting. It would be kinda cool to see what can be done in different 'classes' of model. How far can you get with a local model running on an 8gb card, or just using an M2 mac. How far can you get with a custom tuned chatgpt vs a programming specialized model. Etc...
  - michaelteter 3y ago
    > I would love to see teams compete, in a separate venue/ranking, based on their own model's ability to write code with minimal prompting. Yes, an AIdvent of Code would be interesting...
    - HPsquared 3y ago
      I wonder if, in future, there will be specific days and seasons to commemorate AI achievements. Like, if Wall-E was a real story would we have a holiday to honor him? Maybe even just observed by the machines / operators?
gedy 3y ago
There is a lot of people (and cultures) who crave status and clout, no matter the route. Cheaters in Ivy League colleges, etc.
Buttons840 3y ago
Or is it like a drag race where the first vehicle across the line is celebrated? There's two views: (1) programming as a craft and an art, a human creation, and (2) programming as a hurdle to overcome to accomplish more important things. People with both views might wish to practice.
yunwal 3y ago
> Is it fun to hit <tab> over and over? Last year, AI was only able to solve the first 3 or 4 problems, so it's really not like using AI takes all human ingenuity out of the equation. What it does is enable you to autocomplete sub-problems that are already solved problems so you can focus on the good stuff. I'd personally prefer if there were 2 leaderboards - one with AI, and one without. Motorcycle races are fun too :)
- deleted 3y ago
  [deleted]
- mckn1ght 3y ago
  I agree that two leaderboards would be a fine middle ground. Just put a checkbox next to the submission box.
reedf1 3y ago
I guess it's not really an apt analogy. It's more like if a the challenge was to get from city A to city B in the fastest time by any means necessary. People would run it until the horse was domesticated, and then people would ride them until the motorcycle or car was invented, and so on. That's not to say that you can't still do a foot race between A and B. It's just now you have to constrain it, it has become something different.
setgree 3y ago
> To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? Weirdly enough, basically this happens all the time at races [0]. Not a motorcycle, but a runner in my local competitive scene used a bicycle to fake her strava data for a HM in 2017 [1]. I don't get it either. [0] https://www.wired.com/story/marathon-investigation-cheaters-racing-data/ https://www.wired.com/story/marathon-investigation-cheaters-... [1] https://washingtonpost.com/news/morning-mix/wp/2017/02/23/how-an-ohio-blogger-discovered-a-runner-a-harvard-grad-cheating-in-a-half-marathon-across-the-county/ https://washingtonpost.com/news/morning-mix/wp/2017/02/23/ho...
- mckn1ght 3y ago
  Geez, the number of support requests I’ve sent to Strava over the years asking them to clean up their data. I don’t know about Android, but iOS can fairly reliably tell the difference between walking, running, bicycling, driving… They should be able to couple this with some better data science to filter things out like deliberate cheating, forgetting to set the right activity type or stop tracking before you drive away, or glitches in eg the GPS radio that result in huge changes in location or altitude, pushing you to the top of leaderboards with impossible results.
BananaaRepublik 3y ago
Because AoC has a leaderboard? Whenever you have a leaderboard of any kind, there will be people trying to get up there. It's also likely that a lot of AI users will be skilled developers too. They already know they can solve the problems, they also think they deserve the leaderboard rankings, so using AI is ok.
- HKH2 3y ago
  When everyone else is using steroids...
fatherzine 3y ago
Gamification is a powerful motivator. There is a leaderboard, climbing it is a reward in itself.
candiddevmike 3y ago
Why even have a leaderboard for this? At least make it anonymous.
- Jtsummers 3y ago
  It's optionally anonymous. People don't have to publicize their names on AoC. With Github (and I assume the other identity providers) you get an option of: anonymous #some number, your name (from Github), your username (from Github). IIRC, the anonymous option is the default.
mckn1ght 3y ago
> I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance This reminds me of the Project Euler position on people posting their solutions publicly. The assumption is others would seek them out, get credit they don’t deserve and not learn as much. I understand trying to prevent skewing the results, but personally it doesn’t affect me since I’m not doing it to compete in the website, I’m doing it for self edification. I believe this actually helps me compete in the thing that really matters, getting a job, bringing high quality products to market and generally solving real problems people have. Are LLMs starting to solve some of that? Sure. So then I want to compete for the jobs that still can’t be solved by them and involve more complicated and interesting solutions. If people want to handicap their problem solving abilities or compete for jobs supervising boilerplate generation, all the better for me. I’ll take a hopefully more rewarding career over making the leaderboard on AoC and PE. Though I do feel sorry for their maintainers. > Is it fun to hit <tab> over and over? It’s probably a mixture of entertainment (I’ve seen countless apps where it looks like people just tap the screen as fast as possible without even really watching what’s going on) and imagining they’re learning something by watching answers appear (although IMO the lack of engagement leads to lack of retention; sort of like Socrates’ position on reading vs discussion).
mikkqu 3y ago
Have you ever heard about cheaters in Counter Strike?
zeroCalories 3y ago
I think this depends entirely on the competition. Athletes "cheat" by taking ped, but it's still interesting to see how far they're able to take themselves. Similarly, if an AI assisted programmer is able to solve aoc's relatively complex problems at lightning speed that's interesting too. But if the comonm understanding is that people should solve it without AI, obviously I agree ot feels shitty to go against that.
golergka 3y ago
> To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Almost. But the crucial difference is — imagine an alternative universe, where humanity didn't have horses or any other animals or devices, even rickshas or wheels, to help moving around the world, and suddenly the first primitive cars were being invented just a couple of years ago. Wouldn't you feel enthusiastic to use those new devices and test them everywhere?
koromak 3y ago
Look at how many bots infest online multiplayer games. Some portion of humans get a kick out of cheating.
daniel-s 3y ago
Probably the same people that cheat at online videogames.
zelon88 3y ago
I agree with the sentiment, but I believe your motorcycle analogy is a bit flawed. Bringing a motorcycle to a 5K is obviously cheating. The motorcycle is a factor of 10 faster than its closest competitor. That is not where LLM/AI is currently at. The code generated by AI is not 10 times better than human code. It is just human code cleverly regurgitated. Comparing AI/LLM to the motorcycle is giving it too much credit. AI is more like a 500CC motorcycle vs a 300CC motorcycle. Sure they are pretty close, and the human might win once in a while, but the AI has just enough of an advantage that is guaranteed to burn the human out and win 95+% of the time.
empath-nirvana 3y ago
I tried solving last years AOC with chatgpt a few months ago and while it did just straight up produce solutions for the first 2 or 3 questions with minimal work on my part, after that, it required a lot of work to get it to a solution. It would have been faster to just write the code myself. I use copilot a lot and use chatgpt a lot to help with coding problems and doing things like writing short bash scripts for me, so I feel like I have a good grasp of what it's capable of doing, and the amount of work you'd have to do to break the problem into small enough chunks that it can understand and solve it is similar, if not more, than the amount of work you'd have to do to just solve it yourself. At least in its current state.
forgotpwd16 3y ago
It's just like cheating in online games. Heck people will cheat even on games with no ranking at stake.
michaelteter 3y ago
> mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance I think it defeats the purpose of the AoC to use AI, but I can imagine for some people it's a different game to see how they can use a screw (gaming term) to circumvent the system. On the other hand, there may be some value to learning how to apply AI tools as a solver. If test cases are available for a given problem space, and AI can be applied to build code to meet those "requirements", then it is probably a skill we should be learning. While I don't believe people are going to play fair here, it would be nice if AoC submissions had an [AI-assist] checkbox. Then there could be two leaderboards, and the AI folks could still have some good competition with each other. Ultimately, the AoC leaderboards are unfair as they favor people in timezones that fit the release of each "day". So being on the leaderboard isn't really something that many people can do, even if they are brilliant problem solvers.
precompute 3y ago
https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
Suzuran 3y ago
Because it means I won and everything else lost. "Got mine, fuck you!" Everything is a zero-sum game when you are sufficiently competitive.
prometheon1 3y ago
Are you saying we should not have motorcycle races? I agree that motorcyclists and runners should probably not compete in the same race, but there can be a competition where participants try to code the solution and another competition where participants try to build a machine that can code the solution at the same time.

by lsy 3y ago

frankfrank13 3y ago

amazing analogy

mistrial9 3y ago

Lance Armstrong Foundation on the line for you...

lgeorget 3y ago
The incentives to cheat in a professional competition are more obvious since the money you earn is related to your performance. In the case of AoC, the only objective incentive I can think about is impressing a recruiter. Maybe it's enough if you're motivated enough.
- HKH2 3y ago
  Why do people cheat in video games?
  - lgeorget 3y ago
    Good point. I play chess and some players end up thinking that cheating allows them to reach their real level and that it's just bad luck or conspiracy that prevent them to get to that level without resorting to cheating. Talk about cognitive dissonance.
  - datadrivenangel 3y ago
    Money. Being really good at a game can be a source of income for E-sports or streaming.
    - HKH2 3y ago
      Online games have always been rife with cheats.

HPsquared 3y ago

People can tell if you use a motorbike though. More like, performance enhancing drugs... or designing an anthropomorphic running robot that completes the race for you. Designing such a robot would be an achievement in its own right, but not really in the spirit of the competition.

KeplerBoy 3y ago
People can also tell if you submit within seconds.
dghlsakjg 3y ago
If you designed and built the robot, its one thing (still not right, but interesting). If you just rented some time on someone else's robot, its just plain cheating. I would love to see teams compete, in a separate venue/ranking, based on their own model's ability to write code with minimal prompting. It would be kinda cool to see what can be done in different 'classes' of model. How far can you get with a local model running on an 8gb card, or just using an M2 mac. How far can you get with a custom tuned chatgpt vs a programming specialized model. Etc...
- michaelteter 3y ago
  > I would love to see teams compete, in a separate venue/ranking, based on their own model's ability to write code with minimal prompting. Yes, an AIdvent of Code would be interesting...
  - HPsquared 3y ago
    I wonder if, in future, there will be specific days and seasons to commemorate AI achievements. Like, if Wall-E was a real story would we have a holiday to honor him? Maybe even just observed by the machines / operators?

gedy 3y ago

There is a lot of people (and cultures) who crave status and clout, no matter the route. Cheaters in Ivy League colleges, etc.

Buttons840 3y ago

Or is it like a drag race where the first vehicle across the line is celebrated? There's two views: (1) programming as a craft and an art, a human creation, and (2) programming as a hurdle to overcome to accomplish more important things. People with both views might wish to practice.

yunwal 3y ago

> Is it fun to hit <tab> over and over? Last year, AI was only able to solve the first 3 or 4 problems, so it's really not like using AI takes all human ingenuity out of the equation. What it does is enable you to autocomplete sub-problems that are already solved problems so you can focus on the good stuff. I'd personally prefer if there were 2 leaderboards - one with AI, and one without. Motorcycle races are fun too :)

deleted 3y ago
[deleted]
mckn1ght 3y ago
I agree that two leaderboards would be a fine middle ground. Just put a checkbox next to the submission box.

reedf1 3y ago

I guess it's not really an apt analogy. It's more like if a the challenge was to get from city A to city B in the fastest time by any means necessary. People would run it until the horse was domesticated, and then people would ride them until the motorcycle or car was invented, and so on. That's not to say that you can't still do a foot race between A and B. It's just now you have to constrain it, it has become something different.

setgree 3y ago

> To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? Weirdly enough, basically this happens all the time at races [0]. Not a motorcycle, but a runner in my local competitive scene used a bicycle to fake her strava data for a HM in 2017 [1]. I don't get it either. [0] https://www.wired.com/story/marathon-investigation-cheaters-racing-data/ https://www.wired.com/story/marathon-investigation-cheaters-... [1] https://washingtonpost.com/news/morning-mix/wp/2017/02/23/how-an-ohio-blogger-discovered-a-runner-a-harvard-grad-cheating-in-a-half-marathon-across-the-county/ https://washingtonpost.com/news/morning-mix/wp/2017/02/23/ho...

mckn1ght 3y ago
Geez, the number of support requests I’ve sent to Strava over the years asking them to clean up their data. I don’t know about Android, but iOS can fairly reliably tell the difference between walking, running, bicycling, driving… They should be able to couple this with some better data science to filter things out like deliberate cheating, forgetting to set the right activity type or stop tracking before you drive away, or glitches in eg the GPS radio that result in huge changes in location or altitude, pushing you to the top of leaderboards with impossible results.

BananaaRepublik 3y ago

Because AoC has a leaderboard? Whenever you have a leaderboard of any kind, there will be people trying to get up there. It's also likely that a lot of AI users will be skilled developers too. They already know they can solve the problems, they also think they deserve the leaderboard rankings, so using AI is ok.

HKH2 3y ago
When everyone else is using steroids...

fatherzine 3y ago

Gamification is a powerful motivator. There is a leaderboard, climbing it is a reward in itself.

candiddevmike 3y ago

Why even have a leaderboard for this? At least make it anonymous.

Jtsummers 3y ago
It's optionally anonymous. People don't have to publicize their names on AoC. With Github (and I assume the other identity providers) you get an option of: anonymous #some number, your name (from Github), your username (from Github). IIRC, the anonymous option is the default.

mckn1ght 3y ago

> I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance This reminds me of the Project Euler position on people posting their solutions publicly. The assumption is others would seek them out, get credit they don’t deserve and not learn as much. I understand trying to prevent skewing the results, but personally it doesn’t affect me since I’m not doing it to compete in the website, I’m doing it for self edification. I believe this actually helps me compete in the thing that really matters, getting a job, bringing high quality products to market and generally solving real problems people have. Are LLMs starting to solve some of that? Sure. So then I want to compete for the jobs that still can’t be solved by them and involve more complicated and interesting solutions. If people want to handicap their problem solving abilities or compete for jobs supervising boilerplate generation, all the better for me. I’ll take a hopefully more rewarding career over making the leaderboard on AoC and PE. Though I do feel sorry for their maintainers. > Is it fun to hit <tab> over and over? It’s probably a mixture of entertainment (I’ve seen countless apps where it looks like people just tap the screen as fast as possible without even really watching what’s going on) and imagining they’re learning something by watching answers appear (although IMO the lack of engagement leads to lack of retention; sort of like Socrates’ position on reading vs discussion).

mikkqu 3y ago

Have you ever heard about cheaters in Counter Strike?

zeroCalories 3y ago

I think this depends entirely on the competition. Athletes "cheat" by taking ped, but it's still interesting to see how far they're able to take themselves. Similarly, if an AI assisted programmer is able to solve aoc's relatively complex problems at lightning speed that's interesting too. But if the comonm understanding is that people should solve it without AI, obviously I agree ot feels shitty to go against that.

golergka 3y ago

> To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Almost. But the crucial difference is — imagine an alternative universe, where humanity didn't have horses or any other animals or devices, even rickshas or wheels, to help moving around the world, and suddenly the first primitive cars were being invented just a couple of years ago. Wouldn't you feel enthusiastic to use those new devices and test them everywhere?

koromak 3y ago

Look at how many bots infest online multiplayer games. Some portion of humans get a kick out of cheating.

daniel-s 3y ago

Probably the same people that cheat at online videogames.

zelon88 3y ago

I agree with the sentiment, but I believe your motorcycle analogy is a bit flawed. Bringing a motorcycle to a 5K is obviously cheating. The motorcycle is a factor of 10 faster than its closest competitor. That is not where LLM/AI is currently at. The code generated by AI is not 10 times better than human code. It is just human code cleverly regurgitated. Comparing AI/LLM to the motorcycle is giving it too much credit. AI is more like a 500CC motorcycle vs a 300CC motorcycle. Sure they are pretty close, and the human might win once in a while, but the AI has just enough of an advantage that is guaranteed to burn the human out and win 95+% of the time.

empath-nirvana 3y ago

I tried solving last years AOC with chatgpt a few months ago and while it did just straight up produce solutions for the first 2 or 3 questions with minimal work on my part, after that, it required a lot of work to get it to a solution. It would have been faster to just write the code myself. I use copilot a lot and use chatgpt a lot to help with coding problems and doing things like writing short bash scripts for me, so I feel like I have a good grasp of what it's capable of doing, and the amount of work you'd have to do to break the problem into small enough chunks that it can understand and solve it is similar, if not more, than the amount of work you'd have to do to just solve it yourself. At least in its current state.

forgotpwd16 3y ago

It's just like cheating in online games. Heck people will cheat even on games with no ranking at stake.

michaelteter 3y ago

> mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance I think it defeats the purpose of the AoC to use AI, but I can imagine for some people it's a different game to see how they can use a screw (gaming term) to circumvent the system. On the other hand, there may be some value to learning how to apply AI tools as a solver. If test cases are available for a given problem space, and AI can be applied to build code to meet those "requirements", then it is probably a skill we should be learning. While I don't believe people are going to play fair here, it would be nice if AoC submissions had an [AI-assist] checkbox. Then there could be two leaderboards, and the AI folks could still have some good competition with each other. Ultimately, the AoC leaderboards are unfair as they favor people in timezones that fit the release of each "day". So being on the leaderboard isn't really something that many people can do, even if they are brilliant problem solvers.

precompute 3y ago

https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

Suzuran 3y ago

Because it means I won and everything else lost. "Got mine, fuck you!" Everything is a zero-sum game when you are sufficiently competitive.

prometheon1 3y ago

Are you saying we should not have motorcycle races? I agree that motorcyclists and runners should probably not compete in the same race, but there can be a competition where participants try to code the solution and another competition where participants try to build a machine that can code the solution at the same time.