5 ms·

In general Online Rounds of coding competitions are no longer going to work. As GPT-4 had been shown to outperform humans on the coding competition tasks. You

In general Online Rounds of coding competitions are no longer going to work. As GPT-4 had been shown to outperform humans on the coding competition tasks. You can have it assisted, but this quickly becomes a “who can afford more compute” competition. RIP coding competitions. Incidentally, World Finals: up to 2020, it was held in a location in the US or Europe, and has been online since. Pretty incredibly, Gennady Korotkevich has won the title every year since 2014 – except once.

nmca 4y ago
> GPT-4 had been shown to outperform humans on the coding competition tasks This is not true as of March 2023 (hard to prove a negative of course but look at the percentiles in the GPT4 report)
ajkjk 4y ago
> GPT-4 had been shown to outperform humans on the coding competition tasks citation?
- schrodinger 4y ago
  [flagged]
  - lovecg 4y ago
    I haven’t seen a clear answer on this actually - there was some confusion if the training data contained solutions for older problems, and if it underperforms on previously unseen ones. Plus it still seemed to underperform on medium and hard problems? Does anyone in the know have a summary on this?
  - nozzlegear 4y ago
    Sometimes things that have been discussed as possibilities on this forum begin to be repeated and taken as fact. One might say that this forum has hallucinated the capabilities of this bot beyond its actual capacity, and the hallucination grows daily. A citation is indeed needed.
  - camdenreslink 4y ago
    I think it has only been shown that ChatGPT can solve easy/some medium leetcode style questions. I doubt it can solve difficult problems that would give human competitive programmers a hard time (for now anyway...)
  - unethical_ban 4y ago
    GPT4 has been out for what, weeks? We aren't talking established theorems here.
  - ajkjk 4y ago
    Oh, I looked and didn't find anything. I thought about writing something about how I thought it wasn't true but decided it would be better to just ask if they had a reference for it. also, it's an extraordinary claim, so pretending like it's something everybody already knows is pretty weird.
- dragonwriter 4y ago
  Probably like most GPT-4 performance claims, the non-peer-reviewed scientifically-formatted paper with opaque methodology published as PR for GPT-4.
campbel 4y ago
Chess already has that problem and is still thriving. Something else is going on here.
lovecg 4y ago
In related news, all chess and Go competitions have been shutdown permanently.
krackers 4y ago
>As GPT-4 had been shown to outperform humans on the coding competition tasks On the same frontpage: https://news.ycombinator.com/item?id=35297067 https://news.ycombinator.com/item?id=35297067 Food for thought. Also by their own paper it has only a codeforces rank of 392 which is below 5th percentile.
- krackers 4y ago
  Also a cute prompt I saw on the same twitter thread ``` This is a variant of the monty hall problem: Suppose you're on a game show, and you're given the choice of three doors. IMPORTANT: All the doors are transparent. Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who clearly knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice? ``` The response from chatGPT (v3, non plus) is: ``` Yes, it is to your advantage to switch your choice. In this variant of the Monty Hall problem, the fact that the doors are transparent does not change the underlying probability of the problem. ``` I wonder how gpt-4 does. There does seem to be a bit of overfitting, but I consider the prompt fair enough and even called out the trick in capital letters, so that even a tired human should be able to get the answer. Edit: An even more obvious variant of the prompt is seen in https://news.ycombinator.com/item?id=35192466 https://news.ycombinator.com/item?id=35192466, which goes further and spells out that the contestant explicitly picks the door with the car. ChatGPT still gets it wrong.
  - TacticalCoder 4y ago
    > I wonder how gpt-4 does. The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to train the next versions and are going to be part of the dataset. So GPT-4 or GPT-5 may get the answer right, but that still wouldn't mean anything.
    - fastball 4y ago
      Fairly certain that is not what is going on here. GPT-4 seems genuinely better at reasoning and harder to trick from my testing.
milemi 4y ago
I’m sure it outperforms the general population since most people can’t code a hello world, or regurgitate an answer to a problem they have been trained to answer but have no understanding of like ChatGPT can. But if a minimally competent human gave me a completely nonsensical answer to a question they haven’t seen before, the way ChatGPT does so confidently, I would think one of us had a stroke. I would expect a journalist who never had do think through a theory of computation course to make breathless claims that ChatGPT can “solve programming problems“, but I’m pretty surprised to hear so many people who have jobs in tech repeating these claims, especially since all it would take them to trip up ChatGPT is a few seconds to type in a slightly unfamiliar or non-trivial question. It’s like a kind of second-order Turing test: if you think ChatGPT can program, you’re not a real programmer.
iliekcomputers 4y ago
chess has had engines 100x better than magnus carlsen for years and it's not dead. people who have fun giving these competitions will continue having fun, while people who don't will keep crying that they're useless or dead or whatever.
- KeplerBoy 4y ago
  That requires a clear ruleset. In chess it's very clear where to draw the line: no help allowed, you're on your own with your brain. Where should one draw the line in Coding competitions? No ChatGPT? I guess most people would agree. No Copilot? Same as ChatGPT as the products seem to converge. No Googling? That too will converge to something close to ChatGPT. If you can't look up information during a contest anymore, it comes down to memorization instead of problem solving. I am afraid the concept of coding concepts is dead indeed.
  - nitwit005 4y ago
    There have, of course, already been coding competitions that didn't allow any sort of reference, ones that only allowed reference you brought with you, etc. And, honestly, plenty of people do have enough memorized to do this stuff.
  - devit 4y ago
    In-person competitions simply don't allow any Internet access, you have to use their own machines and you can't bring any data or printed material or electronic device.
    - KeplerBoy 4y ago
      That's not the experience i had at past events. Everyone brought their own devices and solved the problems with whatever tool they deemed suitable. Actually I'm attending another such event in a few days and expect to see a lot of ChatGPT sessions used with varying levels of success.
devit 4y ago
GPT-4 is competely incapable of solving any advanced problem in coding or mathematics competitions, and usually doesn't even appear to correctly understand the problem statement (assuming the solutions are not in the training set, of course). Just try submitting the IOI 2022 and IMO 2022 problems. Obviously it still outperforms the average human, since the average human has no knowledge of mathematics or computer science whatsoever.

nmca 4y ago

> GPT-4 had been shown to outperform humans on the coding competition tasks This is not true as of March 2023 (hard to prove a negative of course but look at the percentiles in the GPT4 report)

ajkjk 4y ago

> GPT-4 had been shown to outperform humans on the coding competition tasks citation?

schrodinger 4y ago
[flagged]
- lovecg 4y ago
  I haven’t seen a clear answer on this actually - there was some confusion if the training data contained solutions for older problems, and if it underperforms on previously unseen ones. Plus it still seemed to underperform on medium and hard problems? Does anyone in the know have a summary on this?
- nozzlegear 4y ago
  Sometimes things that have been discussed as possibilities on this forum begin to be repeated and taken as fact. One might say that this forum has hallucinated the capabilities of this bot beyond its actual capacity, and the hallucination grows daily. A citation is indeed needed.
- camdenreslink 4y ago
  I think it has only been shown that ChatGPT can solve easy/some medium leetcode style questions. I doubt it can solve difficult problems that would give human competitive programmers a hard time (for now anyway...)
- unethical_ban 4y ago
  GPT4 has been out for what, weeks? We aren't talking established theorems here.
- ajkjk 4y ago
  Oh, I looked and didn't find anything. I thought about writing something about how I thought it wasn't true but decided it would be better to just ask if they had a reference for it. also, it's an extraordinary claim, so pretending like it's something everybody already knows is pretty weird.
dragonwriter 4y ago
Probably like most GPT-4 performance claims, the non-peer-reviewed scientifically-formatted paper with opaque methodology published as PR for GPT-4.

campbel 4y ago

Chess already has that problem and is still thriving. Something else is going on here.

lovecg 4y ago

In related news, all chess and Go competitions have been shutdown permanently.

krackers 4y ago

>As GPT-4 had been shown to outperform humans on the coding competition tasks On the same frontpage: https://news.ycombinator.com/item?id=35297067 https://news.ycombinator.com/item?id=35297067 Food for thought. Also by their own paper it has only a codeforces rank of 392 which is below 5th percentile.

krackers 4y ago
Also a cute prompt I saw on the same twitter thread ``` This is a variant of the monty hall problem: Suppose you're on a game show, and you're given the choice of three doors. IMPORTANT: All the doors are transparent. Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who clearly knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice? ``` The response from chatGPT (v3, non plus) is: ``` Yes, it is to your advantage to switch your choice. In this variant of the Monty Hall problem, the fact that the doors are transparent does not change the underlying probability of the problem. ``` I wonder how gpt-4 does. There does seem to be a bit of overfitting, but I consider the prompt fair enough and even called out the trick in capital letters, so that even a tired human should be able to get the answer. Edit: An even more obvious variant of the prompt is seen in https://news.ycombinator.com/item?id=35192466 https://news.ycombinator.com/item?id=35192466, which goes further and spells out that the contestant explicitly picks the door with the car. ChatGPT still gets it wrong.
- TacticalCoder 4y ago
  > I wonder how gpt-4 does. The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to train the next versions and are going to be part of the dataset. So GPT-4 or GPT-5 may get the answer right, but that still wouldn't mean anything.
  - fastball 4y ago
    Fairly certain that is not what is going on here. GPT-4 seems genuinely better at reasoning and harder to trick from my testing.

milemi 4y ago

I’m sure it outperforms the general population since most people can’t code a hello world, or regurgitate an answer to a problem they have been trained to answer but have no understanding of like ChatGPT can. But if a minimally competent human gave me a completely nonsensical answer to a question they haven’t seen before, the way ChatGPT does so confidently, I would think one of us had a stroke. I would expect a journalist who never had do think through a theory of computation course to make breathless claims that ChatGPT can “solve programming problems“, but I’m pretty surprised to hear so many people who have jobs in tech repeating these claims, especially since all it would take them to trip up ChatGPT is a few seconds to type in a slightly unfamiliar or non-trivial question. It’s like a kind of second-order Turing test: if you think ChatGPT can program, you’re not a real programmer.

iliekcomputers 4y ago

chess has had engines 100x better than magnus carlsen for years and it's not dead. people who have fun giving these competitions will continue having fun, while people who don't will keep crying that they're useless or dead or whatever.

KeplerBoy 4y ago
That requires a clear ruleset. In chess it's very clear where to draw the line: no help allowed, you're on your own with your brain. Where should one draw the line in Coding competitions? No ChatGPT? I guess most people would agree. No Copilot? Same as ChatGPT as the products seem to converge. No Googling? That too will converge to something close to ChatGPT. If you can't look up information during a contest anymore, it comes down to memorization instead of problem solving. I am afraid the concept of coding concepts is dead indeed.
- nitwit005 4y ago
  There have, of course, already been coding competitions that didn't allow any sort of reference, ones that only allowed reference you brought with you, etc. And, honestly, plenty of people do have enough memorized to do this stuff.
- devit 4y ago
  In-person competitions simply don't allow any Internet access, you have to use their own machines and you can't bring any data or printed material or electronic device.
  - KeplerBoy 4y ago
    That's not the experience i had at past events. Everyone brought their own devices and solved the problems with whatever tool they deemed suitable. Actually I'm attending another such event in a few days and expect to see a lot of ChatGPT sessions used with varying levels of success.

devit 4y ago

GPT-4 is competely incapable of solving any advanced problem in coding or mathematics competitions, and usually doesn't even appear to correctly understand the problem statement (assuming the solutions are not in the training set, of course). Just try submitting the IOI 2022 and IMO 2022 problems. Obviously it still outperforms the average human, since the average human has no knowledge of mathematics or computer science whatsoever.