6 ms·

This guy seems to laugh away the fact that he gave the prompt in terribly broken and chunked up formats. I don’t think it’s surprising the model did poorly. May

This guy seems to laugh away the fact that he gave the prompt in terribly broken and chunked up formats. I don’t think it’s surprising the model did poorly. Maybe the contamination issue is true. But maybe it’s also true that the model does fine on novel codeforce problems when you don’t feed it a garbage prompt?

famouswaffles 4y ago
On code gen with some reflection baked in (ie feedback and memory), it shoout up to 88% (from 67%). There's some contamination to be sure but also still quite a lot to be done to better output
YeGoblynQueenne 4y ago
>> I don’t think it’s surprising the model did poorly. But it did poorly only on the problems it hadn't seen before. Was it prompted differently on one kind of problem, compared to the other?
- meh8881 4y ago
  But you can do a task you’ve done before with poor specification too. Sure, maybe it is contamination. But who cares? We only ought to judge the tool on its performance for carrying out good instructions.
contravariant 4y ago
Unless the formatting is somehow very different for pre 2021 problems it is still a strong signal that it is (up to a point) just parroting a solution it had heard somewhere rather than inferring it some way else. This is neither good nor bad, that will depend on what you want to use the model for.
- meh8881 4y ago
  That’s not my point. Maybe it is parroting a solution for something it has seen before, but still capable of writing solutions to problems it has not seen before when they’re well specified. The presence of contamination does not mean the general capability is intrinsically poor.
krackers 4y ago
The prompting might be an issue, but I think the larger sign that it's not quite there yet in terms of symbolic reasoning is that even based on their own paper, gpt-4 gets only a 30 on AMC 10 (out of 150), whereas leaving the test blank would get you a 37.5. And this is on a closed-ended multiple choice-test, so the conditions should be favorable for it to dominate. Edit: Although this might be unfair considering that LLMs are known to be poor at calculation. (Maybe it would do better at proof-style, like USAMO?). I wonder how well chatGPT with WA integration would do.
jprete 4y ago
The need for prompt engineering is an indicator of GPT failing the problem, not succeeding at it.
- bamboozled 4y ago
  The need for “tools” is a bit of a red flag too.
  - hughesjj 4y ago
    Why? I'm significantly less productive and competent without tools, and this applies to every human I've ever known.
- sdenton4 4y ago
  Like how the need for roads is an indication that cars don't solve transportation problems. People adapt to maximize the utility of their tools; always have, always will.
- meh8881 4y ago
  I disagree. We cannot expect these tools to work if we don’t communicate what we want from them. Can’t do that for humans either. We shouldn’t be confusing clear communication with “prompt engineering”.
tarruda 4y ago
Maybe the OP is also contaminated a bias of wanting to find errors in GPT-4? Finding flaws in GPT-4 and ignoring the fact that we are at the dawn of AI, that seems like a good remedy to calm the AI existential crisis anxiety nowadays.

famouswaffles 4y ago

On code gen with some reflection baked in (ie feedback and memory), it shoout up to 88% (from 67%). There's some contamination to be sure but also still quite a lot to be done to better output

YeGoblynQueenne 4y ago

>> I don’t think it’s surprising the model did poorly. But it did poorly only on the problems it hadn't seen before. Was it prompted differently on one kind of problem, compared to the other?

meh8881 4y ago
But you can do a task you’ve done before with poor specification too. Sure, maybe it is contamination. But who cares? We only ought to judge the tool on its performance for carrying out good instructions.

contravariant 4y ago

Unless the formatting is somehow very different for pre 2021 problems it is still a strong signal that it is (up to a point) just parroting a solution it had heard somewhere rather than inferring it some way else. This is neither good nor bad, that will depend on what you want to use the model for.

meh8881 4y ago
That’s not my point. Maybe it is parroting a solution for something it has seen before, but still capable of writing solutions to problems it has not seen before when they’re well specified. The presence of contamination does not mean the general capability is intrinsically poor.

krackers 4y ago

The prompting might be an issue, but I think the larger sign that it's not quite there yet in terms of symbolic reasoning is that even based on their own paper, gpt-4 gets only a 30 on AMC 10 (out of 150), whereas leaving the test blank would get you a 37.5. And this is on a closed-ended multiple choice-test, so the conditions should be favorable for it to dominate. Edit: Although this might be unfair considering that LLMs are known to be poor at calculation. (Maybe it would do better at proof-style, like USAMO?). I wonder how well chatGPT with WA integration would do.

jprete 4y ago

The need for prompt engineering is an indicator of GPT failing the problem, not succeeding at it.

bamboozled 4y ago
The need for “tools” is a bit of a red flag too.
- hughesjj 4y ago
  Why? I'm significantly less productive and competent without tools, and this applies to every human I've ever known.
sdenton4 4y ago
Like how the need for roads is an indication that cars don't solve transportation problems. People adapt to maximize the utility of their tools; always have, always will.
meh8881 4y ago
I disagree. We cannot expect these tools to work if we don’t communicate what we want from them. Can’t do that for humans either. We shouldn’t be confusing clear communication with “prompt engineering”.

tarruda 4y ago

Maybe the OP is also contaminated a bias of wanting to find errors in GPT-4? Finding flaws in GPT-4 and ignoring the fact that we are at the dawn of AI, that seems like a good remedy to calm the AI existential crisis anxiety nowadays.