4 ms·

OpenAI already benchmarks their GPTs on leetcode problems and even includes a Codeforces rating. It is not impressive at all and there's almost no progress from

OpenAI already benchmarks their GPTs on leetcode problems and even includes a Codeforces rating. It is not impressive at all and there's almost no progress from GPT 2 to 4. I agree, why does this grade school math problem matter if the model can't solve problems that are very precisely stated and have a very narrow solution space (at least more narrow than some vague natural language instruction)?

bcherry 3y ago
I suspect the truth (if this claim is true) is a lot more nuanced than "it did grade-school math", and there's more context surrounding this claim which insiders have and makes it much more interesting.
- bhy 3y ago
  Like most research they likely started with a smaller model like GPT 2 or 3 and shown that they can significantly boost the performance to the extent of solving grade school math.
maxdoop 3y ago
Are you saying GPT-4 is not much different than GPT2?
- vjerancrnjak 3y ago
  Leetcode and codeforces. AlphaCode is much better.
sweezyjeezy 3y ago
Here is something that I think would be a big breakthrough: I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt. Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.
- margorczynski 3y ago
  > It actually learns this to update its weights - not just uses it as part of a prompt. In a way both are the same thing - memory that is in a feedback loop with the network that does the calculation. Just that the weights give much faster access, no "serde". Maybe the goal is not to modify the weights but train the network so that it can effectively use a "memory block" in the way it works. Now this is in a way faked by re-feeding the output it produces concatenated with the original phrase. Don't we as humans effectively extend our memory by using all kind of text, written or digital? Just the issue is that it is slow to utilize, for a computer using fast RAM that wouldn't be much of an issue.
deleted 3y ago
[deleted]

bcherry 3y ago

I suspect the truth (if this claim is true) is a lot more nuanced than "it did grade-school math", and there's more context surrounding this claim which insiders have and makes it much more interesting.

bhy 3y ago
Like most research they likely started with a smaller model like GPT 2 or 3 and shown that they can significantly boost the performance to the extent of solving grade school math.

maxdoop 3y ago

Are you saying GPT-4 is not much different than GPT2?

vjerancrnjak 3y ago
Leetcode and codeforces. AlphaCode is much better.

sweezyjeezy 3y ago

Here is something that I think would be a big breakthrough: I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt. Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.

margorczynski 3y ago
> It actually learns this to update its weights - not just uses it as part of a prompt. In a way both are the same thing - memory that is in a feedback loop with the network that does the calculation. Just that the weights give much faster access, no "serde". Maybe the goal is not to modify the weights but train the network so that it can effectively use a "memory block" in the way it works. Now this is in a way faked by re-feeding the output it produces concatenated with the original phrase. Don't we as humans effectively extend our memory by using all kind of text, written or digital? Just the issue is that it is slow to utilize, for a computer using fast RAM that wouldn't be much of an issue.

deleted 3y ago

[deleted]