3 ms·

It also completely ignores the developments around GPT tool use. If you’re evaluating the AI field as a whole, why are you restricting your analysis to the raw

by valine 3y ago

It also completely ignores the developments around GPT tool use. If you’re evaluating the AI field as a whole, why are you restricting your analysis to the raw output of GPTs? Arguably the most impressive display of intelligence exhibited by these models is their ability to use tools like calculators to augment their own abilities. It doesn’t matter if the model can’t do three digit multiplication without access to a calculator. Most people can’t mentally multiply large numbers either.

rcme 3y ago
Most people can do 32-bit binary addition using a pen and paper. And the point of using addition as an illustrative example is that the rules of addition are extremely simple, yet a binary addition problem can easily be scaled to N bits of entropy. This means it's very easy to show that the system fails to apply simple logic rules in such cases.
- carrolldunham 3y ago
  It didn't fail logic, it failed to read your string. You didn't even notice that the terms being added in the 'show working' version don't match the given strings. And you are elsewhere talking about adding spaces to make the digits 'tokens'... you have no idea what you are talking about at all.
  - rcme 3y ago
    You can input the space-separated strings into OpenAI's tokenizer: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You'll see that it does properly tokenize each binary digital as a separate token. But again, the tokenization doesn't matter. I could say to you, please add these two binary numbers: 101 0011 11001 + 00 11 10 100011 And you could do it.
- valine 3y ago
  If your goal is to simulate a human doing addition with pen and paper it only seems fair to give the GPT read/write access to a text file. Outputting the answer directly from the model is much more akin to mental math.
  - rcme 3y ago
    ChatGPT does have read write access to a text file. It's allowed to show its work as part of its output, which it can write tokens to, and then read those tokens back as part of generating the next tokens.

by valine 3y ago

rcme 3y ago

Most people can do 32-bit binary addition using a pen and paper. And the point of using addition as an illustrative example is that the rules of addition are extremely simple, yet a binary addition problem can easily be scaled to N bits of entropy. This means it's very easy to show that the system fails to apply simple logic rules in such cases.

carrolldunham 3y ago
It didn't fail logic, it failed to read your string. You didn't even notice that the terms being added in the 'show working' version don't match the given strings. And you are elsewhere talking about adding spaces to make the digits 'tokens'... you have no idea what you are talking about at all.
- rcme 3y ago
  You can input the space-separated strings into OpenAI's tokenizer: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You'll see that it does properly tokenize each binary digital as a separate token. But again, the tokenization doesn't matter. I could say to you, please add these two binary numbers: 101 0011 11001 + 00 11 10 100011 And you could do it.
valine 3y ago
If your goal is to simulate a human doing addition with pen and paper it only seems fair to give the GPT read/write access to a text file. Outputting the answer directly from the model is much more akin to mental math.
- rcme 3y ago
  ChatGPT does have read write access to a text file. It's allowed to show its work as part of its output, which it can write tokens to, and then read those tokens back as part of generating the next tokens.