5 ms·
Dunno if there's some trickery going on under the hood, but GPT-4 does basic arithmetic correctly.
by ravenstine 4y ago
Dunno if there's some trickery going on under the hood, but GPT-4 does basic arithmetic correctly.
- krainboltgreene 4y agoIt statistically outputs the next probable value of text. There is a lot of math related text in the CommonCrawl (likely the biggest source of it's training). That is all the company who owns it claims that it does. There is no calculation, therefor no basic arithmetic being done correctly.
- jackmott42 4y agoemergent behavior does happen with neural networks. It is correct to say that GPT 4 doesn't do math correctly, but it may be incorrect to say a large language model can't in principle.
- roflyear 4y agoIs there any evidence it can? It's not incorrect to say there is no God for example
- kgeist 4y agoIf it's only statistics, what is the neural network for?
- kgwgk 4y agoImplement the probabilistic model of token sequences and fit it to the training data? You could say the same for a character recognition system.
- deleted 4y ago[deleted]
- famouswaffles 4y agoIt's extremely easy to test arithmetic on random numbers it'll never have seen in the training set. GPT-4 can do arithmetic just fine.
- ResearchCode 4y agoAnd it will give you some random number with maybe the right order of magnitude. Doesn't seem to do arithmetic very fine at all actually.
- famouswaffles 4y agoYeah that's not true lol. It's arithmetic is not perfect (mostly multiplication, addition is fine) but there's nothing random about even the wrong/non-exact numbers
- krainboltgreene 4y agowe're talking about an llm dude how am i supposed to have a conversation about someone who is gassing up "not perfect [arithmetic]" (something a wrist watch from the 80's can do) and won't even believe what the creators of said machine say about how it works
- throwaway8503 4y agoThis is what I got on ChatGPT today. I assume it used GPT4: Prompt ChatGPT Actual Match 397,356 * 930,547 369,685,207,932 369,758,433,732 FALSE 36,330 * 26,951 979,458,630 979,129,830 FALSE 8,681 * 9,330 80,911,430 80,993,730 FALSE 278 * 903 250,734 251,034 FALSE 82 * 77 6,314 6,314 TRUE Edit: # of correct digits (counting from leftmost) only exceeds 3 on the smallest pair. It drops to two, as well, on the 3x3 set.
- optimalsolver 4y agoUnless you're using Plus (black icon), you're using GPT-3.5 (green icon).
- ogogmad 4y agoYou probably used GPT-3.5. That said, I didn't manage to get GPT-4 to calculate 36,330 * 26,951 correctly. I suggested casting out 9s, casting out 11s, doing long multiplication, reversing the digits - nothing. I have a theory that it does arithmetic badly because the logic goes right-to-left, when LLMs write left-to-right. If the digits were to be reversed, it might not make as many mistakes. I ran out of attempts before I could test this properly.
- IX-103 4y agoDid you adjust the prompt to ask it how a famous mathematician would answer the question? Or what a calculator would say the answer is? Sometimes LLMs get math wrong because people got math wrong on the training data and so they match the error frequency (https://learnprompting.org/docs/basics/roles https://learnprompting.org/docs/basics/roles).
- roflyear 4y agoYes that's exactly what people are saying here. It's not a criticism of the tool it's an example of what the tool is and how it functions.
- deleted 4y ago