4 ms·
I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond si
by Nasrudith 6y ago
I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond simple input/output memorization. Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.
- dwohnitmok 6y ago> Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser. I suspect this is how many humans do arithmetic (especially considering how many people conflate numbers with their representation as digits). So if GPT-3 is doing that, that's pretty impressive.
- ja3k 6y agoYou don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.
- gwern 6y agoYou do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.
- ralfd 6y agoBy the way: Arithmetic accuracy is better if dollar sign and commas are added (financial data in the training set): http://gptprompts.wikidot.com/logic:math http://gptprompts.wikidot.com/logic:math