> The accuracy largely comes from the fact that addition rarely requires carrying more than a single digit. So it's easy to pattern match from single digit prob

> The accuracy largely comes from the fact that addition rarely requires carrying more than a single digit. So it's easy to pattern match from single digit problems that it was previously trained on. Again, not at all true due to BPEs. [12] [65] [25] [42] [185] + [580] [22] [75] [80] [93] --------------------------- [706] [75] [300] [278] (note that GPT-3 is never told that the BPE [580] is composed of the digits [5], [8], and [0]. It has to guess this from the contexts [580] occurs in.) > With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits. You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and were forced to do it without any working out. Quick, what's 542983 * 39486? No writing anything down, you have to output the numbers in order, and a single wrong digit is a fail. (That's easy mode, I won't even bother asking you to do BPEs.) ML models can learn multiplication, obviously they can learn multiplication, they just can't do it in this absurd adversarial context. GPT-f[1] was doing Metamath proofs on 9-digit division (again, a vastly harder context, they involve ~10k proof steps) with 50-50 accuracy, and we have a toy proof of concept[2] for straight multiplication. [1] https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2009.03393 [2] https://github.com/Thopliterce/transformer-arithmetic https://github.com/Thopliterce/transformer-arithmetic