I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on tw

I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on two symbols with an operation '+'. Of course it can generate code to add two numbers, because the code to add two numbers exists in its training data. What transformers seem to be good at is generalizing over the training data. Where they seem to perform poorly is data outside of the training set. Binary numbers are interesting because you can add a bit of entropy just by increasing the size of your binary number, so it's very easy to generate examples outside of the training data. You also see this, albeit less directly, when trying to ask it to generate code that wasn't part of its training data. E.g. https://twitter.com/cHHillee/status/1635790330854526981 https://twitter.com/cHHillee/status/1635790330854526981