5 ms·
An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout re
by agapron1 4y ago
An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results.
Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.
- nine_k 4y agoA sequential one-bit adder, like in pocket calculators or earliest computers with delay-line memory, could likely be invented in the process.
- Jensson 4y agoYeah, the current solution is similar to overfitting, this wont generalize to harder math where the operation doesn't correspond to the activation function of the network.
- rileymat2 4y agoIs this true? If it can add, it can probably subtract? If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents? I don’t know, but cannot jump to your conclusion without much more domain knowledge.
- eru 4y ago> If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents? It was just a simple feed-forward network. It can't do arbitrary amounts of repeated addition (nor repeat any other operation arbitrarily often).
- sebzim4500 4y agoNone of these require arbitrary amounts of repeated addition though. E.g. multiplying two 8 bit numbers requires at most 7 additions.
- eru 4y agoAre you doing the equivalent of repeated squaring here? Otherwise, you'd need up to 255 (or so) additions to multiply two 8 bit numbers, I think?
- adwn 4y agoAn 8x8 bit multiplication only requires 7 additions, either in parallel or sequentially. Remember long-form multiplication? [1] It's the same principle. Of course, high-speed digital multiplication circuits use a much more optimized, much more complex implementation. [1] https://en.wikipedia.org/wiki/Multiplication_algorithm#Example https://en.wikipedia.org/wiki/Multiplication_algorithm#Examp...
- eru 4y agoOK. It sounded to me like the comment I originally replied to (https://news.ycombinator.com/item?id=34400296 https://news.ycombinator.com/item?id=34400296) suggested to implement multiplication as naive repeated addition.
- rileymat2 4y agoYeah I did not mean a loop, repeated in the network.
- im3w1l 4y agoRepeated additions may be hard, but computing a*b as exp(log(a)+log(b)) should be learnable.
- torginus 4y agoA 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.
- GTP 4y agoYes, but the adder that the NN came up with has no carry.
- amelius 4y agoCarry is just the 9th bit?
- GTP 4y agoNot really, to compose adders you need specific lines for carry.
- deleted 4y ago[deleted]
- stabbles 4y agoIt's better to say the NN "converged to" something than "came up with" something.
- canadianfella 4y agoWhy?
- stabbles 4y agoThere is no intelligence or reasoning involved in stochastic gradient descent
- sebzim4500 4y agoIt's not harder computationally but finding that solution with SGD might be hard. I'd be really interested in the results.
- ape4 4y agoIts converted the input into its native language, in a way