4 ms·
Poor performance is more likely due to how transformer neural networks view numbers. It memorises them like words instead of modeling their numerical structure.
by williamtrask 5y ago
Poor performance is more likely due to how transformer neural networks view numbers. It memorises them like words instead of modeling their numerical structure. Thus even if it’s seen the number 3456 and 3458, it knows nothing of 3457. Totally different embedding.
It’s like a kid memorising a multiplication table instead of learning the more general principle of multiplication (related: this illusion is why big models are so popular. Memorise more stuff.)
Paper (NeurIPS/DeepMind): https://arxiv.org/abs/1808.00508 https://arxiv.org/abs/1808.00508
- eutectic 5y agoThat depends on the tokenization scheme.
- plutonorm 5y agoIt's recently been shown that even though the numbers are represented with different tokens, the network learns to form an internal representation that understands the progression from one token to the next.
- nikolayasdf123 5y agoThe idea that each number has to be inside ones Brain or Neural Network or Token is plainly wrong. Network has to grasp the "abstract" number, but it clearly did not grasp that concept.
- Isinlor 5y agoHow would you test if it grasped the concept?
- catach 5y agoBeing able to extrapolate to numbers that were not in the training set, perhaps? At least that'd be a basic part of the requirement.
- Isinlor 5y agoSure: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 https://arxiv.org/abs/2201.04600 (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com/ http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 https://arxiv.org/abs/2201.02177 Both of these models can generalize to numbers it have not seen.
- deleted 5y ago[deleted]
- YeGoblynQueenne 5y agoAs far as I can tell from a quick heuristic perusal, the "Generalization Beyond Overfitting" paper reports "generalisation" _on the validation set_. That's not particularly impressive and it's not particularly "generalisation" either. Actually, I really don't grokk this (if I may). I often see deep learning work reporting generalisation on the validation set. What's up with that? Why is generalisation on the validation set more interesting than on the test set, let alone OOD data?
- Isinlor 5y agoThe point of the paper is to show that NN can still learn long after fully memorizing the train dataset. This behavior goes against current paradigm of thinking about training NNs. It is just very unexpected, similarly as double descent is unexpected from classical statistics point of view that more parameters lead to more over-fitting. They could have split validation test set into validation and test sets, but I don't know what that would achieve in their case. Fig. 1 center shows different train / validate splits. Fig 2. shows a swoop between different optimization algorithms if you are concerned about hyperparameters over-fitting. But to me really interesting is the Fig 3. that shows that NN learned the structure of the problem.
- YeGoblynQueenne 5y ago
- plutonorm 5y agohttps://arxiv.org/pdf/2201.02177.pdf https://arxiv.org/pdf/2201.02177.pdf This paper shows fairly conclusively that the network 'groks' modular addition.
- pfortuny 5y agoModulo 97. This is what it is. Not "general arithmetic".
- Isinlor 5y agoTake a look at this paper: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 https://arxiv.org/abs/2201.04600 If you look at embedding visualization it is very clear that the model learns order of numbers. (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com/ http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) There is also: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 https://arxiv.org/abs/2201.02177 Again, looking at visualizations the model very clearly grasps the structure of the function it models.
- pfortuny 5y agoModulo 97 (the arxiv paper). That is what they do. It is quite easy to grok operations modulo 97.
- YeGoblynQueenne 5y agoThe "Deep Symbolic Regression" paper reports very poor generalisation results that break off after a small n (where n is the number of tokens in the predicted sequence). It works some of the time for n = 1 (predicts the next token) but accuracy drops off for n = 10. No results are reported for N > 10 as far as I can tell in the "Out of Domain Generalization" section (which is the meat and potatoes of the "generalization" claim). tl;dr they can sometimes generalise to the next 1 to 10 tokens (digits or operators), but no more. This kind of short-term "generalisation" on OOD data is standard in neural nets trying to approximate symbolic regressions or things like grammars etc as far as I know. I do like they use 'Out of Domain" rather than "Out of Distribution" as a target though. That makes more sense.
- Isinlor 5y agoI don't think you will find any human that will extrapolate sequence generated with more than 10 operators. And longer input sequences are actually easier to handle - fig 1. the right most graph. If you think you can do better than their program then: Seq1: [0, 1, 2, 3, 6, 7, 13, 26, 32, 58, 116, 142, 258, 516] Seq2: [2, 2, 3, 5, 10, 12, 22, 44, 54, 98, 196, 240, 436, 872] Seq3: [3, 1, 8, 9, 18, 19, 37, 74, 92, 166, 332, 406, 738, 1476] Their program is able to guess correct continuation with one more sequence element. SHA1 hash for verification: bef5e213340f91258b3b9a0042c9c083dd91cb80
- nicholast 5y agoThe cool thing about math applications is just how easy it would be to generate synthetic data. That these large language models haven't attempted to supplement their gigabytes+ scale data sets with such is an oversight.
- williamtrask 5y agoOr you could just use a 50cent calculator. Note, you’d need to train such a model on data teaching it about the relationship of every number to every other number when run through every function. Yes, infinite synthetic data, but you’re just memorising stiff you can already generate
- not2b 5y agoOr build a model that has "peripherals". Oh, I'm being asked to do math. Let's put it in my calculator app. Everything doesn't have to be in one uniform network. Evidently the brain works that way: the cortex is built on top of older components, so it doesn't have to figure out basic metabolism the same way it has to learn to identify people.
- moffkalast 5y ago> this illusion is why big models are so popular. Memorise more stuff It's all just a compressed lookup table that can handle in-betweens.