Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW h

Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model. There's no reason to get excited until results generalise to larger models and real-world architectures, trained on real-world datasets. Unfortunately this is a lot to ask because the "mechanistic interpretability" approach doesn't seem to have the power to analyse such mega-structures. So I don't think anyone has learned anything about "grokking" from the paper: those who want to believe, will still believe, those who want evidence, will not find evidence. I'm more interested in the claims of algorithm learning. There's an obvious test that the authros haven't tried: hand-code the algorithm they reverse-engineered and compare it to the trained model. To clarify, the authors claim that they reverse engineered a certain algorithm to perform modular arithmetic off the weights of their trained neural net that was showing 100% performance on their training set. The algorithm, which they call the "Fourrier multiplication algorithm" is outlined in figure 1 and further elucidated in Section 3.1. The algorithm goes over some frequencies, five of which seem to be predictive of most of the model's performance. So both the algorithm and its parameters are known (or anyway assumed). In that case, it should be possible to hand-code the algorithm and run it [2]. Then, given some set of inputs X, taken from the experiments' test set, first a) test the algorithm's accuracy and then b) compare the algorithm's output to the outputs of the model, given the same inputs. The point of this is to check, respectively, that a) the proposed algorithm is a correct addition algorithm (and that it is an algorithm, i.e. it always terminates) and b) that its behaviour matches exactly the behaviour of the model it is claimed to be encoded in. Note that the authors base their certainty on the correctness of the algorithm, and its fidelity as an algorithmic encoding of their model, to a series of ablation studies. It's genuinely refreshing to observe this almost extinct species in the wild, but it is not enough. Not when there is a stronger test available (the one I point out above). I'm almost disappointed to see an ablation study, because it looks like an excuse to not go the whole hog. In the same spirit, what is entirely missing is a proof of correctness of the algorithm. I have never seen this algorithm before and eyballing it, it "looks OK", but that's not how algorithmic correctness is normally ascertained. There's all this maths in the paper, but not a proof in sight. What's the point of maths then, to calculate large numbers? _________________________________ [1] The paper was accepted for publication in ICLR 2023 so congratulations to the authors. [2] To be perfectly clear about what I mean: I mean it should be possible to code the algorithm as a computer program in a language like, e.g., Python. The "algorithm" is an encoding of the trained model in a different, but equivalent form. It's now possible to encode it in yet another different, but equivalent, form, and run it, to perform "inference".