6 ms·

A Mechanistic Interpretability Analysis of Grokking

202 points by famouswaffles 3y ago

lsy 3y ago
To me the odd takeaway here is that the model eventually converges to a close approximation but never really lands at the core of the problem. Loss never reaches zero even for the memorized set, and the "grokked" loss is usually higher. Wouldn't you expect something totally bounded like "addition mod 113" to be trainable to the point of zero loss? I think the field is looking for that final phase change that takes us from statistical methods to actual understanding.
wwarner 3y ago
I really believe phase changes in models are going to tell us a lot about what they’ve learned. Though it has to be said that seeing that sharp improvement in a model during training is the exception rather than the rule — normally you just see gradual improvement until you’ve reached sufficient accuracy and then decide to stop training. So it’s not clear to me that every highly accurate model will contain a phase change.
- iNic 3y ago
  We know that models like GPT-4 go through phase changes, not in train/test loss, but for loss on specific tasks. This suggests the gradient descent is learning one thing after another, possibly in order of "difficulty".
airgapstopgap 3y ago
Not really a characterization of this work (I like mechanistic interpretability even though I consider it a detour in the long run), but of the submission title (edit: it was "Tiny Transformer trained for addition learns bizarre addition algorithm" at the point of this comment being posted). It's bad. I suspect it's informed by Rob Miles' characterization of this algorithm as "bizarre and inhuman"[1] AI safety/alignment discourse as a whole is so incredibly bad. It's autodidactic and weird in the bad sense, weird like people who think they are brave iconoclasts but just haven't the curiosity and humility to learn the basics of the field are weird. Add some money from Dustin Moskovitz to Open Philanthropy to Robert Miles' propaganda firehose on youtube, add policy lobbying and buying bankrupt crypto podcasters, and you have the worst that could come from the culture of public intellectualism. The notions they have inserted into this potentially vital field, like inner/outer misalignment and "mesaoptimization" [2] (where simple overfitting is the real and more productive explanation) are deeply misleading, technically illiterate and stoking public fears in directions we should care about the least, siphoning resources and attention from real issues informed by specifics of contemporary ML (chiefly data and curriculum engineering). They're close to putting a lid on LLM development – because they're getting spooked by some vague sense of growing "capabilities"; even though LLMs are evidently a vastly better approach to safe-yet-capable-general-AI than anything any of their thought leaders have ever come up with in decades of SIAI/MIRI/Lesswrong procrastination. (For one example of a decent takedown of their approach I recommend [3]). Why is this algorithm bizarre, or even inhuman? Could anyone show me the specific functional connectivity graph and activation functions in my brain that I use when doing addition, with the circuit going through spatial-quantitative cortex and phonological loop, populations querying cached results in long-term memory, individual numbers marked as interesting or banal by the highest-level associative units? Would it look anything like a neat and sensible algorithm we'd come up with using a global top-down representation of the solution target space? By this standard, humans are inhuman, humans are shoggoths, as they should be, because emergent data-driven structures in general-purpose substrate generally do not resolve to anything like provable optimality, even if they get very close in performance. This framing adds nothing but clicks and insecurity. 1. https://twitter.com/robertskmiles/status/1663534255249453056 https://twitter.com/robertskmiles/status/1663534255249453056 2. https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI 3. https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky#The_difficulty_of_alignment https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objecti...
- comp_throw7 3y ago
  How can the title of the submission (which is just the title of the article) be informed by Miles' description of _the same article_ (which he posted today, nearly a year after the article was posted)?
  - famouswaffles 3y ago
    I didn't personally change it but the title of the post was different when i originally submitted. Didn't call it inhuman though, just bizarre which it is...depending on your frame of reference.
  - sp332 3y ago
    "Tiny Transformer trained for addition learns bizarre addition algorithm" was the title.
- famouswaffles 3y ago
  I don't think being bizarre or not adds much to the fear of alignment. Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process. There isn't really any solace in the other option (surprisingly human) in terms of alignment fears. Let me put it this way. Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal. So personally, "Human SuperIntelligence" doesn't exactly warm the heart.
  - airgapstopgap 3y ago
    > Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process I think this is just uninspected, vague intuition. What does it mean to control the learning process? No, we control the data, the deterministic learning rule, weights and activations, every step of the way is controllable – it's just it's intractable to control it all. Moreover, if it were tractable, would we know what to do? How would that be different from the problem of programming an AI from scratch? > There isn't really any solace in the other option (surprisingly human) in terms of alignment fears. If there weren't, I suppose major AI doomers (Yudkowsky, Leahy, Besinger etc.) wouldn't have been putting such an emphasis in their Lovecraft-inspired rhetoric on the alienness and inscrutability of those evil matrices of floating point numbers. No, the idea that human mental architecture is safer (and that DL does not approximate it) is very much at the core of alignment fears. E.g. Besinger on why he expects AGI Ruin [1]: "We're building "AI" in the sense of building powerful general search processes (and search processes for search processes), not building "AI" in the sense of building friendly ~humans but in silicon … The key differences between humans and "things that are more easily approximated as random search processes than as humans-plus-a-bit-of-noise" lies in lots of complicated machinery in the human brain. … This doesn't mean the problem is unsolvable; but it means that you either need to reproduce that internal machinery, in a lot of detail, in AI, or you need to build some new kind of machinery that’s safe for reasons other than the specific reasons humans are safe." > Let me put it this way. Unaligned intelligence is dangerous. I would ask you not to condescend but I have learned that this is an impossible request with the AI risk crowd, because you never really encounter pushback in your para-academic hothouse and so come to believe that, indeed, pretty basic intuitions are knockdown arguments only silly people could dismiss. You define intelligence in a certain way that has nothing to do with how you identify it in the wild. The definition is something like "intelligence is an optimization process"; the thing under consideration can be an LLM or a diffusion model that seems really good at its job. You mean to say "processes shaping real-world outcomes that are not optimizing for outcomes people consider good are dangerous". This is true but trivial. The onus is on you to tie this consideration to intelligence in general or to "capabilities" of arbitrarily powerful ML models. 1. https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-reasons-i-expect-agi-ruin https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-...
sukilot 3y ago
Implemented addition via: cos(wx) sin (wy) + sin(wx) cos(wy) But how did it compute that intermediate addition?
- furyofantares 3y ago
  What it learned to do was modular addition, not addition.
siddiqi123 3y ago
You don't talk about human analogs of grokking, and that makes sense for a technical paper like this. Nonetheless, grokking also seems to happen in humans, and everybody has had "Aha!" moments before. Can you maybe comment a bit on the relation to human learning? It seems clear that human grokking is not a process that purely depends on the number of training samples seen but also on the availability of hypotheses. People grok faster if you provide them with symbolic descriptions of what goes on. What are your thoughts on the representation and transfer of the resulting structure, e.g., via language/token streams?
astrange 3y ago
I think people should read at least slightly reputable ML papers instead of AI papers by crazy autodidactic AI experts who are making up all of their terms and won't stop telling you it's going to take over the world because of some math they did once.
- nschiefer 3y ago
  Good news for readers who prefer reputable ML papers: this work was accepted to ICLR 2023! https://openreview.net/forum?id=9XFSbDPmdW https://openreview.net/forum?id=9XFSbDPmdW
  - skinner_ 3y ago
    Thanks! What a self-own, ICLR Spotlight paper, that's close to the top of the reputability food chain!
- rain1 3y ago
  tell me you're posting from an armchair without telling me you're posting from an armchair
Solvency 3y ago
Why has the term grokking been formalized like this? Honest question. Why can't we stick with normal words like "learning" or "understanding"? It just seems like a slow creep of needless semantic complexity in an already esoteric and niche space.
- dkarl 3y ago
  I'm going to guess that in AI those terms have been so cheapened by now that they are useless.
- urthor 3y ago
  Because if you've spent 80 hours a week researching, you want some fun in the 85th hour of your work week spent writing the research up.
  - twic 3y ago
    A stronger hypothesis: the main goal of most AI researchers is to make other AI researchers think they are cool, and using a word from classic science fiction helps do that.
    - bmacho 3y ago
      That seems obviously false to me, according to what I know about people in general, and researchers in general.
- tomrod 3y ago
  Grokking comes from Heinlein, and it is more than a synonym for both of those words. Good intro here: https://en.m.wikipedia.org/wiki/Grok https://en.m.wikipedia.org/wiki/Grok
- ska 3y ago
  The term "grokking" has been used for something like "deep understanding" for many decades now, after it's introduction in an early 1960's novel. Perhaps not falling into completely general use, but popular within a large swath of technical communities, certainly. I'm not sure it's particularly well targeted here, but they are going for something deeper than "learning" which already has a standard usage in the field.
adamnemecek 3y ago
I have been looking into this as well. I'm strongly convinced that Chu duality or similar (i.e. the duality between states and transition between states is at play). At first, the model learns only the states but over time it learns the transformations. Ironically, this is a similar process to QTF renormalization.
joe_the_user 3y ago
Here, "Grokking" seems to be defined as a quick shift in the degree of generalization of a model. I'm not sure this is in any way an interesting phenomena. The original paper doesn't seem to be referenced [1] and I'd speculate the interest in the "alignment" community. [1] https://scholar.google.com/scholar?cites=4466239674951044045&as_sdt=2005&sciodt=0,5&hl=en https://scholar.google.com/scholar?cites=4466239674951044045...
newhouseb 3y ago
The algorithm learned here actually makes a lot of sense when you spend more time understanding how transformers typically work. Namely: once you include layer normalization your model is more or less forced to find ways to represent absolute quantities in a way that won't be normalized away and a great way to achieve this is to... store things as rotations of a unit tensor! With that as your primitive, it's fairly natural to rotate around a circle to compute modular addition. I'd be curious to explore if a different algorithm is learned if one were to stop normalizing at various points. I wouldn't be surprised if a large hurdle to mechanistic interpretability turns out to be that the models have learned complicated rotations in non-obvious coordinate spaces that are tricky to identify after the fact.
- sebzim4500 3y ago
  In the model he was using he didn't use layer norm, right?
  - newhouseb 3y ago
    Oh good catch! The author defines a layer norm layer but then... comments it out in the actual implementation (I missed the fact that it was commented out). So that answers my second question of what happens without it. Anecdotally in my own interpretability work (without layer norm), my models also learn rotations fairly frequently. I attributed this to the way I was doing positional embeddings (as rotations), but perhaps there's more to it.
    - newhouseb 3y ago
      Thinking about this more, softmax is also a form of normalization that could likely contribute to this phenomenon.
ggm 3y ago
Repurposing language to apply it as analogous labels is bad. As a term of art it really has specific meaning. Casual reading outside the cognoscenti will misunderstand it to have literal value as "hallucinating" and "learning" already are. Don't do this. Grokking does not imply intelligence or even understanding. There is no necessary introspection or inductive quality.
- chaxor 3y ago
  Sure ok, there's a bit of namespace conflict now, happens quite a lot. I also don't love it, but it's there now, and it's really not the worst name. It's a good area though, and the paper seems interesting. Do you have any real problems with it?
  - ggm 3y ago
    My only complaint is that it adds to the set of core concept-labels which imply things about AI which aren't correct.
    - cubefox 3y ago
      Probably most words for which the origin is known started out as a metaphor. Examples: hallucinate - mid 17th century (in the sense ‘be deceived, have illusions’): from Latin hallucinat- ‘gone astray in thought’, from the verb hallucinari, from Greek alussein ‘be uneasy or distraught’. induction - late Middle English: from Latin inductio(n- ), from the verb inducere ‘lead into’ Wikipedia: > Metaphor: Change based on similarity between concepts, e.g., mouse "rodent" → "computer device". > In historical onomasiology or in historical linguistics, a metaphor is defined as a semantic change based on a similarity in form or function between the original concept and the target concept named by a word. > In cognitive linguistics, conceptual metaphor, or cognitive metaphor, refers to the understanding of one idea, or conceptual domain, in terms of another. An example of this is the understanding of quantity in terms of directionality (e.g. "the price of peace is rising") or the understanding of time in terms of money (e.g. "I spent time at work today").
abrax3141 3y ago
Possibly related: “Observation of Phase Transitions in Spreading Activation Networks” (Science, 1987) https://drive.google.com/file/d/1axt_8KfP2wd2p2cwMt5IGEfaekUhX-c4 https://drive.google.com/file/d/1axt_8KfP2wd2p2cwMt5IGEfaekU...
YeGoblynQueenne 3y ago
Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model. There's no reason to get excited until results generalise to larger models and real-world architectures, trained on real-world datasets. Unfortunately this is a lot to ask because the "mechanistic interpretability" approach doesn't seem to have the power to analyse such mega-structures. So I don't think anyone has learned anything about "grokking" from the paper: those who want to believe, will still believe, those who want evidence, will not find evidence. I'm more interested in the claims of algorithm learning. There's an obvious test that the authros haven't tried: hand-code the algorithm they reverse-engineered and compare it to the trained model. To clarify, the authors claim that they reverse engineered a certain algorithm to perform modular arithmetic off the weights of their trained neural net that was showing 100% performance on their training set. The algorithm, which they call the "Fourrier multiplication algorithm" is outlined in figure 1 and further elucidated in Section 3.1. The algorithm goes over some frequencies, five of which seem to be predictive of most of the model's performance. So both the algorithm and its parameters are known (or anyway assumed). In that case, it should be possible to hand-code the algorithm and run it [2]. Then, given some set of inputs X, taken from the experiments' test set, first a) test the algorithm's accuracy and then b) compare the algorithm's output to the outputs of the model, given the same inputs. The point of this is to check, respectively, that a) the proposed algorithm is a correct addition algorithm (and that it is an algorithm, i.e. it always terminates) and b) that its behaviour matches exactly the behaviour of the model it is claimed to be encoded in. Note that the authors base their certainty on the correctness of the algorithm, and its fidelity as an algorithmic encoding of their model, to a series of ablation studies. It's genuinely refreshing to observe this almost extinct species in the wild, but it is not enough. Not when there is a stronger test available (the one I point out above). I'm almost disappointed to see an ablation study, because it looks like an excuse to not go the whole hog. In the same spirit, what is entirely missing is a proof of correctness of the algorithm. I have never seen this algorithm before and eyballing it, it "looks OK", but that's not how algorithmic correctness is normally ascertained. There's all this maths in the paper, but not a proof in sight. What's the point of maths then, to calculate large numbers? _________________________________ [1] The paper was accepted for publication in ICLR 2023 so congratulations to the authors. [2] To be perfectly clear about what I mean: I mean it should be possible to code the algorithm as a computer program in a language like, e.g., Python. The "algorithm" is an encoding of the trained model in a different, but equivalent form. It's now possible to encode it in yet another different, but equivalent, form, and run it, to perform "inference".
boerseth 3y ago
Generally speaking, a phase transition occurs when the symmetries of a system are forced to change, either becoming restricted or liberated, as the result of some parameter changing, perhaps past some critical value. The parameter can be a number of things, like density (pushing more particles into a box until the gas becomes more of a liquid) or temperature (like removing the total kinetic energy of the particles in a liquid until they can no longer move around eachother). In the examples of the article, however, the phase transitions occur as the result of duration of training. Time is not a thermodynamic state variable, so these are not phase transitions in the strict sense of the word. However, I think it would be interesting to see the same experiment with something like stochastic gradient descent or the Metropolis algorithm, using some control parameter T to gradually reduce the randomness. This might let us find critical values of T for specific problems, which is useful because that is the value at which you want to spend the most time. It may also be that for more involved problems, grokking is not achievable by naive gradient descent, as the landscape may be fraught with local minima. Perhaps, in order to find the way to the global minimum, to anneal the system into the lowest free-energy state, it might even be necessary to use a temperature-like control parameter, and mimic nature's own optimisation algorithm.
- boerseth 3y ago
  Thinking about it more thoroughly, I am pretty convinced that the author's conclusion is correct. At the start, the neural net does not encode anything like an FFT or clever modulo arithmetic. The phase space of similarly-behaved weight-values is quite huge, when the neural net is not very sophisticated. But then eventually gradient descent finds its way into an implementation of FFT and all the rest, which solves the problem exactly. The phase space of weight-values that exhibit this behaviour is far, far smaller than the phase space I described in the previous paragraph. This means that the set of symmetries has been changed, which is the definition of a phase transition. The reason we see a first-order (sudden) phase transition is perhaps because of the size of the neural net being trained. Perhaps there aren't enough parameters available to encode a half-assed solution, and so there is no continuous transition from clueless to grokked. It either "gets it" or it doesn't.
deleted 3y ago
[deleted]

202 points by famouswaffles 3y ago

lsy 3y ago

To me the odd takeaway here is that the model eventually converges to a close approximation but never really lands at the core of the problem. Loss never reaches zero even for the memorized set, and the "grokked" loss is usually higher. Wouldn't you expect something totally bounded like "addition mod 113" to be trainable to the point of zero loss? I think the field is looking for that final phase change that takes us from statistical methods to actual understanding.

wwarner 3y ago

I really believe phase changes in models are going to tell us a lot about what they’ve learned. Though it has to be said that seeing that sharp improvement in a model during training is the exception rather than the rule — normally you just see gradual improvement until you’ve reached sufficient accuracy and then decide to stop training. So it’s not clear to me that every highly accurate model will contain a phase change.

iNic 3y ago
We know that models like GPT-4 go through phase changes, not in train/test loss, but for loss on specific tasks. This suggests the gradient descent is learning one thing after another, possibly in order of "difficulty".

airgapstopgap 3y ago

Not really a characterization of this work (I like mechanistic interpretability even though I consider it a detour in the long run), but of the submission title (edit: it was "Tiny Transformer trained for addition learns bizarre addition algorithm" at the point of this comment being posted). It's bad. I suspect it's informed by Rob Miles' characterization of this algorithm as "bizarre and inhuman"[1] AI safety/alignment discourse as a whole is so incredibly bad. It's autodidactic and weird in the bad sense, weird like people who think they are brave iconoclasts but just haven't the curiosity and humility to learn the basics of the field are weird. Add some money from Dustin Moskovitz to Open Philanthropy to Robert Miles' propaganda firehose on youtube, add policy lobbying and buying bankrupt crypto podcasters, and you have the worst that could come from the culture of public intellectualism. The notions they have inserted into this potentially vital field, like inner/outer misalignment and "mesaoptimization" [2] (where simple overfitting is the real and more productive explanation) are deeply misleading, technically illiterate and stoking public fears in directions we should care about the least, siphoning resources and attention from real issues informed by specifics of contemporary ML (chiefly data and curriculum engineering). They're close to putting a lid on LLM development – because they're getting spooked by some vague sense of growing "capabilities"; even though LLMs are evidently a vastly better approach to safe-yet-capable-general-AI than anything any of their thought leaders have ever come up with in decades of SIAI/MIRI/Lesswrong procrastination. (For one example of a decent takedown of their approach I recommend [3]). Why is this algorithm bizarre, or even inhuman? Could anyone show me the specific functional connectivity graph and activation functions in my brain that I use when doing addition, with the circuit going through spatial-quantitative cortex and phonological loop, populations querying cached results in long-term memory, individual numbers marked as interesting or banal by the highest-level associative units? Would it look anything like a neat and sensible algorithm we'd come up with using a global top-down representation of the solution target space? By this standard, humans are inhuman, humans are shoggoths, as they should be, because emergent data-driven structures in general-purpose substrate generally do not resolve to anything like provable optimality, even if they get very close in performance. This framing adds nothing but clicks and insecurity. 1. https://twitter.com/robertskmiles/status/1663534255249453056 https://twitter.com/robertskmiles/status/1663534255249453056 2. https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI 3. https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky#The_difficulty_of_alignment https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objecti...

comp_throw7 3y ago
How can the title of the submission (which is just the title of the article) be informed by Miles' description of _the same article_ (which he posted today, nearly a year after the article was posted)?
- famouswaffles 3y ago
  I didn't personally change it but the title of the post was different when i originally submitted. Didn't call it inhuman though, just bizarre which it is...depending on your frame of reference.
- sp332 3y ago
  "Tiny Transformer trained for addition learns bizarre addition algorithm" was the title.
famouswaffles 3y ago
I don't think being bizarre or not adds much to the fear of alignment. Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process. There isn't really any solace in the other option (surprisingly human) in terms of alignment fears. Let me put it this way. Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal. So personally, "Human SuperIntelligence" doesn't exactly warm the heart.
- airgapstopgap 3y ago
  > Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process I think this is just uninspected, vague intuition. What does it mean to control the learning process? No, we control the data, the deterministic learning rule, weights and activations, every step of the way is controllable – it's just it's intractable to control it all. Moreover, if it were tractable, would we know what to do? How would that be different from the problem of programming an AI from scratch? > There isn't really any solace in the other option (surprisingly human) in terms of alignment fears. If there weren't, I suppose major AI doomers (Yudkowsky, Leahy, Besinger etc.) wouldn't have been putting such an emphasis in their Lovecraft-inspired rhetoric on the alienness and inscrutability of those evil matrices of floating point numbers. No, the idea that human mental architecture is safer (and that DL does not approximate it) is very much at the core of alignment fears. E.g. Besinger on why he expects AGI Ruin [1]: "We're building "AI" in the sense of building powerful general search processes (and search processes for search processes), not building "AI" in the sense of building friendly ~humans but in silicon … The key differences between humans and "things that are more easily approximated as random search processes than as humans-plus-a-bit-of-noise" lies in lots of complicated machinery in the human brain. … This doesn't mean the problem is unsolvable; but it means that you either need to reproduce that internal machinery, in a lot of detail, in AI, or you need to build some new kind of machinery that’s safe for reasons other than the specific reasons humans are safe." > Let me put it this way. Unaligned intelligence is dangerous. I would ask you not to condescend but I have learned that this is an impossible request with the AI risk crowd, because you never really encounter pushback in your para-academic hothouse and so come to believe that, indeed, pretty basic intuitions are knockdown arguments only silly people could dismiss. You define intelligence in a certain way that has nothing to do with how you identify it in the wild. The definition is something like "intelligence is an optimization process"; the thing under consideration can be an LLM or a diffusion model that seems really good at its job. You mean to say "processes shaping real-world outcomes that are not optimizing for outcomes people consider good are dangerous". This is true but trivial. The onus is on you to tie this consideration to intelligence in general or to "capabilities" of arbitrarily powerful ML models. 1. https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-reasons-i-expect-agi-ruin https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-...

sukilot 3y ago

Implemented addition via: cos(wx) sin (wy) + sin(wx) cos(wy) But how did it compute that intermediate addition?

furyofantares 3y ago
What it learned to do was modular addition, not addition.

siddiqi123 3y ago

You don't talk about human analogs of grokking, and that makes sense for a technical paper like this. Nonetheless, grokking also seems to happen in humans, and everybody has had "Aha!" moments before. Can you maybe comment a bit on the relation to human learning? It seems clear that human grokking is not a process that purely depends on the number of training samples seen but also on the availability of hypotheses. People grok faster if you provide them with symbolic descriptions of what goes on. What are your thoughts on the representation and transfer of the resulting structure, e.g., via language/token streams?

astrange 3y ago

I think people should read at least slightly reputable ML papers instead of AI papers by crazy autodidactic AI experts who are making up all of their terms and won't stop telling you it's going to take over the world because of some math they did once.

nschiefer 3y ago
Good news for readers who prefer reputable ML papers: this work was accepted to ICLR 2023! https://openreview.net/forum?id=9XFSbDPmdW https://openreview.net/forum?id=9XFSbDPmdW
- skinner_ 3y ago
  Thanks! What a self-own, ICLR Spotlight paper, that's close to the top of the reputability food chain!
rain1 3y ago
tell me you're posting from an armchair without telling me you're posting from an armchair

Solvency 3y ago

Why has the term grokking been formalized like this? Honest question. Why can't we stick with normal words like "learning" or "understanding"? It just seems like a slow creep of needless semantic complexity in an already esoteric and niche space.

dkarl 3y ago
I'm going to guess that in AI those terms have been so cheapened by now that they are useless.
urthor 3y ago
Because if you've spent 80 hours a week researching, you want some fun in the 85th hour of your work week spent writing the research up.
- twic 3y ago
  A stronger hypothesis: the main goal of most AI researchers is to make other AI researchers think they are cool, and using a word from classic science fiction helps do that.
  - bmacho 3y ago
    That seems obviously false to me, according to what I know about people in general, and researchers in general.
tomrod 3y ago
Grokking comes from Heinlein, and it is more than a synonym for both of those words. Good intro here: https://en.m.wikipedia.org/wiki/Grok https://en.m.wikipedia.org/wiki/Grok
ska 3y ago
The term "grokking" has been used for something like "deep understanding" for many decades now, after it's introduction in an early 1960's novel. Perhaps not falling into completely general use, but popular within a large swath of technical communities, certainly. I'm not sure it's particularly well targeted here, but they are going for something deeper than "learning" which already has a standard usage in the field.

adamnemecek 3y ago

I have been looking into this as well. I'm strongly convinced that Chu duality or similar (i.e. the duality between states and transition between states is at play). At first, the model learns only the states but over time it learns the transformations. Ironically, this is a similar process to QTF renormalization.

joe_the_user 3y ago

Here, "Grokking" seems to be defined as a quick shift in the degree of generalization of a model. I'm not sure this is in any way an interesting phenomena. The original paper doesn't seem to be referenced [1] and I'd speculate the interest in the "alignment" community. [1] https://scholar.google.com/scholar?cites=4466239674951044045&as_sdt=2005&sciodt=0,5&hl=en https://scholar.google.com/scholar?cites=4466239674951044045...

newhouseb 3y ago

The algorithm learned here actually makes a lot of sense when you spend more time understanding how transformers typically work. Namely: once you include layer normalization your model is more or less forced to find ways to represent absolute quantities in a way that won't be normalized away and a great way to achieve this is to... store things as rotations of a unit tensor! With that as your primitive, it's fairly natural to rotate around a circle to compute modular addition. I'd be curious to explore if a different algorithm is learned if one were to stop normalizing at various points. I wouldn't be surprised if a large hurdle to mechanistic interpretability turns out to be that the models have learned complicated rotations in non-obvious coordinate spaces that are tricky to identify after the fact.

sebzim4500 3y ago
In the model he was using he didn't use layer norm, right?
- newhouseb 3y ago
  Oh good catch! The author defines a layer norm layer but then... comments it out in the actual implementation (I missed the fact that it was commented out). So that answers my second question of what happens without it. Anecdotally in my own interpretability work (without layer norm), my models also learn rotations fairly frequently. I attributed this to the way I was doing positional embeddings (as rotations), but perhaps there's more to it.
  - newhouseb 3y ago
    Thinking about this more, softmax is also a form of normalization that could likely contribute to this phenomenon.

ggm 3y ago

Repurposing language to apply it as analogous labels is bad. As a term of art it really has specific meaning. Casual reading outside the cognoscenti will misunderstand it to have literal value as "hallucinating" and "learning" already are. Don't do this. Grokking does not imply intelligence or even understanding. There is no necessary introspection or inductive quality.

chaxor 3y ago
Sure ok, there's a bit of namespace conflict now, happens quite a lot. I also don't love it, but it's there now, and it's really not the worst name. It's a good area though, and the paper seems interesting. Do you have any real problems with it?
- ggm 3y ago
  My only complaint is that it adds to the set of core concept-labels which imply things about AI which aren't correct.
  - cubefox 3y ago
    Probably most words for which the origin is known started out as a metaphor. Examples: hallucinate - mid 17th century (in the sense ‘be deceived, have illusions’): from Latin hallucinat- ‘gone astray in thought’, from the verb hallucinari, from Greek alussein ‘be uneasy or distraught’. induction - late Middle English: from Latin inductio(n- ), from the verb inducere ‘lead into’ Wikipedia: > Metaphor: Change based on similarity between concepts, e.g., mouse "rodent" → "computer device". > In historical onomasiology or in historical linguistics, a metaphor is defined as a semantic change based on a similarity in form or function between the original concept and the target concept named by a word. > In cognitive linguistics, conceptual metaphor, or cognitive metaphor, refers to the understanding of one idea, or conceptual domain, in terms of another. An example of this is the understanding of quantity in terms of directionality (e.g. "the price of peace is rising") or the understanding of time in terms of money (e.g. "I spent time at work today").

abrax3141 3y ago

Possibly related: “Observation of Phase Transitions in Spreading Activation Networks” (Science, 1987) https://drive.google.com/file/d/1axt_8KfP2wd2p2cwMt5IGEfaekUhX-c4 https://drive.google.com/file/d/1axt_8KfP2wd2p2cwMt5IGEfaekU...

YeGoblynQueenne 3y ago

Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model. There's no reason to get excited until results generalise to larger models and real-world architectures, trained on real-world datasets. Unfortunately this is a lot to ask because the "mechanistic interpretability" approach doesn't seem to have the power to analyse such mega-structures. So I don't think anyone has learned anything about "grokking" from the paper: those who want to believe, will still believe, those who want evidence, will not find evidence. I'm more interested in the claims of algorithm learning. There's an obvious test that the authros haven't tried: hand-code the algorithm they reverse-engineered and compare it to the trained model. To clarify, the authors claim that they reverse engineered a certain algorithm to perform modular arithmetic off the weights of their trained neural net that was showing 100% performance on their training set. The algorithm, which they call the "Fourrier multiplication algorithm" is outlined in figure 1 and further elucidated in Section 3.1. The algorithm goes over some frequencies, five of which seem to be predictive of most of the model's performance. So both the algorithm and its parameters are known (or anyway assumed). In that case, it should be possible to hand-code the algorithm and run it [2]. Then, given some set of inputs X, taken from the experiments' test set, first a) test the algorithm's accuracy and then b) compare the algorithm's output to the outputs of the model, given the same inputs. The point of this is to check, respectively, that a) the proposed algorithm is a correct addition algorithm (and that it is an algorithm, i.e. it always terminates) and b) that its behaviour matches exactly the behaviour of the model it is claimed to be encoded in. Note that the authors base their certainty on the correctness of the algorithm, and its fidelity as an algorithmic encoding of their model, to a series of ablation studies. It's genuinely refreshing to observe this almost extinct species in the wild, but it is not enough. Not when there is a stronger test available (the one I point out above). I'm almost disappointed to see an ablation study, because it looks like an excuse to not go the whole hog. In the same spirit, what is entirely missing is a proof of correctness of the algorithm. I have never seen this algorithm before and eyballing it, it "looks OK", but that's not how algorithmic correctness is normally ascertained. There's all this maths in the paper, but not a proof in sight. What's the point of maths then, to calculate large numbers? _________________________________ [1] The paper was accepted for publication in ICLR 2023 so congratulations to the authors. [2] To be perfectly clear about what I mean: I mean it should be possible to code the algorithm as a computer program in a language like, e.g., Python. The "algorithm" is an encoding of the trained model in a different, but equivalent form. It's now possible to encode it in yet another different, but equivalent, form, and run it, to perform "inference".

boerseth 3y ago

Generally speaking, a phase transition occurs when the symmetries of a system are forced to change, either becoming restricted or liberated, as the result of some parameter changing, perhaps past some critical value. The parameter can be a number of things, like density (pushing more particles into a box until the gas becomes more of a liquid) or temperature (like removing the total kinetic energy of the particles in a liquid until they can no longer move around eachother). In the examples of the article, however, the phase transitions occur as the result of duration of training. Time is not a thermodynamic state variable, so these are not phase transitions in the strict sense of the word. However, I think it would be interesting to see the same experiment with something like stochastic gradient descent or the Metropolis algorithm, using some control parameter T to gradually reduce the randomness. This might let us find critical values of T for specific problems, which is useful because that is the value at which you want to spend the most time. It may also be that for more involved problems, grokking is not achievable by naive gradient descent, as the landscape may be fraught with local minima. Perhaps, in order to find the way to the global minimum, to anneal the system into the lowest free-energy state, it might even be necessary to use a temperature-like control parameter, and mimic nature's own optimisation algorithm.

boerseth 3y ago
Thinking about it more thoroughly, I am pretty convinced that the author's conclusion is correct. At the start, the neural net does not encode anything like an FFT or clever modulo arithmetic. The phase space of similarly-behaved weight-values is quite huge, when the neural net is not very sophisticated. But then eventually gradient descent finds its way into an implementation of FFT and all the rest, which solves the problem exactly. The phase space of weight-values that exhibit this behaviour is far, far smaller than the phase space I described in the previous paragraph. This means that the set of symmetries has been changed, which is the definition of a phase transition. The reason we see a first-order (sudden) phase transition is perhaps because of the size of the neural net being trained. Perhaps there aren't enough parameters available to encode a half-assed solution, and so there is no continuous transition from clueless to grokked. It either "gets it" or it doesn't.

deleted 3y ago

[deleted]