3 ms·

They do. For example, training transformers on addition creates circuits that perform modular addition. See "A Mechanistic Interpretability Analysis of Grokking

They do. For example, training transformers on addition creates circuits that perform modular addition. See "A Mechanistic Interpretability Analysis of Grokking"

sedatk 4y ago
My fascination just got doubled! Thank you for the reference.
mac01021 4y ago
Is training with dropout still a thing? Does that mess this up?