Transformers - and all other ML models - are ways to represent computer programs. You can think of them as a programming language designed to be easy for optimi

Transformers - and all other ML models - are ways to represent computer programs. You can think of them as a programming language designed to be easy for optimization instead of for human understanding. It's not clear to anybody exactly what kind of program structure LLMs have internally. Figuring that out is a major goal for the field of mechanistic interpretability.