We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For exam

We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217