We can do the same with LLMs. We can tease out some macro structure behaviors by looking at intermediate output from the feed forward network. But you see how

We can do the same with LLMs. We can tease out some macro structure behaviors by looking at intermediate output from the feed forward network. But you see how this is far from true understanding. It's like removing the WiFi module from a PC and you lose WiFi, therefore you know the WiFi module does WiFi stuff but in the end this action contributes nothing towards cohesive understanding of the PC.