I do not think it is possible to guess model behavior by just looking at the weights. My feeling is that this may even have a mathematical proof. Your thought

I do not think it is possible to guess model behavior by just looking at the weights. My feeling is that this may even have a mathematical proof. Your thought experiment is not really the same: it would require the model to be self-conscious, to understand when it expresses a behavior contradicting the expectations and to measure those patterns in vivo (what btw is neuron activation in case of AI model?)