I’m not sure. For any configuration of the model parameters, there is a straight line to many other parameter configurations that are arbitrarily “bad”. If you

I’m not sure. For any configuration of the model parameters, there is a straight line to many other parameter configurations that are arbitrarily “bad”. If you can find any set of input tokens that follows one of those directions then the model will behave badly. The problem is that you can’t really tell the model “don’t process tokens that move you in a bad direction” because it has to process the tokens to know the direction it is moving and it has to keep track of which _endpoints_ (not just directions) are bad. Otherwise we could do something the conjugate gradient steps to move towards where we want to go without ever stepping in that direction explicitly. So…I think that the problem is that the attack surface is huge, which means you’ll never be able to plug all the holes—and there’s likely no good function for easily determining the badness of a position or direction of change.