The problem is that transformers have a state space which includes attention over their input. Thus, it's very hard to manipulate. I think you're right, but it

The problem is that transformers have a state space which includes attention over their input. Thus, it's very hard to manipulate. I think you're right, but it will require moving to more recurrent architectures.