I bet that tweaking the positional embedding to add an explicit token role indication plus some careful training to help the model learn to use it would make a

I bet that tweaking the positional embedding to add an explicit token role indication plus some careful training to help the model learn to use it would make a big difference.