I’m no expert, but as long as they’re represented by tokens in the end, they’re just tokens. Even if you train the transformer to treat them specially, a token

I’m no expert, but as long as they’re represented by tokens in the end, they’re just tokens. Even if you train the transformer to treat them specially, a token is a token, and there’s no free lunch. At best, you’re going to be trading off between paying attention to this would-be security boundary and delivering high-quality results; the more you focus on one, the more you lose on the other.