Without even actually retraining everything, I would be curious how different the tokens are with your technique compared to using an off-the-shelf solution lik

Without even actually retraining everything, I would be curious how different the tokens are with your technique compared to using an off-the-shelf solution like SentencePiece with the same output vocabulary size.