>These are not the same things. You mean because the token ids are in a different order? This is irrelevant to a transformer model, there is no inductive bias

>These are not the same things. You mean because the token ids are in a different order? This is irrelevant to a transformer model, there is no inductive bias that similar tokens should have related token ids. >Interestingly, the code FB provided doesn't seem to have any special handling for digits Yeah, token handling is done in sentencepiece.