BPE is complete. Every valid Unicode string can be encoded with any BPE tokenizer. BPE basically starts with a token for every valid value for a Unicode byte

BPE is complete. Every valid Unicode string can be encoded with any BPE tokenizer. BPE basically starts with a token for every valid value for a Unicode byte and then creates new tokens by looking at common pairs of bytes (‘t’ followed by ‘h’ becomes a new token ’th’)