3 ms·
You can input the space-separated strings into OpenAI's tokenizer: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You'll see that
by rcme 3y ago
You can input the space-separated strings into OpenAI's tokenizer: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
You'll see that it does properly tokenize each binary digital as a separate token.
But again, the tokenization doesn't matter. I could say to you, please add these two binary numbers:
101 0011 11001
+ 00 11 10 100011
And you could do it.