2 ms·
If you put it into the tokenizer https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer you can see that it helps in some places but not in
by sp332 3y ago
If you put it into the tokenizer https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer you can see that it helps in some places but not in others. It pulled out "SEA/OF/TRAN/QU/ILITY", but I think it broke up every instance of the word "THE".