3 ms·
You have to represent spaces in some way (you want to make a distinction between therapist and the rapist), different tokenizers do it differently - one option
by PeterisP 3y ago
You have to represent spaces in some way (you want to make a distinction between therapist and the rapist), different tokenizers do it differently - one option is to include space as part of the token, another commonly used option is to include the lack of space as part of the token by adding a specific mark representing "the word goes on" at the end.