4 ms·
but that's not about using a tokenizer vs a regex, it's about using a normalization step, which would also work with the regex.
by riffraff 2y ago
but that's not about using a tokenizer vs a regex, it's about using a normalization step, which would also work with the regex.
- ReleaseCandidat 2y agoYes, that's true. Except it isn't (well, doesn't have to be) an extra step in the tokenizer. Most of the time you do not want to run the whole string or part of it through the kompatibility normalization but just some code points (like the sign of a float). Which could of course be done with a matchgroup of a regexp too. I have just made the observationthrough the last two decades, that it's easier to not forget about such cases when not using regexps.