4 ms·
It's not about categories they don't help with such problems, but comparison using compatible normalisation (either NFKC or NFKD). Using that, e.g. 1 compares e
by ReleaseCandidat 2y ago
It's not about categories they don't help with such problems, but comparison using compatible normalisation (either NFKC or NFKD).
Using that, e.g. 1 compares equal to I and i (the roman number literals) the 1 in a circle and all the other Unicode code points which have the samé meaning.
- riffraff 2y agobut that's not about using a tokenizer vs a regex, it's about using a normalization step, which would also work with the regex.
- ReleaseCandidat 2y agoYes, that's true. Except it isn't (well, doesn't have to be) an extra step in the tokenizer. Most of the time you do not want to run the whole string or part of it through the kompatibility normalization but just some code points (like the sign of a float). Which could of course be done with a matchgroup of a regexp too. I have just made the observationthrough the last two decades, that it's easier to not forget about such cases when not using regexps.