4 ms·
That's why you don't want to use regexes to parse something, but an actual tokenizer which uses Unicode _k_ompatibility normalisation (NFKC or NFKD) for compari
by ReleaseCandidat 2y ago
That's why you don't want to use regexes to parse something, but an actual tokenizer which uses Unicode _k_ompatibility normalisation (NFKC or NFKD) for comparisons. Although I'm not sure if that works with the Dollar emoji (which HN doesn't like to display)
- riffraff 2y agoRegexes can handle Unicode categories just fine, if he'd written a tokenizer it would still have failed. Which in fact it did when he removed the regex.
- ReleaseCandidat 2y agoIt's not about categories they don't help with such problems, but comparison using compatible normalisation (either NFKC or NFKD). Using that, e.g. 1 compares equal to I and i (the roman number literals) the 1 in a circle and all the other Unicode code points which have the samé meaning.
- riffraff 2y agobut that's not about using a tokenizer vs a regex, it's about using a normalization step, which would also work with the regex.
- ReleaseCandidat 2y agoYes, that's true. Except it isn't (well, doesn't have to be) an extra step in the tokenizer. Most of the time you do not want to run the whole string or part of it through the kompatibility normalization but just some code points (like the sign of a float). Which could of course be done with a matchgroup of a regexp too. I have just made the observationthrough the last two decades, that it's easier to not forget about such cases when not using regexps.