3 ms·
> The Unicode Standard does not specify how character properties should be extracted from a grapheme cluster. Programming languages that define "character" to
by Dylan16807 3y ago
> The Unicode Standard does not specify how character properties should be extracted from a grapheme cluster. Programming languages that define "character" to mean grapheme cluster (like Swift) need to establish their own ad-hoc rules.
Right. Which means not just iterating by code point.
> The author suggests iterating text as grapheme clusters, but fails to consider that this breaks tokenizers, e.g. a tokenizer for a comma-separated list [1] won't see the comma as "just a comma" if the value after it begins with a combining character.
I don't think they're talking about tokenizers. It's a general purpose rule.
Also I would argue that a CSV file with non-attached combining characters doesn't qualify as "text".