2 ms·
This is really just an ad for Dochia's testing product. But the first half of the post really is an interesting problem -- what to do about invisible Unicode c
by crazygringo 1y ago
This is really just an ad for Dochia's testing product.
But the first half of the post really is an interesting problem -- what to do about invisible Unicode characters that wind up in a username login field, thus becoming an invalid user, because the username was copied-pasted from a source that inserted things. The post lists potential sources as:
> Copy-paste from PDFs or Word docs: Rich-text formats often inject hidden control characters.
> Email clients and chat apps: Some insert soft hyphens, directionality markers, or non-breaking spaces.
> Keyboards and IMEs: Certain language input systems add combining marks or zero-width joiners.
But of course it's part of a broader Unicode problem, like the fact that there are two ways of representing common accented characters (precomposed vs decomposed) that are also not equivalent, or that multiple accents can be in a different order. Normalization handles those cases, but it doesn't do anything about nonprinting characters.
Is there not any common method for Unicode we should be using to check for, essentially, "grapheme comparison" that doesn't just normalize but ignores non-printing codepoints?