3 ms·
I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing
by tomwojcik 5y ago
I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing I found yesterday (conincidence).
Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6
), are made of two code points.
EDIT: Flag unicode.
- chrismorgan 5y agoCorrection: Unicode strings are made up of scalar values, not code points. The difference is that accursed mess that UTF-16 foist upon us, surrogates;oU+D800–U+DFFF are code points, but are not valid scalar values and cannot occur in Unicode strings (even if things like JavaScript are stupid in allowing potentially-malformed UTF-16, which is incompatible with Unicode, and Python similarly stupid in having strings that are not Unicode but are instead sequences of code points).
- tomwojcik 5y agoThanks. I think I will need to read a proper book / article on this subject as encoding in general keeps haunting me periodically.