3 ms·
Unicode grapheme clusters (letters/ideograms plus accent marks) are inherently variable length, but most devs haven’t read about how that works. We shouldn’t be
by erik_seaberg 6y ago
Unicode grapheme clusters (letters/ideograms plus accent marks) are inherently variable length, but most devs haven’t read about how that works. We shouldn’t be working with single codepoints or bytes for the same reason we don’t work with US-ASCII one bit at a time, because most slices of a bit string will be nonsense.
- bjourne 6y agoTreating strings as arrays of bytes is very reasonable. It is the assumption that each byte is exactly one character that causes trouble.
- astrange 6y agoYou should treat them as strings, the backing data store is an implementation detail. What do you want to do with individual bytes if you access them? Whatever you do is likely to be incorrect in some language, even English with emoji mixed in or Zalgo text. This also lets you implement compressed small strings (eg in tagged pointers) which is a good memory optimization.