> You probably want to normalize if you want to use them as keys in a hash table, which is pretty common. Not necessarily. Sometimes this might be useful, but

> You probably want to normalize if you want to use them as keys in a hash table, which is pretty common. Not necessarily. Sometimes this might be useful, but sometimes it will just make it worse. > Does the invalid byte go with the 1st or the 2nd? What about multiple invalid bytes? What seems to me the most logical way to do it, if you do not then need to decode it into Unicode code point numbers, is: If the byte is in range 0x80 to 0xBF, then it belongs to the same code point as the previous byte; otherwise, the byte is the start of a new code point. (However, this does not answer the question of what if the first byte is in range 0x80 to 0xBF?) However, that is usually unnecessary. Splitting a string by Unicode code points is rarely helpful (unless you want to convert to UTF-16 or UTF-32) and is often harmful, anyways. Just use sequences of bytes. > if you're allowing invalid UTF-8 you're giving up Unicode support entirely, by necessity. It is often useful to do regular expressions on sequences of bytes instead of on sequences of Unicode code points anyways; and if you have ASCII only, then doing it this way is more efficient anyways (even if the text contains non-ASCII characters). (PCRE has a UTF-8-oriented mode and byte-oriented mode, so both are possible.) Also, you can match a UTF-8 code point (or a character code in EUC-JP or some other multibyte encoding) with a byte-oriented regular expression if you need to, anyways, so still you might not need Unicode regular expressions. > another section discusses how the length of a string is not a well defined quantity I think that the number of bytes is generally the most useful measure, although in some contexts it is useful to measure something else.