>> to split a string into words [...] For Unicode, I don't even know where to look up how to do it. > That's not a problem with unicode. So how do I know or m

>> to split a string into words [...] For Unicode, I don't even know where to look up how to do it. > That's not a problem with unicode. So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider. >> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)? > The exact same question exists about ASCII, since \xFF is not valid ASCII. Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible. > It's painful that your random manglings don't make sense? If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks). >> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction. > They don't. They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong? >> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side. > Which is already an issue in ASCII. Some common sense helps here. Show me what serious fonts don't support the ASCII print range. > And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues. I'm totally with you.