5 ms·

> Define text. UTF-8 for all new projects. > If Unicode, how much Unicode should be allowed? I think the question is more like "how much should be supported"

> Define text. UTF-8 for all new projects. > If Unicode, how much Unicode should be allowed? I think the question is more like "how much should be supported", which is project dependent. If it's just a question of "allowance" then the answer is probably "all of it". If you're a UTF-8 processor, you shouldn't go out of your way to discard or disallow certain codepoints without a good reason. > How do you identify the end of a Unicode text stream, other than closing the input stream (or explicitly waiting for a out-of-band signal like a form submission)? I think the UNIX-y answer here is "end of text stream" and "closed stream" are the same. But if you do want to wrap text streams in another text stream you have a couple options: HTTP uses Content-Length and Content-Encoding: chunked (length-prefixing), while programming and markup languages use delimiter characters which must be escaped in the inner stream. What you definitely should not do is, amusingly, what C does: Reserve a text character (NUL) to use as a delimiter, and hope that character doesn't appear in your content.

lmm 4y ago
> UTF-8 for all new projects. Unless and until Unicode gets proper support for Japanese, please don't do this.
- rcoveson 4y ago
  I can’t tell from your comment, but the only thing I can think of that you might be referring to is Han unification?
  - lmm 4y ago
    Yep. Han unification in practice means unicode-based programs replace Japanese characters with Chinese characters that look different (and ugly). The unicode consortium alleges there are workarounds like switching fonts but none of them have ever been effectively implemented in practice; at best you get a program that will display Japanese correctly (at the cost of displaying Chinese poorly) if some obscure configuration option is set.
lelanthran 4y ago
> What you definitely should not do is, amusingly, what C does: Reserve a text character (NUL) to use as a delimiter, and hope that character doesn't appear in your content. I see this sentiment often, but I don't get it. Which human language uses U+0000. What glyph is it in any human language? If we're using unicode to represent human languages, exactly what problem does U+0000 present?
- rcoveson 4y ago
  Whether it's useful in human languages is irrelevant. It's a matter of supporting the entirety of a specified domain. You can't use NUL to mean something special if you purportedly support "UTF-8 data" because some downstream process (not necessarily an end user) might take you at your word and use NUL to represent something significant to them. Even with the fair assumption that no end user will ever use NUL to mean anything, only the highest layer of the stack can make NUL a reserved character. Better to just support the whole of UTF-8 and leave the sinning to downstream.
  - lelanthran 4y ago
    > Whether it's useful in human languages is irrelevant. I dunno, I think this is the only relevant bits. The whole point of unicode is to represent human languages. It's not, and was never sold as, a way to represent anything other than human languages. Once you get into the argument that "Well, we need to be prepared for peer systems that use unicode for something other than human languages", you may as well give up. > might take you at your word and use NUL to represent something significant to them. Then they aren't supporting human languages anyway. And while I see this argument often, I have yet to see anyone using U+0000 for in any way other than simply ignoring it (because no human language uses it so there's no point in displaying it or accepting it as input), or using it as a terminating character. If you have some examples that actually reserve nul for anything other than termination, I'd love to see it, because thus far all the warnings about how U+0000 should not be used as a terminator is a big nothing-burger.
    - rcoveson 4y ago

lmm 4y ago

> UTF-8 for all new projects. Unless and until Unicode gets proper support for Japanese, please don't do this.

rcoveson 4y ago
I can’t tell from your comment, but the only thing I can think of that you might be referring to is Han unification?
- lmm 4y ago
  Yep. Han unification in practice means unicode-based programs replace Japanese characters with Chinese characters that look different (and ugly). The unicode consortium alleges there are workarounds like switching fonts but none of them have ever been effectively implemented in practice; at best you get a program that will display Japanese correctly (at the cost of displaying Chinese poorly) if some obscure configuration option is set.

lelanthran 4y ago

> What you definitely should not do is, amusingly, what C does: Reserve a text character (NUL) to use as a delimiter, and hope that character doesn't appear in your content. I see this sentiment often, but I don't get it. Which human language uses U+0000. What glyph is it in any human language? If we're using unicode to represent human languages, exactly what problem does U+0000 present?

rcoveson 4y ago
Whether it's useful in human languages is irrelevant. It's a matter of supporting the entirety of a specified domain. You can't use NUL to mean something special if you purportedly support "UTF-8 data" because some downstream process (not necessarily an end user) might take you at your word and use NUL to represent something significant to them. Even with the fair assumption that no end user will ever use NUL to mean anything, only the highest layer of the stack can make NUL a reserved character. Better to just support the whole of UTF-8 and leave the sinning to downstream.
- lelanthran 4y ago
  > Whether it's useful in human languages is irrelevant. I dunno, I think this is the only relevant bits. The whole point of unicode is to represent human languages. It's not, and was never sold as, a way to represent anything other than human languages. Once you get into the argument that "Well, we need to be prepared for peer systems that use unicode for something other than human languages", you may as well give up. > might take you at your word and use NUL to represent something significant to them. Then they aren't supporting human languages anyway. And while I see this argument often, I have yet to see anyone using U+0000 for in any way other than simply ignoring it (because no human language uses it so there's no point in displaying it or accepting it as input), or using it as a terminating character. If you have some examples that actually reserve nul for anything other than termination, I'd love to see it, because thus far all the warnings about how U+0000 should not be used as a terminator is a big nothing-burger.
  - rcoveson 4y ago