4 ms·

Define text. Pure ASCII or Unicode? If Unicode, how much Unicode should be allowed? How do you identify the end of a Unicode text stream, other than closing the

Define text. Pure ASCII or Unicode? If Unicode, how much Unicode should be allowed? How do you identify the end of a Unicode text stream, other than closing the input stream (or explicitly waiting for a out-of-band signal like a form submission)?

anotheronebites 4y ago
ASCII and unicode are subtly different things. you should compare ASCII with one of the UTF encodings of unicode. the 'unicode' for ASCII's would be just the english alphabet extended with digits and a few more special control characters.
- falcolas 4y ago
  Constraining Unicode encodings to a fewer than 4 bytes means we limit how many countries can use text interfaces in their language. Or how much data from those countries can be passed between programs.
  - blueflow 4y ago
    We do not have enough countries to fill up all that space. For UTF-8 with a 4-byte restriction, less than 18% of the available space is currently allocated to blocks.
- giantrobot 4y ago
  That's what the GP is saying. You can't really tell a priori what encoding to use for a stream of "text" (bytes). Without some sort of metadata about the stream you just have to guess. Convention will help you make an informed guess but it's not guaranteed to be correct. Then stuff breaks in unexpected and stupid ways.
eimrine 4y ago
> define text Text interface is any non-GUI?
- falcolas 4y ago
  GUIs can have representations of text, and text input fields though...
- f1shy 4y ago
  Where is the line between GUI and TUI? Or TUI and CLI? Also, where does a system like Genera fits? Was it CLI, TUI or GUI?!
- mostlylurks 4y ago
  There are many more types of interfaces than GUIs and text-based interfaces. Voice user interfaces (VUIs), for instance, which have become commonplace in recent years, especially with virtual assistants like Alexa or Siri.
  - jrm4 4y ago
    VUI's are just text interfaces with a different input system and more error checking. They implement operations that give you convenience in exchange for versatility.
    - mostlylurks 4y ago
      Even if the two types of interfaces share strong similarities in their most commonplace usage, they are nevertheless accessed through two distinct mediums (from which they get their names (TUI / VUI)). Text is not speech, and speech is not text, and the possibility space of what can be realistically (in a human-friendly manner) provided as input or output on each is quite different. On VUIs, for instance, you can do things like query for a song by humming it [0], which would be very difficult to provide a purely textual alternative for for use in a TUI, whereas text-based interfaces make it easier to do things like looking at large swathes of structured content or copypasting a long piece of text to use as input. [0]: https://blog.google/products/search/hum-to-search/ https://blog.google/products/search/hum-to-search/
akira2501 4y ago
> How do you identify the end of a Unicode text stream "Why not just a zero byte?"
- falcolas 4y ago
  You'd need 4 of them to avoid ambiguity. The C string delineation model has a lot of issues we probably don't want to keep. :)
  - rmasters 4y ago
    In some encodings? One zero is sufficient for UTF-8.
every 4y ago
"Pure" ASCII would be 7-bit US-ASCII derived directly from the teletypewriter command set. It consists of exactly 128 character codes, of which roughly 30% are unprintable. Anything beyond that, such as code page 437, are not "pure". It should also be noted that the first 128 character codes of UTF-8 are "pure" 7-bit US-ASCII...
jrm4 4y ago
"Define text" is a pretty good question, but "ASCII" v. "Unicode" is frankly one of those deeply nerdy things that won't matter much at all in the grand scheme of all of this. I suspect those familiar with pre-computer languages/linguistics would be able to come up with a pretty good working definition here, but something like "communication systems that utilize a relatively small number of repeating symbols to convey meaning, often associated with vocalizations?"
mjevans 4y ago
8 bit clean octet transfer; any interpretation of value and/or protocol other than an out of band end of stream / termination is up to the programs involved. Is that a valid UTF-8 encoded JSON? Formatted newline delimitated record sets? Some formatting of data no one's thought up yet? Dunno, unless the program is instructed to do a specific thing with / to the data don't care.
rcoveson 4y ago
> Define text. UTF-8 for all new projects. > If Unicode, how much Unicode should be allowed? I think the question is more like "how much should be supported", which is project dependent. If it's just a question of "allowance" then the answer is probably "all of it". If you're a UTF-8 processor, you shouldn't go out of your way to discard or disallow certain codepoints without a good reason. > How do you identify the end of a Unicode text stream, other than closing the input stream (or explicitly waiting for a out-of-band signal like a form submission)? I think the UNIX-y answer here is "end of text stream" and "closed stream" are the same. But if you do want to wrap text streams in another text stream you have a couple options: HTTP uses Content-Length and Content-Encoding: chunked (length-prefixing), while programming and markup languages use delimiter characters which must be escaped in the inner stream. What you definitely should not do is, amusingly, what C does: Reserve a text character (NUL) to use as a delimiter, and hope that character doesn't appear in your content.
- lmm 4y ago
  > UTF-8 for all new projects. Unless and until Unicode gets proper support for Japanese, please don't do this.
  - rcoveson 4y ago
    I can’t tell from your comment, but the only thing I can think of that you might be referring to is Han unification?
    - lmm 4y ago
      Yep. Han unification in practice means unicode-based programs replace Japanese characters with Chinese characters that look different (and ugly). The unicode consortium alleges there are workarounds like switching fonts but none of them have ever been effectively implemented in practice; at best you get a program that will display Japanese correctly (at the cost of displaying Chinese poorly) if some obscure configuration option is set.
- lelanthran 4y ago

anotheronebites 4y ago

ASCII and unicode are subtly different things. you should compare ASCII with one of the UTF encodings of unicode. the 'unicode' for ASCII's would be just the english alphabet extended with digits and a few more special control characters.

falcolas 4y ago
Constraining Unicode encodings to a fewer than 4 bytes means we limit how many countries can use text interfaces in their language. Or how much data from those countries can be passed between programs.
- blueflow 4y ago
  We do not have enough countries to fill up all that space. For UTF-8 with a 4-byte restriction, less than 18% of the available space is currently allocated to blocks.
giantrobot 4y ago
That's what the GP is saying. You can't really tell a priori what encoding to use for a stream of "text" (bytes). Without some sort of metadata about the stream you just have to guess. Convention will help you make an informed guess but it's not guaranteed to be correct. Then stuff breaks in unexpected and stupid ways.

eimrine 4y ago

> define text Text interface is any non-GUI?

falcolas 4y ago
GUIs can have representations of text, and text input fields though...
f1shy 4y ago
Where is the line between GUI and TUI? Or TUI and CLI? Also, where does a system like Genera fits? Was it CLI, TUI or GUI?!
mostlylurks 4y ago
There are many more types of interfaces than GUIs and text-based interfaces. Voice user interfaces (VUIs), for instance, which have become commonplace in recent years, especially with virtual assistants like Alexa or Siri.
- jrm4 4y ago
  VUI's are just text interfaces with a different input system and more error checking. They implement operations that give you convenience in exchange for versatility.
  - mostlylurks 4y ago
    Even if the two types of interfaces share strong similarities in their most commonplace usage, they are nevertheless accessed through two distinct mediums (from which they get their names (TUI / VUI)). Text is not speech, and speech is not text, and the possibility space of what can be realistically (in a human-friendly manner) provided as input or output on each is quite different. On VUIs, for instance, you can do things like query for a song by humming it [0], which would be very difficult to provide a purely textual alternative for for use in a TUI, whereas text-based interfaces make it easier to do things like looking at large swathes of structured content or copypasting a long piece of text to use as input. [0]: https://blog.google/products/search/hum-to-search/ https://blog.google/products/search/hum-to-search/

akira2501 4y ago

> How do you identify the end of a Unicode text stream "Why not just a zero byte?"

falcolas 4y ago
You'd need 4 of them to avoid ambiguity. The C string delineation model has a lot of issues we probably don't want to keep. :)
- rmasters 4y ago
  In some encodings? One zero is sufficient for UTF-8.

every 4y ago

"Pure" ASCII would be 7-bit US-ASCII derived directly from the teletypewriter command set. It consists of exactly 128 character codes, of which roughly 30% are unprintable. Anything beyond that, such as code page 437, are not "pure". It should also be noted that the first 128 character codes of UTF-8 are "pure" 7-bit US-ASCII...

jrm4 4y ago

"Define text" is a pretty good question, but "ASCII" v. "Unicode" is frankly one of those deeply nerdy things that won't matter much at all in the grand scheme of all of this. I suspect those familiar with pre-computer languages/linguistics would be able to come up with a pretty good working definition here, but something like "communication systems that utilize a relatively small number of repeating symbols to convey meaning, often associated with vocalizations?"

mjevans 4y ago

8 bit clean octet transfer; any interpretation of value and/or protocol other than an out of band end of stream / termination is up to the programs involved. Is that a valid UTF-8 encoded JSON? Formatted newline delimitated record sets? Some formatting of data no one's thought up yet? Dunno, unless the program is instructed to do a specific thing with / to the data don't care.

rcoveson 4y ago

> Define text. UTF-8 for all new projects. > If Unicode, how much Unicode should be allowed? I think the question is more like "how much should be supported", which is project dependent. If it's just a question of "allowance" then the answer is probably "all of it". If you're a UTF-8 processor, you shouldn't go out of your way to discard or disallow certain codepoints without a good reason. > How do you identify the end of a Unicode text stream, other than closing the input stream (or explicitly waiting for a out-of-band signal like a form submission)? I think the UNIX-y answer here is "end of text stream" and "closed stream" are the same. But if you do want to wrap text streams in another text stream you have a couple options: HTTP uses Content-Length and Content-Encoding: chunked (length-prefixing), while programming and markup languages use delimiter characters which must be escaped in the inner stream. What you definitely should not do is, amusingly, what C does: Reserve a text character (NUL) to use as a delimiter, and hope that character doesn't appear in your content.

lmm 4y ago
> UTF-8 for all new projects. Unless and until Unicode gets proper support for Japanese, please don't do this.
- rcoveson 4y ago
  I can’t tell from your comment, but the only thing I can think of that you might be referring to is Han unification?
  - lmm 4y ago
    Yep. Han unification in practice means unicode-based programs replace Japanese characters with Chinese characters that look different (and ugly). The unicode consortium alleges there are workarounds like switching fonts but none of them have ever been effectively implemented in practice; at best you get a program that will display Japanese correctly (at the cost of displaying Chinese poorly) if some obscure configuration option is set.
lelanthran 4y ago