3 ms·

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) st

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with logical length 1. It's _data_ length is variable, since a Unicode character is variable length. So many Unicode-containing string design problems arise because of the stubborn insistence of having an integral character type. I should be able to extract UTF-8, UTF-16 or whatever encoding I want from a `text` value. Something like `c_str()` would be pretty important, but the semantics would be a design problem, not an encoding problem. Any Unicode-encoding string should be able to encode U+0000, so you'd need to figure out how to handle that from `c_str()` (perhaps a substitution ASCII character could be specified to encode embedded nulls). Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn't need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.

shadowgovt 6y ago
Essentially, different tools for different applications. "A string is a vector of characters, which happen to each be one byte in length" was more of an artifact of a time where there happened to be representational overlap than some deep truism about proper data structure. Strings intended to be displayed to humans are specialized constructs, much as a "button" or a "file handle" are. A buffer of unstructured bytes is a separate specialized construct, suitable for tasks unrelated to "displaying text to a human."
lisper 6y ago
I fully endorse the general idea here, but this: > `someText.firstCharacter()` would have a return type of `text`, with logical length 1 is a huge mistake. There are operations that make sense on characters that do not make sense on texts whose length happens to be 1. The most obvious of these is inquiring about the numerical value of the unicode code point of a character. Conflating characters and texts-of-length-1 is a mistake of the same order as conflating strings and byte vectors. Python makes this mistake even in version 3. As a result, a function like this: def f(s, n, m): return ord(s[n:m]) will return a value iff m is one more than n. Not good.
- donaldihunter 6y ago
  Only if you ignore the rest of what the post said. First it should make things easy for ‘normal’ tasks, then it should make everything else possible. > Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn't need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.
  - lisper 6y ago
    Except that the "users" of a string type are programmers, and a "normal task" for a programmer often requires things like this. I'll give you an example from a project I am currently working on: a spam filter. One of the things my filter does is count the number of Chinese characters in a string. I implement this as n1<=ord(c)<=n2 where n1 and n2 are integers representing the start and end of the range of Unicode Chinese characters. This seems like a "normal task" to me and I don't see how conflating characters and texts-of-length-1 would make this any easier.
- Koshkin 6y ago
  Functors are everywhere. That's why we need monads!
  - lisper 6y ago
    Gnats and sledgehammers something something...
donaldihunter 6y ago
This! Raku introduced the concept of NFG - Normal Form Grapheme - as a way to represent any Unicode string in its logical ‘visual character’ grapheme form. Sequences of combining characters that don’t have a canonical single codepoint form are given a synthetic codepoint so that string methods including regexes can operate on grapheme characters without ever causing splitting side effects. Of course there are methods for manipulating at the codepoint level as well.
BlueTemplar 6y ago
> Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with logical length 1. It's _data_ length is variable, since a Unicode character is variable length. I don't see how you came to the "opposite conclusion" when the author basically says the same thing ?
specialist 6y ago
My future perfect programming language will have explicit native types for strings: ansi, utf8, utf16, and unicode. The nth of any array works as expected for that type. Convert to bytes as needed. ansi "abcd"[1] -> byte utf8 "abcd"[1] -> char utf16 "abcd"[1] -> char utf8 "abcd".toBytes()[1] -> byte unicode "abcd"[1] -> word Were it still the 90s, I'd probably care about locales, so somehow imbue ansi arrays with that metadata.

shadowgovt 6y ago

Essentially, different tools for different applications. "A string is a vector of characters, which happen to each be one byte in length" was more of an artifact of a time where there happened to be representational overlap than some deep truism about proper data structure. Strings intended to be displayed to humans are specialized constructs, much as a "button" or a "file handle" are. A buffer of unstructured bytes is a separate specialized construct, suitable for tasks unrelated to "displaying text to a human."

lisper 6y ago

I fully endorse the general idea here, but this: > `someText.firstCharacter()` would have a return type of `text`, with logical length 1 is a huge mistake. There are operations that make sense on characters that do not make sense on texts whose length happens to be 1. The most obvious of these is inquiring about the numerical value of the unicode code point of a character. Conflating characters and texts-of-length-1 is a mistake of the same order as conflating strings and byte vectors. Python makes this mistake even in version 3. As a result, a function like this: def f(s, n, m): return ord(s[n:m]) will return a value iff m is one more than n. Not good.

donaldihunter 6y ago
Only if you ignore the rest of what the post said. First it should make things easy for ‘normal’ tasks, then it should make everything else possible. > Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn't need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.
- lisper 6y ago
  Except that the "users" of a string type are programmers, and a "normal task" for a programmer often requires things like this. I'll give you an example from a project I am currently working on: a spam filter. One of the things my filter does is count the number of Chinese characters in a string. I implement this as n1<=ord(c)<=n2 where n1 and n2 are integers representing the start and end of the range of Unicode Chinese characters. This seems like a "normal task" to me and I don't see how conflating characters and texts-of-length-1 would make this any easier.
Koshkin 6y ago
Functors are everywhere. That's why we need monads!
- lisper 6y ago
  Gnats and sledgehammers something something...

donaldihunter 6y ago

This! Raku introduced the concept of NFG - Normal Form Grapheme - as a way to represent any Unicode string in its logical ‘visual character’ grapheme form. Sequences of combining characters that don’t have a canonical single codepoint form are given a synthetic codepoint so that string methods including regexes can operate on grapheme characters without ever causing splitting side effects. Of course there are methods for manipulating at the codepoint level as well.

BlueTemplar 6y ago

> Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with logical length 1. It's _data_ length is variable, since a Unicode character is variable length. I don't see how you came to the "opposite conclusion" when the author basically says the same thing ?

specialist 6y ago

My future perfect programming language will have explicit native types for strings: ansi, utf8, utf16, and unicode. The nth of any array works as expected for that type. Convert to bytes as needed. ansi "abcd"[1] -> byte utf8 "abcd"[1] -> char utf16 "abcd"[1] -> char utf8 "abcd".toBytes()[1] -> byte unicode "abcd"[1] -> word Were it still the 90s, I'd probably care about locales, so somehow imbue ansi arrays with that metadata.