4 ms·

Can someone explain, what are the rules for substring(m, n) given all the madness that's today's Unicode? Is it standardized or it's up to the implementations?

by mojuba 6y ago

_ZeD_ 6y ago
it think the only resonable rule for substring(m, n) is "don't"
- mojuba 6y ago
  So string is no longer a "string of characters", it is in fact a program (not Turing complete) that you need to execute. Though substring(m, n) still makes sense in at least interactive text manipulation: how do you do copy/paste?
  - techdragon 6y ago
    Not really. A Unicode string is more like a sequence of data built from simple binary structs, which belong to a smallish group of valid structs. Additionally, some but not all, of these structs can be used to infer the validity of subsequent structs in the sequence if your parsing in a more byte-at-a-time fashion. Alternately if your happy dealing with a little less forward compatibility and go for explicit enumeration of all groups of valid bytes you can be a lot more sure of things but it’s harder to make this method as performant as the byte-at-a-time method, which given the complete ubiquity of string processing in software... leads to the dominance of the byte-at-a-time method.
  - truefossil 6y ago
    The safest path is to consider it a blob. There is some library that can render it magically and that's the only wise thing you can do. The internal structure is hard to understand. Also, definitions change over time. So, you better leave it all to professionals.
    - spookthesunset 6y ago
      The thing about Unicode is.... anybody who tried to do it “more simple” would eventually just develop a crappier version of Unicode. Unicode is complex because the sum of all human language is complex. Short of a ground up rewrite of the worlds languages, you cannot boil away most of that complexity... it has to go somewhere. And even if you did manage to “rewrite” the worlds languages to be simple and remove accidental complexity I assert that over centuries it would devolve right back into a complex mess again. Why? Languages represent (and literally shape and constrain) how humans think and humans are a messy bunch of meat sacks living in a huge world rich in weird crazy things to feel and talk about.
      - kps 6y ago
        There are definitely crappy things about Unicode that are separate from language. - Several writing systems are widely scattered across multiple ‘Supplement’/‘Extended’/‘Extensions’ blocks. - Operators (e.g. combining forms, joiners) are a mishmash of postfix, infix, and halffix. They should have been (a) in an easily tested reserved block (e.g. 0xF0nn for binary operators, 0xFmnn for unary), so that you could parse over a sequence even if it contains specific operators from a later version — i.e. separate syntax from semantics, and (b) uniformly prefix, so that read-ahead isn't required to find the end of a sequence (and dead keys become just like normal characters).
  - roel_v 6y ago
    "So string is no longer a "string of characters"" It hasn't been for 30 years.
  - goto11 6y ago
    No it is not a program - at least not anymore than an ASCII string is a program. It is just that there isn't a simple 1:1 correspondence between bytes and characters and glyphs as in unicode, so you cant just extract an arbitrary byte-sequence from a string and expect it to render correctly.
    - mojuba 6y ago
      > there isn't a simple 1:1 correspondence between bytes and characters and glyphs There isn't a simple 1:1 correspondence between anything at all. The only definitive thing about Unicode strings is the beginning where you should start your parsing. Then the way things are supposed to be displayed to be Unicode-compliant look more like some virtual machine analyzing the code. How is this different from any other declarative language?
deleted 6y ago
[deleted]
EMM_386 6y ago
It is up to the implementation. This is a good read on aspects of it: https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/
thristian 6y ago
It depends what your string is a string of. Slicing by byte-offset is pretty unhelpful, given how many Unicode characters occupy more than one byte. In an encoding like UTF-16, that's "all of them" but even in UTF-8 it's still "most of them". Slicing by UTF-16 code-unit is still pretty unhelpful, since a lot of Unicode characters (such as emoji) do not fit in 16 bits, and are encoded as "surrogate pairs". If you happen to slice a surrogate pair in half, you've made a mess. Slicing by code-points (the numbers allocated by the Unicode consortium) is better, but not great. A shape like the "é" in "café" could be written as U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Those are separate code-points, but if you slice between them you'll wind up with "cafe" and an isolated acute accent that will stick to whatever it's next to, like this:́ When combining characters stick to a base character, the result is called a "grapheme cluster". Slicing by grapheme clusters is the best option, but it's expensive since you need a bunch of data from the Unicode database to find the edges of each cluster - it depends on the properties assigned to each character.
- andreareina 6y ago
  Doesn't splitting by grapheme cluster also depends on which version of the unicode standard you use, since new standards come with new combinations?
RedNifre 6y ago
Maybe have m,n refer to grapheme clusters instead of bytes/code points?
- mojuba 6y ago
  Apparently it's what Swift does when you try to get the length of a string. Though there's no more plain substring() since Swift 5, it was removed to indicate it's no longer O(1). You will get different results across languages though.
kevincox 6y ago
The standard answer is "don't". Just treat text is a blob, but the other question is what are you trying to accomplish? - Are you trying to control the rendered length? In that case the perfect solution is actually rendering the string. - Are you limiting storage size? Then you need to find a good split point that is <N bytes. This is probably done using extended grapheme clusters. (Although this also isn't perfect) I'm sure there are other use cases as well. But at the end of the day try to avoid splitting text if it can be helped.

Can someone explain, what are the rules for substring(m, n) given all the madness that's today's Unicode? Is it standardized or it's up to the implementations?

by mojuba 6y ago

Can someone explain, what are the rules for substring(m, n) given all the madness that's today's Unicode? Is it standardized or it's up to the implementations?

_ZeD_ 6y ago

it think the only resonable rule for substring(m, n) is "don't"

mojuba 6y ago
So string is no longer a "string of characters", it is in fact a program (not Turing complete) that you need to execute. Though substring(m, n) still makes sense in at least interactive text manipulation: how do you do copy/paste?
- techdragon 6y ago
  Not really. A Unicode string is more like a sequence of data built from simple binary structs, which belong to a smallish group of valid structs. Additionally, some but not all, of these structs can be used to infer the validity of subsequent structs in the sequence if your parsing in a more byte-at-a-time fashion. Alternately if your happy dealing with a little less forward compatibility and go for explicit enumeration of all groups of valid bytes you can be a lot more sure of things but it’s harder to make this method as performant as the byte-at-a-time method, which given the complete ubiquity of string processing in software... leads to the dominance of the byte-at-a-time method.
- truefossil 6y ago
  The safest path is to consider it a blob. There is some library that can render it magically and that's the only wise thing you can do. The internal structure is hard to understand. Also, definitions change over time. So, you better leave it all to professionals.
  - spookthesunset 6y ago
    The thing about Unicode is.... anybody who tried to do it “more simple” would eventually just develop a crappier version of Unicode. Unicode is complex because the sum of all human language is complex. Short of a ground up rewrite of the worlds languages, you cannot boil away most of that complexity... it has to go somewhere. And even if you did manage to “rewrite” the worlds languages to be simple and remove accidental complexity I assert that over centuries it would devolve right back into a complex mess again. Why? Languages represent (and literally shape and constrain) how humans think and humans are a messy bunch of meat sacks living in a huge world rich in weird crazy things to feel and talk about.
    - kps 6y ago
      There are definitely crappy things about Unicode that are separate from language. - Several writing systems are widely scattered across multiple ‘Supplement’/‘Extended’/‘Extensions’ blocks. - Operators (e.g. combining forms, joiners) are a mishmash of postfix, infix, and halffix. They should have been (a) in an easily tested reserved block (e.g. 0xF0nn for binary operators, 0xFmnn for unary), so that you could parse over a sequence even if it contains specific operators from a later version — i.e. separate syntax from semantics, and (b) uniformly prefix, so that read-ahead isn't required to find the end of a sequence (and dead keys become just like normal characters).
- roel_v 6y ago
  "So string is no longer a "string of characters"" It hasn't been for 30 years.
- goto11 6y ago
  No it is not a program - at least not anymore than an ASCII string is a program. It is just that there isn't a simple 1:1 correspondence between bytes and characters and glyphs as in unicode, so you cant just extract an arbitrary byte-sequence from a string and expect it to render correctly.
  - mojuba 6y ago
    > there isn't a simple 1:1 correspondence between bytes and characters and glyphs There isn't a simple 1:1 correspondence between anything at all. The only definitive thing about Unicode strings is the beginning where you should start your parsing. Then the way things are supposed to be displayed to be Unicode-compliant look more like some virtual machine analyzing the code. How is this different from any other declarative language?

deleted 6y ago

[deleted]

EMM_386 6y ago

It is up to the implementation. This is a good read on aspects of it: https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/

thristian 6y ago

It depends what your string is a string of. Slicing by byte-offset is pretty unhelpful, given how many Unicode characters occupy more than one byte. In an encoding like UTF-16, that's "all of them" but even in UTF-8 it's still "most of them". Slicing by UTF-16 code-unit is still pretty unhelpful, since a lot of Unicode characters (such as emoji) do not fit in 16 bits, and are encoded as "surrogate pairs". If you happen to slice a surrogate pair in half, you've made a mess. Slicing by code-points (the numbers allocated by the Unicode consortium) is better, but not great. A shape like the "é" in "café" could be written as U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Those are separate code-points, but if you slice between them you'll wind up with "cafe" and an isolated acute accent that will stick to whatever it's next to, like this:́ When combining characters stick to a base character, the result is called a "grapheme cluster". Slicing by grapheme clusters is the best option, but it's expensive since you need a bunch of data from the Unicode database to find the edges of each cluster - it depends on the properties assigned to each character.

andreareina 6y ago
Doesn't splitting by grapheme cluster also depends on which version of the unicode standard you use, since new standards come with new combinations?

RedNifre 6y ago

Maybe have m,n refer to grapheme clusters instead of bytes/code points?

mojuba 6y ago
Apparently it's what Swift does when you try to get the length of a string. Though there's no more plain substring() since Swift 5, it was removed to indicate it's no longer O(1). You will get different results across languages though.

kevincox 6y ago

The standard answer is "don't". Just treat text is a blob, but the other question is what are you trying to accomplish? - Are you trying to control the rendered length? In that case the perfect solution is actually rendering the string. - Are you limiting storage size? Then you need to find a good split point that is <N bytes. This is probably done using extended grapheme clusters. (Although this also isn't perfect) I'm sure there are other use cases as well. But at the end of the day try to avoid splitting text if it can be helped.