3 ms·

The author’s original blog post praised Swift because it chose grapheme clusters as its default text abstraction. This might be fine for languages primarily int

by hgs3 4y ago

The author’s original blog post praised Swift because it chose grapheme clusters as its default text abstraction. This might be fine for languages primarily intended to operate on UI’s or DSL's but it’s a bad idea for general purpose languages. Grapheme clusters can change between Unicode versions, between locales, and confuse parsers (imagine a CSV parser that doesn’t see a “comma” anymore because the comma got unexpectedly clumped into a grapheme cluster). Furthermore, Unicode assigns properties to individual code points - not grapheme clusters. You can't query these properties if all you see are graphemes. The correct abstraction for working with Unicode text is the code point. UTF whatever is an implementation detail and not something to get hung up on. Python is one of the few languages that gets this right.

deleted 4y ago
[deleted]
Jasper_ 4y ago
> Furthermore, Unicode assigns properties to individual code points - not grapheme clusters. You can't query these properties if all you see are graphemes. Languages like Swift give you the ability to iterate over encoded bytes, graphemes, or code points. You can access these properties if you really want to, however, the point is that these properties might not be useful in practice. I kind-of disagree with the author that grapheme clusters meaningfully solve the problem. I think it's just another level of kicking the can down the road, with its own set of problems. But I think that it's at least closer than code points. > The correct abstraction for working with Unicode text is the code point. I am not aware of any scenario where the code point is meaningfully what you want. Feel free to inform me of a case. > UTF whatever is an implementation detail and not something to get hung up on. Grapheme clusters are not part of the transmission format.
- hgs3 4y ago
  > I am not aware of any scenario where the code point is meaningfully what you want. The code point is the most meaningful thing. I'm confused as to what you think you're parsing when you parse text? > Grapheme clusters are not part of the transmission format. Absolutely. The intent of my comment was to point out how most programming languages expose strings as some specific UTF encoding rather than exposing them as a collection of code points - exposing strings as UTF anything is a leaky abstraction. An exception could be made for lower level languages, like C, which define strings as pointers to the encoded data.
asiachick 4y ago
Meh, Python picked a subset of unicode as valid identifiers effectively baking non-inclusiveness into the language. Swift, said "all are welcome"

by hgs3 4y ago

deleted 4y ago

[deleted]

Jasper_ 4y ago

> Furthermore, Unicode assigns properties to individual code points - not grapheme clusters. You can't query these properties if all you see are graphemes. Languages like Swift give you the ability to iterate over encoded bytes, graphemes, or code points. You can access these properties if you really want to, however, the point is that these properties might not be useful in practice. I kind-of disagree with the author that grapheme clusters meaningfully solve the problem. I think it's just another level of kicking the can down the road, with its own set of problems. But I think that it's at least closer than code points. > The correct abstraction for working with Unicode text is the code point. I am not aware of any scenario where the code point is meaningfully what you want. Feel free to inform me of a case. > UTF whatever is an implementation detail and not something to get hung up on. Grapheme clusters are not part of the transmission format.

hgs3 4y ago
> I am not aware of any scenario where the code point is meaningfully what you want. The code point is the most meaningful thing. I'm confused as to what you think you're parsing when you parse text? > Grapheme clusters are not part of the transmission format. Absolutely. The intent of my comment was to point out how most programming languages expose strings as some specific UTF encoding rather than exposing them as a collection of code points - exposing strings as UTF anything is a leaky abstraction. An exception could be made for lower level languages, like C, which define strings as pointers to the encoded data.

asiachick 4y ago

Meh, Python picked a subset of unicode as valid identifiers effectively baking non-inclusiveness into the language. Swift, said "all are welcome"