4 ms·

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundame

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code points, or its grapheme clusters. All positions within the string are represented using abstract iterators that don't reveal their numeric quantities. Diametrically opposite in the design space is Go (close to C and C++), in which a string is simply an array of bytes, conventionally treated as UTF-8. To access code points requires explicit decoding (or a range loop), and to access grapheme clusters requires a text package. Ruby is another interesting choice: a string is a pair of a (mutable!) byte array and value indicating how to decode those bytes into code points. This representation works well when you have to deal with strings in obscure encodings as it preserves the original bytes, so, for example, you can read file names from a directory then delete those files. There are a great many other designs in wide use---Haskell, Java, and Python being the most unnecessarily complicated---but these three designs seemed the most defensible to me.

vvanders 5y ago
Java's MUTF-8 makes me want to eject it into the Sun. Makes interop with JNI extra fun.
mannerheim 5y ago
Haskell just uses a list of Unicode code points, i.e., type String = [Char]. Not a good choice, but not a particularly complicated one either. What does make it a bit complicated is that there are about four other different string types to choose between, besides the default String type, which should never be used for anything performance-sensitive.
ademarre 5y ago
Now I am curious to look at Swift's strings. I am a big fan of explicit encodings. I have long believed that an ideal text API would not use string as a concrete data type. Instead, encodings themselves (UTF-8 et al.) should be concrete types and string should be nothing more than an abstraction over them for polymorphism.
astrange 5y ago
> Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code points, or its grapheme clusters. All positions within the string are represented using abstract iterators that don't reveal their numeric quantities. This is probably the only right way because 1. it admits multiple implementations (like packing into tagged pointers) and 2. a string isn't simply an array of anything. Strings are made of all of bytes/codepoints/grapheme clusters/words/lines/etc at different levels.
- adonovan 5y ago
  If my survey revealed anything, it's that there is no one right way.
elcaro 5y ago
Raku also has a specific string type (Str) that is not just a bag of bytes. Strings are normalized (NFC) when they are created, and multi-codepoint graphemes are represented internally using synthetic codepoints (negative integers), but this is all opaque to the user. This also means that string indexing in Raku is O(1). Raku passes all the tests. He mentions he didn't find any language that upper-cases "baﬄe" correctly but Raku does. Here's my REPL test > my $n = "noe\x[0308]l" noël > $n.chars 4 > $n.flip lëon > $n.substr(0, 3) noë > my $b = "baﬄe" baﬄe > $b.substr(2, 1).uniname LATIN SMALL LIGATURE FFL > $b.uc BAFFLE > "noe\x[0308]l" eq "no\x[00EB]l" True HN doesn't handle the cat emojis, so they are not included, but they worked fine.
- b2gills 5y ago
  Technically it is MoarVM that handles graphemes as synthetic codepoints. We call it NFG for Normalized Form Grapheme. The JVM and JavaScript backends currently use the built-in string handling features. Which means that they are just as broken as the rest of the code running on them. --- Of further note, you can use ignorecase in regexes to match "baﬄe". "baﬄe" ~~ /:ignorecase BAFfLE /;

vvanders 5y ago

Java's MUTF-8 makes me want to eject it into the Sun. Makes interop with JNI extra fun.

mannerheim 5y ago

Haskell just uses a list of Unicode code points, i.e., type String = [Char]. Not a good choice, but not a particularly complicated one either. What does make it a bit complicated is that there are about four other different string types to choose between, besides the default String type, which should never be used for anything performance-sensitive.

ademarre 5y ago

Now I am curious to look at Swift's strings. I am a big fan of explicit encodings. I have long believed that an ideal text API would not use string as a concrete data type. Instead, encodings themselves (UTF-8 et al.) should be concrete types and string should be nothing more than an abstraction over them for polymorphism.

astrange 5y ago

> Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code points, or its grapheme clusters. All positions within the string are represented using abstract iterators that don't reveal their numeric quantities. This is probably the only right way because 1. it admits multiple implementations (like packing into tagged pointers) and 2. a string isn't simply an array of anything. Strings are made of all of bytes/codepoints/grapheme clusters/words/lines/etc at different levels.

adonovan 5y ago
If my survey revealed anything, it's that there is no one right way.

elcaro 5y ago

Raku also has a specific string type (Str) that is not just a bag of bytes. Strings are normalized (NFC) when they are created, and multi-codepoint graphemes are represented internally using synthetic codepoints (negative integers), but this is all opaque to the user. This also means that string indexing in Raku is O(1). Raku passes all the tests. He mentions he didn't find any language that upper-cases "baﬄe" correctly but Raku does. Here's my REPL test > my $n = "noe\x[0308]l" noël > $n.chars 4 > $n.flip lëon > $n.substr(0, 3) noë > my $b = "baﬄe" baﬄe > $b.substr(2, 1).uniname LATIN SMALL LIGATURE FFL > $b.uc BAFFLE > "noe\x[0308]l" eq "no\x[00EB]l" True HN doesn't handle the cat emojis, so they are not included, but they worked fine.

b2gills 5y ago
Technically it is MoarVM that handles graphemes as synthetic codepoints. We call it NFG for Normalized Form Grapheme. The JVM and JavaScript backends currently use the built-in string handling features. Which means that they are just as broken as the rest of the code running on them. --- Of further note, you can use ignorecase in regexes to match "baﬄe". "baﬄe" ~~ /:ignorecase BAFfLE /;