8 ms·

> I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library t

> I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. Foundation seems like the obvious one? NSURL vs. NSString?

steveklabnik 11y ago
In the Rust standard library, we have String, str, OsString, Path, and PathBuf, off the top of my head, with external crates implementing things like URLs, ropes, and others. That said, I completely feel what this blog post is saying: strings are _hard_, especially if you're not just doing ASCII.
- x5n1 11y ago
  why are they so hard?
  - ben0x539 11y ago
    Because we spent a lot of time optimizing every assumption for the ascii-only case, and letting go of that simplification is distressing, I guess.
    - steveklabnik 11y ago
      Yes. And many languages had ASCII-only stuff for a long time, so "real" APIs seem much, much more complex. Of course, that complexity was always there, but us English speakers could mostly just ignore it...
  - Gankro 11y ago
    There's two aspects of complexity with strings: 1) 40 years of crazy encodings and languages. 2) human languages are wildly diverse and basically any assumption you wish to apply is broken. For 1, any system that wants to deal with the outside world needs to deal with: operating system encodings (arbitrary bytes on unix, malformed UCS2 on windows), C representation (null-terminated strings), systems that only work with ASCII, systems that only work with utf8, systems that work with arbitrary encodings/languages (HTML). This is arguably unnecessary complexity that exists because of short-sighted decisions in the past. 2 is the necessary complexity; the fact that languages are really complicated. There are thousands of symbols in writing. Do you try to encode these symbols in a monolithic manner, or in a compositional way? For historical reasons, you can often do both! ë can be a single character, or e with an accent modifier. How do you handle string searching in such a model? Do you match `noel` with `noël`? What's the length of noël? 4 characters? 5 characters? bytes? graphemes? codepoints? Can you correctly reverse noël (do it wrong and you can get leön)? Different letters which have similar/identical representations but different semantics/origins! Is Ε "capital e" or "capital ε"? How do you upper-case or lower-case these letters? Do you expect to_upper(to_lower(char)) to roundtrip (it won't)? Do you expect capitalization to be doable in-place (it's not)? Do you expect capitalization to be region-specific (it is)? Are any of these operations even coherent in a language like Japanese? Why are you trying to do them? God help you if you want to display this text. Are you ready to handle right-to-left text? Are you assuming that your font is monospace (hey there terminal and text editors)? C̢̫a̘̺̯n ̘̜̦̹y̷̫̼̘̩o̶͉u̗̩̻̞ ̻ẹ͡v̴̤͎̹e̶̫̠̤̭̺̤̞n̛̞̹̣̩̲͉̮ ̜͖̪͔̖d̤e̘̯ͅa̺l̟̀ ͚̗̣w̭i̸͇̠̥̣̜̥t̸h̸̻̮̼̙̹ ̗̺̱̣̰̱̙z̟a̺͜l̠̦̖̟̰͍g҉̜͖͓̫ơ̩̹̰͕?̹̳̼̯̘̺̟
    - mikeash 11y ago
      I think my favorite part of this article is how my Zalgo example doesn't render even remotely correctly in any browser I've tried it in.
- Gankro 11y ago
  std provides String (utf8), OsString (bytes/wtf8), PathBuf (bytes/wtf8), CString (null-terminated). Each has their own unsized view: str, OsStr, Path, and CStr. We also provide AsciiExt for those times where you really truly believe you want to be working with a String as Ascii. That said, we generally try to make it as ergonomic as possible to pass a plain str where a Path/OsStr is expected. This is because utf8 is a subset of wtf8, so it's always fine to convert in that direction blindly (and it's really nice to just be like `File::open("foo.txt")` when hacking something together). This is why so many interfaces are riddled with something like `P: As<Path>`. The differentiation largely exists for the other direction, IMO. Paths and OsStrs aren't guaranteed to be valid UTF8, and shouldn't be provided where a proper utf8 string is expected. Path is just a convenience wrapper over OsStr that understands the platform's seperator conventions and provides convenient utilities.
- erickt 11y ago
  Don't forget we kinda sorta have byte characters and strings a la `b'x'` and `b"foo"`, which are really just simpler ways of expressing byte slices. Unfortunately they lack string specific methods until we get specialization (fingers crossed).
kazinator 11y ago
Typing strings to solve bugs is the epitome of an anti-pattern. If you want type, yank the data out of the string representation and into an actual data type! "The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information." -- Alan Perlis, Epigram #34.
- jasode 11y ago
  >, yank the data out of the string representation Well, the very act of "yanking" can be helped by more well-defined behavior policies of "typed strings". It is unavoidable that the programmer will have to confront a contiguous piece of memory (string) with no boundaries between characters whether it's a block of bytes on disk, network stream, or the clipboard buffer. It can be helpful to differentiate the various types of strings with a formal type system before the yanking takes place. The Alan Perlis epigram is wise but I don't think it applies here. I think's he's talking about abusing strings as an adhoc data format by stuffing them with multi-value data and then turning right around and parsing them. For example, concatenating name+":"+id+":"+amount into a string with colons like "John:389432:$5.99", passing the "complex" string around, and then later parsing out the colons with string.split(). Basically, the programmer is making an in-memory csv (comma separated values) data format instead of using "C struct {}" or passing 3 separate formal parameters.
  - mannykannot 11y ago
    > ...before the yanking takes place. I take it that you are advocating for typed strings in addition to fully-formed types for things like URLs and SQL, but I think all that will lead to is a lot of gratuitous casting of strings just so they can be passed to the fully-formed types. Furthermore, few of these casts could be statically checked for validity.
    - jasode 11y ago
      >, but I think all that will lead to is a lot of gratuitous casting of strings But if programmers are not directly doing memcpy() of bytes directly to the stack frame, they are already doing a roundabout version of "casting". If a programmer has to "yank" from a string to put it into a more organized data type, the "yank" has to be done within the framework of semantics. If one "yanks" or "slices", or "substr", etc characters 3 to 15, the exact behavior depends on whether contiguous bytes is ASCII, UTF8, UTF16, or binary blob (which means NUL(0) is valid byte in the middle of the block of bytes). That "meaning overlayed onto those bytes to manipulate it correctly" has to decided somewhere. It is either implicit or explicit. Likewise, if one pulls 4 bytes from disk or network, the programmer has to assign meaning to it and say it is the 4-bytes of an IEEE floating point (float) or 4-bytes of a integer (int). The programmer may later want to do a bitshift (<<) which is not applicable to a float.
fizx 11y ago
/Regexp?/
mikeash 11y ago
Sort of. One spot of trouble is that NSString paths still exist and show up all over the place, even though NSURL is slowly replacing them. Another spot of trouble is that NSURL ends up in a similar situation, because sometimes NSURL means "an arbitrary URL" but often it means "a local file URL and nothing else." NSURL can hold arbitrary URLs with arbitrary schemas, but there are a ton of APIs in the frameworks which take NSURLs but only if they're file: URLs. It would be really nice to have a NSFileRef class or something like that which means, really, this is a reference to a local file, promise.
- cballard 11y ago
  Yeah, I feel like all of the NSString-based APIs should have been deprecated by now?

steveklabnik 11y ago

In the Rust standard library, we have String, str, OsString, Path, and PathBuf, off the top of my head, with external crates implementing things like URLs, ropes, and others. That said, I completely feel what this blog post is saying: strings are _hard_, especially if you're not just doing ASCII.

x5n1 11y ago
why are they so hard?
- ben0x539 11y ago
  Because we spent a lot of time optimizing every assumption for the ascii-only case, and letting go of that simplification is distressing, I guess.
  - steveklabnik 11y ago
    Yes. And many languages had ASCII-only stuff for a long time, so "real" APIs seem much, much more complex. Of course, that complexity was always there, but us English speakers could mostly just ignore it...
- Gankro 11y ago
  There's two aspects of complexity with strings: 1) 40 years of crazy encodings and languages. 2) human languages are wildly diverse and basically any assumption you wish to apply is broken. For 1, any system that wants to deal with the outside world needs to deal with: operating system encodings (arbitrary bytes on unix, malformed UCS2 on windows), C representation (null-terminated strings), systems that only work with ASCII, systems that only work with utf8, systems that work with arbitrary encodings/languages (HTML). This is arguably unnecessary complexity that exists because of short-sighted decisions in the past. 2 is the necessary complexity; the fact that languages are really complicated. There are thousands of symbols in writing. Do you try to encode these symbols in a monolithic manner, or in a compositional way? For historical reasons, you can often do both! ë can be a single character, or e with an accent modifier. How do you handle string searching in such a model? Do you match `noel` with `noël`? What's the length of noël? 4 characters? 5 characters? bytes? graphemes? codepoints? Can you correctly reverse noël (do it wrong and you can get leön)? Different letters which have similar/identical representations but different semantics/origins! Is Ε "capital e" or "capital ε"? How do you upper-case or lower-case these letters? Do you expect to_upper(to_lower(char)) to roundtrip (it won't)? Do you expect capitalization to be doable in-place (it's not)? Do you expect capitalization to be region-specific (it is)? Are any of these operations even coherent in a language like Japanese? Why are you trying to do them? God help you if you want to display this text. Are you ready to handle right-to-left text? Are you assuming that your font is monospace (hey there terminal and text editors)? C̢̫a̘̺̯n ̘̜̦̹y̷̫̼̘̩o̶͉u̗̩̻̞ ̻ẹ͡v̴̤͎̹e̶̫̠̤̭̺̤̞n̛̞̹̣̩̲͉̮ ̜͖̪͔̖d̤e̘̯ͅa̺l̟̀ ͚̗̣w̭i̸͇̠̥̣̜̥t̸h̸̻̮̼̙̹ ̗̺̱̣̰̱̙z̟a̺͜l̠̦̖̟̰͍g҉̜͖͓̫ơ̩̹̰͕?̹̳̼̯̘̺̟
  - mikeash 11y ago
    I think my favorite part of this article is how my Zalgo example doesn't render even remotely correctly in any browser I've tried it in.
Gankro 11y ago
std provides String (utf8), OsString (bytes/wtf8), PathBuf (bytes/wtf8), CString (null-terminated). Each has their own unsized view: str, OsStr, Path, and CStr. We also provide AsciiExt for those times where you really truly believe you want to be working with a String as Ascii. That said, we generally try to make it as ergonomic as possible to pass a plain str where a Path/OsStr is expected. This is because utf8 is a subset of wtf8, so it's always fine to convert in that direction blindly (and it's really nice to just be like `File::open("foo.txt")` when hacking something together). This is why so many interfaces are riddled with something like `P: As<Path>`. The differentiation largely exists for the other direction, IMO. Paths and OsStrs aren't guaranteed to be valid UTF8, and shouldn't be provided where a proper utf8 string is expected. Path is just a convenience wrapper over OsStr that understands the platform's seperator conventions and provides convenient utilities.
erickt 11y ago
Don't forget we kinda sorta have byte characters and strings a la `b'x'` and `b"foo"`, which are really just simpler ways of expressing byte slices. Unfortunately they lack string specific methods until we get specialization (fingers crossed).

kazinator 11y ago

Typing strings to solve bugs is the epitome of an anti-pattern. If you want type, yank the data out of the string representation and into an actual data type! "The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information." -- Alan Perlis, Epigram #34.

jasode 11y ago
>, yank the data out of the string representation Well, the very act of "yanking" can be helped by more well-defined behavior policies of "typed strings". It is unavoidable that the programmer will have to confront a contiguous piece of memory (string) with no boundaries between characters whether it's a block of bytes on disk, network stream, or the clipboard buffer. It can be helpful to differentiate the various types of strings with a formal type system before the yanking takes place. The Alan Perlis epigram is wise but I don't think it applies here. I think's he's talking about abusing strings as an adhoc data format by stuffing them with multi-value data and then turning right around and parsing them. For example, concatenating name+":"+id+":"+amount into a string with colons like "John:389432:$5.99", passing the "complex" string around, and then later parsing out the colons with string.split(). Basically, the programmer is making an in-memory csv (comma separated values) data format instead of using "C struct {}" or passing 3 separate formal parameters.
- mannykannot 11y ago
  > ...before the yanking takes place. I take it that you are advocating for typed strings in addition to fully-formed types for things like URLs and SQL, but I think all that will lead to is a lot of gratuitous casting of strings just so they can be passed to the fully-formed types. Furthermore, few of these casts could be statically checked for validity.
  - jasode 11y ago
    >, but I think all that will lead to is a lot of gratuitous casting of strings But if programmers are not directly doing memcpy() of bytes directly to the stack frame, they are already doing a roundabout version of "casting". If a programmer has to "yank" from a string to put it into a more organized data type, the "yank" has to be done within the framework of semantics. If one "yanks" or "slices", or "substr", etc characters 3 to 15, the exact behavior depends on whether contiguous bytes is ASCII, UTF8, UTF16, or binary blob (which means NUL(0) is valid byte in the middle of the block of bytes). That "meaning overlayed onto those bytes to manipulate it correctly" has to decided somewhere. It is either implicit or explicit. Likewise, if one pulls 4 bytes from disk or network, the programmer has to assign meaning to it and say it is the 4-bytes of an IEEE floating point (float) or 4-bytes of a integer (int). The programmer may later want to do a bitshift (<<) which is not applicable to a float.

fizx 11y ago

/Regexp?/

mikeash 11y ago

Sort of. One spot of trouble is that NSString paths still exist and show up all over the place, even though NSURL is slowly replacing them. Another spot of trouble is that NSURL ends up in a similar situation, because sometimes NSURL means "an arbitrary URL" but often it means "a local file URL and nothing else." NSURL can hold arbitrary URLs with arbitrary schemas, but there are a ton of APIs in the frameworks which take NSURLs but only if they're file: URLs. It would be really nice to have a NSFileRef class or something like that which means, really, this is a reference to a local file, promise.

cballard 11y ago
Yeah, I feel like all of the NSString-based APIs should have been deprecated by now?