4 ms·

This looks like good information about the memory representation, but we would be better off without string as a data type at all, at least not as a concrete ty

This looks like good information about the memory representation, but we would be better off without string as a data type at all, at least not as a concrete type. Strings as they commonly work are leaky abstractions which for decades have bred confusion among programmers over how to handle text properly. Text/character encoding bugs are a matter of type safety. I believe most of the text encoding bugs in the history of software could have been avoided if we used our type systems to handle text encodings explicitly. UTF-8, UTF-16, UTF-32, Windows-1252, ISO-8859-1, US-ASCII, et al.; these should be our essential concrete data types. The concept of a string should just be an abstraction over them for polymorphism. At no point in the lifecycle of a piece of textual data should there ever be any question over how it is encoded. So why don't we let our type systems do this bookkeeping for us?

svat 8y ago
I have written some programs dealing with Indic scripts, and when working with strings I prefer to see them only as a sequence of Unicode codepoints; it's a clean abstraction compared to the encodings which are messy (and half of them don't even support the scripts I care about). While text is in memory inside your programming-language runtime, you don't care about the encoding; for all I know it may not even be one of those encodings. You only have to deal with encodings when dealing with the "external world": when you fetch (or send) bytes from (or to) disk / network / the user. These are anyway out of the scope of the type system, so the type system wouldn't help you: the encoding needs to be communicated to you out-of-band. (See https://nedbatchelder.com/text/unipain.html https://nedbatchelder.com/text/unipain.html for the “Unicode sandwich” approach: convert to bytes at the boundaries of your system.) So using a type system to specify encodings isn't really helpful; what a type system can help you with is clearly distinguishing between bytes and text (separate types), and forcing you to specify an encoding when converting between them.
- ademarre 8y ago
  > I prefer to see them only as a sequence of Unicode codepoints You can still have this with encodings as data types. Encoding is definitely a concern at the boundary, but to say that it is a concern only at the boundary is too simplistic. What if you want to save the cost of converting encodings from that which was input at the boundary? And at some point data will have to leave your application, and this often requires munging text in a number of encodings. > So using a type system to specify encodings isn't really helpful Imagine an error at compile time when you accidentally try to append an ISO-8859-1 string to a UTF-8 string.

svat 8y ago

I have written some programs dealing with Indic scripts, and when working with strings I prefer to see them only as a sequence of Unicode codepoints; it's a clean abstraction compared to the encodings which are messy (and half of them don't even support the scripts I care about). While text is in memory inside your programming-language runtime, you don't care about the encoding; for all I know it may not even be one of those encodings. You only have to deal with encodings when dealing with the "external world": when you fetch (or send) bytes from (or to) disk / network / the user. These are anyway out of the scope of the type system, so the type system wouldn't help you: the encoding needs to be communicated to you out-of-band. (See https://nedbatchelder.com/text/unipain.html https://nedbatchelder.com/text/unipain.html for the “Unicode sandwich” approach: convert to bytes at the boundaries of your system.) So using a type system to specify encodings isn't really helpful; what a type system can help you with is clearly distinguishing between bytes and text (separate types), and forcing you to specify an encoding when converting between them.

ademarre 8y ago
> I prefer to see them only as a sequence of Unicode codepoints You can still have this with encodings as data types. Encoding is definitely a concern at the boundary, but to say that it is a concern only at the boundary is too simplistic. What if you want to save the cost of converting encodings from that which was input at the boundary? And at some point data will have to leave your application, and this often requires munging text in a number of encodings. > So using a type system to specify encodings isn't really helpful Imagine an error at compile time when you accidentally try to append an ISO-8859-1 string to a UTF-8 string.