9 ms·

If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. • All the 2-byte-char languages which were designed for

If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. • All the 2-byte-char languages which were designed for UCS-2 before the Unicode Consortium pulled the rug out from underneath them and obsolesced constant-width UCS-2 in favor of variable-width UTF-16. • Languages which result in silent corruption when you concatenate encoded bytes with a string type (e.g. Perl but there are many examples.) • C, where NUL-terminated strings are the rule, and the standard library is of no help and so Unicode string handling needs to be built from scratch. All those checks which you have to fight to opt into, defying both the language and other lazy programmers (either inside your org, or at an org which develops dependencies you use)? Those checks either happen automatically or are much easier to use without making mistakes in Rust.

ajross 6y ago
Alternatively: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Obviously the world is a big place and there is room for lots of paradigms and worldviews and we aren't supposed to judge too much. But come on. If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period.
- spamizbad 6y ago
  It's the edge-cases that get you.
- magicalhippo 6y ago
  > if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. How so? Delphi for example has wide character-based strings as default, what's wrong with that?
  - estebank 6y ago
    The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as if ASCII was the only encoding in existence could continue treating strings in the same way (using their length to calculate their user-visible length, indexing directly on specific characters to change them, etc). All of those practices are immediately wrong once UTF-32 came in existence and UTF-16 became a variable length encoding. But even if that hadn't happened, what you want to be operating on is not characters, but grapheme clusters, which are equivalent to a vector of chars. Otherwise you won't handle the distinction between ë and ë or emojis correctly.
    - magicalhippo 6y ago
      But how is that different from the underlying encoding being UTF-8? edit: For example, we do a lot of string manipulation in Delphi. We might split a string in multiple pieces and glue them together again somehow. But our separators are fixed, say a tab character, or a semicolon. So this stiching and joining is oblivious to whatever emojis and other funky stuff that might be inbetween. How is this doing it wrong? I mean yea sure you CAN screw it up by individually manipulating characters. But I don't see how an UTF-8 encoded string in itself prevents you from doing the same kind of mistakes.
      - josephg 6y ago
        Splitting and glueing is fine. But imagine 3 systems: system A is obviously wrong. It crashes on any input. System B is subtly wrong. It works most of the time, but you’re getting reports that it crashes if you input Korean characters and you don’t know Korean or how to type those characters. System C is correct. Obviously C is better than A or B, because you want people to have a good experience with your software. But weirdly, system A (broken always) is usually better than system B (broken in weird hard to test ways). The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Code that is broken in subtle ways will get shipped and cause user frustration, churn, support calls, and so on. The problem with UCS-2 is it falls into system B. It works most of the time, for all the languages I can type. It breaks with some inputs I can’t type on my keyboard. So the bugs make it through to production. UTF-8 is more like system A than system B. You get multibyte code sequences as soon as you leave ASCII, so it’s easier to break. (Though it really took emoji for people to be serious about making everything work.)
        magicalhippo 6y ago
        Right, but the claim was this: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly I agree that UTF-8 is a better encoding overall for the majority of cases. I don't think that means UTF-16, which for example Delphi UnicodeStrings are[1], is not proper. edit: maybe this is a language confusion thing. For historically tragic reasons, we're stuck with "char" as the basic element of string types in lots of languages. In Delphi a "widechar" is technically a code unit[2], and may or may not represent a code point. This is how I interpreted the OP. Maybe he meant wide characters as code points, in which I would agree. [1]: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_in_RAD_Studio#New_String_Type:_UnicodeString http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_i... [2]: https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
  - barrkel 6y ago
    I was part of that. Delphi has all the string types you want, since you can declare your preferred code page. String is an alias for UnicodeString (to distinguish from COM WideString) and is UTF-16 for compatibility with Win32 API more than anything. UTF-8 would have meant a lot more temporaries and awkward memory management.
    - magicalhippo 6y ago
      All in all, while the Unicode transition took its time, I must admit it's was very smooth when it did happen. At work we have a codebase that does a lot of string handling. Both in reading and writing all kinds of text files, as well as doing string operations on entered data. Several hundred kLOC of code across the project. We had one guy who spent less than week wall-time to move the whole project, and the only issue we've had since is when other people send us crappy data... if I got a dollar for each XML file with encoding="utf-8" in the header and Windows-1252 encoded data we've received I'd have a fair fortune.
  - josephg 6y ago
    Wide character based strings have a .length field which is easy to reach for and never what you want, because it’s value is meaningless: - It isn’t the number of bytes, unless your string only contains ASCII characters. Works in testing, fails in production. - It isn’t the number of characters because 16 bits isn’t enough space to store the newer Unicode characters. And even if it could, many code sequences (eg emoji) turn multiple code points into a single glyph. I know all this, and I still get tripped up on a regular basis because .length is right there and works with simple strings I type. I have muscle memory. But no, in javascript at least the correct approaches require thought and sometimes pulling in libraries from npm to just make simple string operations be correct. Rust does the right thing here. Strings are UTF-8 internally. They check the encoding is valid when they’re created (so you always know if you have a string, it is valid). You have string.chars().count() and other standard ways to figure out byte length and codepoint length and all the other things you want to know, all right there, built into the standard library.
    - tsimionescu 6y ago
      What could string.chars().count() possibly be used for? At least .length tells ypu how much memory the string will occupy - not likely to be very important in a memory managed language, but at least it has one potential use. I don't see a use for the number of code points in a string.
      - josephg 6y ago
        In collaborative editing systems, we usually treat strings semantically as if they were as arrays of codepoints. Its the only sensible way to do it, because the other options are bad: - We don't use arrays of bytes because different languages have different native string encodings. Converting a UTF8 byte offset to a language that uses UCS2 is slow and complicated, and it opens the door to data corruption. - And we don't use grapheme clusters because what counts as a grapheme cluster keeps changing with each unicode version. (And libraries are big and complicated). So inserting an emoji into a document is treated the same way we would handle inserting a small list at some offset into a larger list. > At least .length tells you how much memory the string will occupy How much memory the string occupies isn't something I've ever wanted to know. I do sometimes want to know how many bytes the string will take up when I store it or send it over the network - but in those cases you pretty much always want UTF-8. And string.length doesn't help you at all with that.
- estebank 6y ago
  > if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Paradoxically, trying to do "the right thing" and being an "early adopter" of (the now called) UCS-2 was a "mistake", as both Java and Windows can attest, by getting "stuck" supporting the worst possible Unicode encoding ad-infinitum. UTF-8 is the "obviously correct" choice (from the hindsight afforded by us talking about this in 2021). I still find it funny that emojis of all things are what actually got the anglosphere to actually write software that isn't completely broken for the other 5.5 billion people out there.
  - lenkite 6y ago
    Java hasn't been UCS-2 since Java 5.0 - nearly a decade.
    - deleted 6y ago
      [deleted]
  - RcouF1uZ4gsC 6y ago
    > actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2. So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.
    - Quekid5 6y ago
      Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)
- diroussel 6y ago
  My understanding is that UTF-8 is not a good representation for non-european alphabets. So do you think UTF-8 is always the best internal string representation? Or just for English speakers? For Mandarine what would be optimal?
  - rectang 6y ago
    Mandarin is an interesting case. Most of the Han characters used by Mandarin fall within the basic multilingual plane and thus occupy 2 bytes in UTF-16 but 3 bytes in UTF-8. However, for web documents, most markup is ASCII which is only one byte. So for Mandarin web documents, the space requirements for UTF-8 and UTF-16 are about a wash. When you add in interoperability concerns, since so much text these days is UTF-8, for Mandarin at least UTF-8 is a perfectly defensible choice. (A harder problem is Japanese — Japan really got screwed over with Han unification, so choosing Shift-JIS over any Unicode encoding is often best.) FWIW I covered the space requirements of various encodings and various languages in this talk for Papers We Love Seattle: https://www.youtube.com/watch?v=mhvaeHoIE24&t=39m14s https://www.youtube.com/watch?v=mhvaeHoIE24&t=39m14s
    - amelius 6y ago
      I genuinely wonder: is the space requirement of text encodings really an important issue in this age of large photo and video content?
      - astrange 6y ago
        Memory for strings is often more important because there's a lot more of them, and image memory can be file-backed more often but strings need to be swapped to disk.
      - jcranmer 6y ago
        Looking at my browser's memory reporting, strings take up about ~2-3% of total memory usage, most of which is probably ASCII. If it were UCS-2, that would make it ~5% of total memory usage, and UCS-4 ~10%. That's small numbers, but as a whole-program impact, it's significant enough to motivate performance engineers to actually try to compress those strings down a bit.
- grishka 6y ago
  > If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period. UTF-8 is a nuisance as an in-memory representation because the characters are variable size. You can't get the length of a string without parsing it start to end, and you can't get a character by index without parsing and counting all the previous ones. 16-bit characters (wchar_t, Java char, whatever NSStrings are made of, etc) work fine in 99% of the cases. UTF-8 is indisputably a good encoding for when you're sending something over the network or putting it into a file or a database.
  - ChrisSD 6y ago
    UTF-16 is a variable length encoding. As is UTF-32 for that matter. A single character can be made of multiple code points.
    - TwoBit 6y ago
      True, but text processing is still easier with UTF-32, because UTF-8 and UTF-16 strings need to be converted to UTF-32 before you can do anything with them.
  - dwheeler 6y ago
    > 16-bit characters... work fine in 99% of the cases. In other words, they don't work :-). UTF-16 is also variable-length. Sometimes a character fits in 16 bits, and sometimes it doesn't. From a practical view it's worse than UTF-8, because tests are less likely to detect bugs before shipping. Even UTF-32 is, in reality, variable-length. Many code points are combining characters, so you need multiple code points to get a single grapheme. If your language or API requires you to do something, then you'll need to do that. But unless there's an API requirement, in most situations UTF-8 is the best choice for network, storage, and processing. There are exceptions, but they're just that... exceptions.
- samatman 6y ago
  I wouldn't go that far, despite having made an early decision to support only UTF-8 in my own software. UCS-2 is in fact broken, but UTF-16 is a valid encoding of Unicode, which can be implemented correctly. So is UTF-32, although I can't imagine why anyone would want to use that one. I can imagine why someone would want to use UTF-16, though: interoperability with Windows, where it's the native encoding. It isn't "wrong, period" to do Unicode in a way which is more convenient for the platform. There is, of course, a ton of work to really implement Unicode correctly, and UTF-16 and UTF-32 can make it tempting to do the wrong thing, instead of biting the bullet and implementing all of the many ways in which codepoints coalesce into grapheme clusters, and making sure all functions for working with strings can recognize the distinction. But it certainly can be done in any of the full encodings.
  - Joker_vD 6y ago
    > interoperability with Windows Even on Windows, it's much easier to just use UTF-8 internally and perform conversion to/from UTF-16 when you're about to do/return from a WinAPI call.
shadowgovt 6y ago
This is my way of thinking about the topic these days. It's not that strings are more complicated in Rust than in other languages, it's that a lot of the other low-level languages are presenting an abstraction that assumes implicitly that a string is some type of sequence of uniform-sized cells, one cell per character, and that representation was an artifact of a specific time in computational history. It's like many other abstractions those languages provide... Seemingly simple at first glance, but if you do the details wrong you're just going to get undefined behavior and your program will be incorrect. Languages that don't expose strings as that abstraction are, in my humble opinion, more reflective of the underlying concept in the modern era.
alerighi 6y ago
All of this is true, IF you assume that you want Unicode string. That, especially on system/embedded software (the kind of software that Rust is targeting) you don't really care about Unicode and you can simply treat strings as array of bytes. And I live in a country where you usually use Unicode characters. But for the purpose of the software that I write, I mostly stick with ASCII. For example I use strings to print debug messages to a serial terminal, or read commands from the serial terminal, or to put URL in the code, make HTTP requests, publish on MQTT topics... for all of these application I just use ASCII strings. Even if I have to represent something on the screen... as long as I have a compiler that supports Unicode as input files (all do these days) I can put Unicode string constants in the code and even print them on screen. It's the terminal (or the GUI I guess, but I don't write software with a GUI) that translates the bytes that I send on the line as Unicode characters. And yes, of course the length of the string doesn't correspond at the characters shown on the screen... but even with Unicode you cannot say that! You can count (and that what Rust does) how many Unicode code points you have, but a characters could be made of more code points (stupid example, the black emoji is composed by a code point that says "make the following black" and then the emoji itself). So to me it's pointless, and I care more about knowing how many bytes a string takes and being able to index the string in O(1), or take pointers in the middle of the string (useful when you are parsing some kind of structured data), and so on. In conclusion Rust is better when you have to handle Unicode string, but most application doesn't have to handle them, and handling them I don't mean passing them around as a black box, not caring how they contain (yes, in theory you should care about not truncating the string in the middle of a code point when truncating strings... in reality, how often do you truncate strings?)
- stouset 6y ago
  Literally nothing is stopping you from using `&[u8]` for 7-bit ASCII.
  - estebank 6y ago
    > So to me it's pointless, and I care more about knowing how many bytes a string takes and being able to index the string in O(1), or take pointers in the middle of the string (useful when you are parsing some kind of structured data), and so on. Which is why you can still sub-slice Strings[1] > Literally nothing is stopping you from using `&[u8]` for 7-bit ASCII. Not only that, there's first party support for literals that are that[2]. [1]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=fe1fa17cd5bfbe5b7d1e4a162a8c106b https://play.rust-lang.org/?version=stable&mode=debug&editio... [2]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=28b9b84ebf317f0e6bae4e00907239c3 https://play.rust-lang.org/?version=stable&mode=debug&editio...
tsimionescu 6y ago
What can you actually do with a known-valid UTF-8 (but otherwise of unkown structure) string that you can't do with a UTF-16 string or even byte-based string? You can't concatenate, split, enumerate, assume they are valid human text, capitalize, count the number of characters, turn to lower/uppercase etc. In general, you need a text handling library to do anything meaningful with text if you want to handle internationalization, and then the text handling library can also handle all of the encoding problems easily, all that matters is having a known encoding so you don't need to guess. It's nice to have specific types for specific encodings of strings, but otherwise I think there is little to be gained by representing strings as anything other than either byte arrays or text.
- mjevans 6y ago
  Incorrect. You CAN concatenate, assuming your inputs are valid the outputs will also be valid; BUT normalization is now ONLY correct if all inputs were of the same format. This can be fixed with libraries if you care, and if you don't, it often doesn't matter. Edit-Additional: If something DOES care, it SHOULD enforce the normalization format it wants on the input boundary. UTF-16 cannot be split any differently than any other encoding, including UTF-32; all must pass through a library that understands compositing characters and sequences. There is no escaping this (other than your own code becoming such a library). Most of the other issues you fault are shared by _any_ encoding of Unicode. However a notable thing is that for very specific functions, E.G. searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. To always do it without issue additional checks must be made around combining characters at the boundary edges. (Which I wouldn't want to maintain, so I'd still call a library unless matching against known will never be such a value, such as quotes or other configuration file control characters.)
  - tsimionescu 6y ago
    > You CAN concatenate, assuming your inputs are valid the outputs will also be valid Is that true even if you mix LTR and RTL text? I have a suspicion there could be problems there, but I'm happy to be wrong. I would still say that safe concatenation is a minor boon for the cost of enforced UTF-8 compliance. > Most of the other issues you fault are shared by _any_ encoding of Unicode. That is exactly my point - that UTF-8, UTF-16, UTF-32, byte strings that could be invalid UTF-8 are all basically equally bad if you want to guarantee meaningful text. A text consisting of two valid UTF-8 code points representing combining characters is no more meaningful than a text consisting of 4 bytes that are invalid UTF-8 code points. > searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. I don't agree, unless you are talking about something extremely 'programmatic' like someone's username or email. For real text apps, you'd want something even more complex than Unicode, that could recognize that 'Ștefan' and 'Stefan' are the same name.
WalterBright 6y ago
The D programming language has from the beginning built-in support for UTF-8, UTF-16 and UTF-32 code units as basic types (char, wchar, and dchar). This was when it wasn't clear which encoding would dominate. It's pretty clear today that UTF-8 dominates, and the other two are useful only for interfacing to systems that need them.
dhosek 6y ago
What's really interesting to me is that a rust character is 32 bits even though a rust string is encoded in UTF-8. I'm still just beginning to explore rust so I'd only gotten as far in the book as to learn about characters but not strings and their relationship to UTF-8. This bit of handling of Unicode has me especially intrigued now.
- mlindner 6y ago
  I believe the strings are still encoded internally as byte arrays however, it's just that when you pull out a character it can be multiple bytes (emojis for example are often 3 bytes), so you need a 4 byte datatype to store them.
  - dhosek 6y ago
    Yes, that's what I understand about how it works. There's a lot to be said against the idea of dealing with Unicode at the codepoint level in most applications though. I'm writing code that would allow commands to be either \ + non-letter character or \ + one or more letters (a la TeX). So that means that I would allow commands to include \bird, \pták, \طائر and \鳥, but what happens if the á in \pták is input as ´+ a rather than á? Is that supposed to be the same code? Or perhaps the user has \Spi¨nalTap (and there code editor has a type compositor that's willing to put the umlaut on the n?) Some of this can be dealt with through Unicode normalization, although there's also the question of whether, e.g., the ohm and angstrom symbols should be treated as symbols or letters if they're input at their symbolic code points rather than as a Greek or Latin letters. Would \+white man shrugging be a valid \+non-letter command since it's multiple code points? It's amazing how much a "simple" specification gets complicated when you start to look at all the ways that Unicode can complicate matters.

ajross 6y ago

Alternatively: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Obviously the world is a big place and there is room for lots of paradigms and worldviews and we aren't supposed to judge too much. But come on. If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period.

spamizbad 6y ago
It's the edge-cases that get you.
magicalhippo 6y ago
> if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. How so? Delphi for example has wide character-based strings as default, what's wrong with that?
- estebank 6y ago
  The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as if ASCII was the only encoding in existence could continue treating strings in the same way (using their length to calculate their user-visible length, indexing directly on specific characters to change them, etc). All of those practices are immediately wrong once UTF-32 came in existence and UTF-16 became a variable length encoding. But even if that hadn't happened, what you want to be operating on is not characters, but grapheme clusters, which are equivalent to a vector of chars. Otherwise you won't handle the distinction between ë and ë or emojis correctly.
  - magicalhippo 6y ago
    But how is that different from the underlying encoding being UTF-8? edit: For example, we do a lot of string manipulation in Delphi. We might split a string in multiple pieces and glue them together again somehow. But our separators are fixed, say a tab character, or a semicolon. So this stiching and joining is oblivious to whatever emojis and other funky stuff that might be inbetween. How is this doing it wrong? I mean yea sure you CAN screw it up by individually manipulating characters. But I don't see how an UTF-8 encoded string in itself prevents you from doing the same kind of mistakes.
    - josephg 6y ago
      Splitting and glueing is fine. But imagine 3 systems: system A is obviously wrong. It crashes on any input. System B is subtly wrong. It works most of the time, but you’re getting reports that it crashes if you input Korean characters and you don’t know Korean or how to type those characters. System C is correct. Obviously C is better than A or B, because you want people to have a good experience with your software. But weirdly, system A (broken always) is usually better than system B (broken in weird hard to test ways). The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Code that is broken in subtle ways will get shipped and cause user frustration, churn, support calls, and so on. The problem with UCS-2 is it falls into system B. It works most of the time, for all the languages I can type. It breaks with some inputs I can’t type on my keyboard. So the bugs make it through to production. UTF-8 is more like system A than system B. You get multibyte code sequences as soon as you leave ASCII, so it’s easier to break. (Though it really took emoji for people to be serious about making everything work.)
      - magicalhippo 6y ago
        Right, but the claim was this: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly I agree that UTF-8 is a better encoding overall for the majority of cases. I don't think that means UTF-16, which for example Delphi UnicodeStrings are[1], is not proper. edit: maybe this is a language confusion thing. For historically tragic reasons, we're stuck with "char" as the basic element of string types in lots of languages. In Delphi a "widechar" is technically a code unit[2], and may or may not represent a code point. This is how I interpreted the OP. Maybe he meant wide characters as code points, in which I would agree. [1]: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_in_RAD_Studio#New_String_Type:_UnicodeString http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_i... [2]: https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
- barrkel 6y ago
  I was part of that. Delphi has all the string types you want, since you can declare your preferred code page. String is an alias for UnicodeString (to distinguish from COM WideString) and is UTF-16 for compatibility with Win32 API more than anything. UTF-8 would have meant a lot more temporaries and awkward memory management.
  - magicalhippo 6y ago
    All in all, while the Unicode transition took its time, I must admit it's was very smooth when it did happen. At work we have a codebase that does a lot of string handling. Both in reading and writing all kinds of text files, as well as doing string operations on entered data. Several hundred kLOC of code across the project. We had one guy who spent less than week wall-time to move the whole project, and the only issue we've had since is when other people send us crappy data... if I got a dollar for each XML file with encoding="utf-8" in the header and Windows-1252 encoded data we've received I'd have a fair fortune.
- josephg 6y ago
  Wide character based strings have a .length field which is easy to reach for and never what you want, because it’s value is meaningless: - It isn’t the number of bytes, unless your string only contains ASCII characters. Works in testing, fails in production. - It isn’t the number of characters because 16 bits isn’t enough space to store the newer Unicode characters. And even if it could, many code sequences (eg emoji) turn multiple code points into a single glyph. I know all this, and I still get tripped up on a regular basis because .length is right there and works with simple strings I type. I have muscle memory. But no, in javascript at least the correct approaches require thought and sometimes pulling in libraries from npm to just make simple string operations be correct. Rust does the right thing here. Strings are UTF-8 internally. They check the encoding is valid when they’re created (so you always know if you have a string, it is valid). You have string.chars().count() and other standard ways to figure out byte length and codepoint length and all the other things you want to know, all right there, built into the standard library.
  - tsimionescu 6y ago
    What could string.chars().count() possibly be used for? At least .length tells ypu how much memory the string will occupy - not likely to be very important in a memory managed language, but at least it has one potential use. I don't see a use for the number of code points in a string.
    - josephg 6y ago
      In collaborative editing systems, we usually treat strings semantically as if they were as arrays of codepoints. Its the only sensible way to do it, because the other options are bad: - We don't use arrays of bytes because different languages have different native string encodings. Converting a UTF8 byte offset to a language that uses UCS2 is slow and complicated, and it opens the door to data corruption. - And we don't use grapheme clusters because what counts as a grapheme cluster keeps changing with each unicode version. (And libraries are big and complicated). So inserting an emoji into a document is treated the same way we would handle inserting a small list at some offset into a larger list. > At least .length tells you how much memory the string will occupy How much memory the string occupies isn't something I've ever wanted to know. I do sometimes want to know how many bytes the string will take up when I store it or send it over the network - but in those cases you pretty much always want UTF-8. And string.length doesn't help you at all with that.
estebank 6y ago
> if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Paradoxically, trying to do "the right thing" and being an "early adopter" of (the now called) UCS-2 was a "mistake", as both Java and Windows can attest, by getting "stuck" supporting the worst possible Unicode encoding ad-infinitum. UTF-8 is the "obviously correct" choice (from the hindsight afforded by us talking about this in 2021). I still find it funny that emojis of all things are what actually got the anglosphere to actually write software that isn't completely broken for the other 5.5 billion people out there.
- lenkite 6y ago
  Java hasn't been UCS-2 since Java 5.0 - nearly a decade.
  - deleted 6y ago
    [deleted]
- RcouF1uZ4gsC 6y ago
  > actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2. So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.
  - Quekid5 6y ago
    Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)
diroussel 6y ago
My understanding is that UTF-8 is not a good representation for non-european alphabets. So do you think UTF-8 is always the best internal string representation? Or just for English speakers? For Mandarine what would be optimal?
- rectang 6y ago
  Mandarin is an interesting case. Most of the Han characters used by Mandarin fall within the basic multilingual plane and thus occupy 2 bytes in UTF-16 but 3 bytes in UTF-8. However, for web documents, most markup is ASCII which is only one byte. So for Mandarin web documents, the space requirements for UTF-8 and UTF-16 are about a wash. When you add in interoperability concerns, since so much text these days is UTF-8, for Mandarin at least UTF-8 is a perfectly defensible choice. (A harder problem is Japanese — Japan really got screwed over with Han unification, so choosing Shift-JIS over any Unicode encoding is often best.) FWIW I covered the space requirements of various encodings and various languages in this talk for Papers We Love Seattle: https://www.youtube.com/watch?v=mhvaeHoIE24&t=39m14s https://www.youtube.com/watch?v=mhvaeHoIE24&t=39m14s
  - amelius 6y ago
    I genuinely wonder: is the space requirement of text encodings really an important issue in this age of large photo and video content?
    - astrange 6y ago
      Memory for strings is often more important because there's a lot more of them, and image memory can be file-backed more often but strings need to be swapped to disk.
    - jcranmer 6y ago
      Looking at my browser's memory reporting, strings take up about ~2-3% of total memory usage, most of which is probably ASCII. If it were UCS-2, that would make it ~5% of total memory usage, and UCS-4 ~10%. That's small numbers, but as a whole-program impact, it's significant enough to motivate performance engineers to actually try to compress those strings down a bit.
grishka 6y ago
> If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period. UTF-8 is a nuisance as an in-memory representation because the characters are variable size. You can't get the length of a string without parsing it start to end, and you can't get a character by index without parsing and counting all the previous ones. 16-bit characters (wchar_t, Java char, whatever NSStrings are made of, etc) work fine in 99% of the cases. UTF-8 is indisputably a good encoding for when you're sending something over the network or putting it into a file or a database.
- ChrisSD 6y ago
  UTF-16 is a variable length encoding. As is UTF-32 for that matter. A single character can be made of multiple code points.
  - TwoBit 6y ago
    True, but text processing is still easier with UTF-32, because UTF-8 and UTF-16 strings need to be converted to UTF-32 before you can do anything with them.
- dwheeler 6y ago
  > 16-bit characters... work fine in 99% of the cases. In other words, they don't work :-). UTF-16 is also variable-length. Sometimes a character fits in 16 bits, and sometimes it doesn't. From a practical view it's worse than UTF-8, because tests are less likely to detect bugs before shipping. Even UTF-32 is, in reality, variable-length. Many code points are combining characters, so you need multiple code points to get a single grapheme. If your language or API requires you to do something, then you'll need to do that. But unless there's an API requirement, in most situations UTF-8 is the best choice for network, storage, and processing. There are exceptions, but they're just that... exceptions.
samatman 6y ago
I wouldn't go that far, despite having made an early decision to support only UTF-8 in my own software. UCS-2 is in fact broken, but UTF-16 is a valid encoding of Unicode, which can be implemented correctly. So is UTF-32, although I can't imagine why anyone would want to use that one. I can imagine why someone would want to use UTF-16, though: interoperability with Windows, where it's the native encoding. It isn't "wrong, period" to do Unicode in a way which is more convenient for the platform. There is, of course, a ton of work to really implement Unicode correctly, and UTF-16 and UTF-32 can make it tempting to do the wrong thing, instead of biting the bullet and implementing all of the many ways in which codepoints coalesce into grapheme clusters, and making sure all functions for working with strings can recognize the distinction. But it certainly can be done in any of the full encodings.
- Joker_vD 6y ago
  > interoperability with Windows Even on Windows, it's much easier to just use UTF-8 internally and perform conversion to/from UTF-16 when you're about to do/return from a WinAPI call.

shadowgovt 6y ago

This is my way of thinking about the topic these days. It's not that strings are more complicated in Rust than in other languages, it's that a lot of the other low-level languages are presenting an abstraction that assumes implicitly that a string is some type of sequence of uniform-sized cells, one cell per character, and that representation was an artifact of a specific time in computational history. It's like many other abstractions those languages provide... Seemingly simple at first glance, but if you do the details wrong you're just going to get undefined behavior and your program will be incorrect. Languages that don't expose strings as that abstraction are, in my humble opinion, more reflective of the underlying concept in the modern era.

alerighi 6y ago

All of this is true, IF you assume that you want Unicode string. That, especially on system/embedded software (the kind of software that Rust is targeting) you don't really care about Unicode and you can simply treat strings as array of bytes. And I live in a country where you usually use Unicode characters. But for the purpose of the software that I write, I mostly stick with ASCII. For example I use strings to print debug messages to a serial terminal, or read commands from the serial terminal, or to put URL in the code, make HTTP requests, publish on MQTT topics... for all of these application I just use ASCII strings. Even if I have to represent something on the screen... as long as I have a compiler that supports Unicode as input files (all do these days) I can put Unicode string constants in the code and even print them on screen. It's the terminal (or the GUI I guess, but I don't write software with a GUI) that translates the bytes that I send on the line as Unicode characters. And yes, of course the length of the string doesn't correspond at the characters shown on the screen... but even with Unicode you cannot say that! You can count (and that what Rust does) how many Unicode code points you have, but a characters could be made of more code points (stupid example, the black emoji is composed by a code point that says "make the following black" and then the emoji itself). So to me it's pointless, and I care more about knowing how many bytes a string takes and being able to index the string in O(1), or take pointers in the middle of the string (useful when you are parsing some kind of structured data), and so on. In conclusion Rust is better when you have to handle Unicode string, but most application doesn't have to handle them, and handling them I don't mean passing them around as a black box, not caring how they contain (yes, in theory you should care about not truncating the string in the middle of a code point when truncating strings... in reality, how often do you truncate strings?)

stouset 6y ago
Literally nothing is stopping you from using `&[u8]` for 7-bit ASCII.
- estebank 6y ago
  > So to me it's pointless, and I care more about knowing how many bytes a string takes and being able to index the string in O(1), or take pointers in the middle of the string (useful when you are parsing some kind of structured data), and so on. Which is why you can still sub-slice Strings[1] > Literally nothing is stopping you from using `&[u8]` for 7-bit ASCII. Not only that, there's first party support for literals that are that[2]. [1]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=fe1fa17cd5bfbe5b7d1e4a162a8c106b https://play.rust-lang.org/?version=stable&mode=debug&editio... [2]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=28b9b84ebf317f0e6bae4e00907239c3 https://play.rust-lang.org/?version=stable&mode=debug&editio...

tsimionescu 6y ago

What can you actually do with a known-valid UTF-8 (but otherwise of unkown structure) string that you can't do with a UTF-16 string or even byte-based string? You can't concatenate, split, enumerate, assume they are valid human text, capitalize, count the number of characters, turn to lower/uppercase etc. In general, you need a text handling library to do anything meaningful with text if you want to handle internationalization, and then the text handling library can also handle all of the encoding problems easily, all that matters is having a known encoding so you don't need to guess. It's nice to have specific types for specific encodings of strings, but otherwise I think there is little to be gained by representing strings as anything other than either byte arrays or text.

mjevans 6y ago
Incorrect. You CAN concatenate, assuming your inputs are valid the outputs will also be valid; BUT normalization is now ONLY correct if all inputs were of the same format. This can be fixed with libraries if you care, and if you don't, it often doesn't matter. Edit-Additional: If something DOES care, it SHOULD enforce the normalization format it wants on the input boundary. UTF-16 cannot be split any differently than any other encoding, including UTF-32; all must pass through a library that understands compositing characters and sequences. There is no escaping this (other than your own code becoming such a library). Most of the other issues you fault are shared by _any_ encoding of Unicode. However a notable thing is that for very specific functions, E.G. searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. To always do it without issue additional checks must be made around combining characters at the boundary edges. (Which I wouldn't want to maintain, so I'd still call a library unless matching against known will never be such a value, such as quotes or other configuration file control characters.)
- tsimionescu 6y ago
  > You CAN concatenate, assuming your inputs are valid the outputs will also be valid Is that true even if you mix LTR and RTL text? I have a suspicion there could be problems there, but I'm happy to be wrong. I would still say that safe concatenation is a minor boon for the cost of enforced UTF-8 compliance. > Most of the other issues you fault are shared by _any_ encoding of Unicode. That is exactly my point - that UTF-8, UTF-16, UTF-32, byte strings that could be invalid UTF-8 are all basically equally bad if you want to guarantee meaningful text. A text consisting of two valid UTF-8 code points representing combining characters is no more meaningful than a text consisting of 4 bytes that are invalid UTF-8 code points. > searching for a given valid Unicode sequence and replacing it with another... (E.G. replace someone's name) you'll nearly always be able to do this in without issue. I don't agree, unless you are talking about something extremely 'programmatic' like someone's username or email. For real text apps, you'd want something even more complex than Unicode, that could recognize that 'Ștefan' and 'Stefan' are the same name.

WalterBright 6y ago

The D programming language has from the beginning built-in support for UTF-8, UTF-16 and UTF-32 code units as basic types (char, wchar, and dchar). This was when it wasn't clear which encoding would dominate. It's pretty clear today that UTF-8 dominates, and the other two are useful only for interfacing to systems that need them.

dhosek 6y ago

What's really interesting to me is that a rust character is 32 bits even though a rust string is encoded in UTF-8. I'm still just beginning to explore rust so I'd only gotten as far in the book as to learn about characters but not strings and their relationship to UTF-8. This bit of handling of Unicode has me especially intrigued now.

mlindner 6y ago
I believe the strings are still encoded internally as byte arrays however, it's just that when you pull out a character it can be multiple bytes (emojis for example are often 3 bytes), so you need a 4 byte datatype to store them.
- dhosek 6y ago
  Yes, that's what I understand about how it works. There's a lot to be said against the idea of dealing with Unicode at the codepoint level in most applications though. I'm writing code that would allow commands to be either \ + non-letter character or \ + one or more letters (a la TeX). So that means that I would allow commands to include \bird, \pták, \طائر and \鳥, but what happens if the á in \pták is input as ´+ a rather than á? Is that supposed to be the same code? Or perhaps the user has \Spi¨nalTap (and there code editor has a type compositor that's willing to put the umlaut on the n?) Some of this can be dealt with through Unicode normalization, although there's also the question of whether, e.g., the ohm and angstrom symbols should be treated as symbols or letters if they're input at their symbolic code points rather than as a Greek or Latin letters. Would \+white man shrugging be a valid \+non-letter command since it's multiple code points? It's amazing how much a "simple" specification gets complicated when you start to look at all the ways that Unicode can complicate matters.