4 ms·
Rust is probably one of the languages which does this crap best, and that's thanks to static typing and deciding to not decide: 1. it has proper, validated uni
by masklinn 8y ago
Rust is probably one of the languages which does this crap best, and that's thanks to static typing and deciding to not decide:
1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal)
2. it has proper bytes, entirely separate from strings
3. it has "the OS layer is a giant pile of shit" OsString, because file paths might be random bag of bytes (UNIX) or random bags of 16-bit values (and possibly some other hare-brained scheme on other platforms but I don't believe rust supports other osstrings currently)
4. and it has nul-terminated bag o'bytes CString
For the latter two, conversion to a "proper" language string is explicitly known to be lossy, and the developer has to decide what to do in that case for their application.
- CyberShadow 8y ago> 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) Sigh, I hoped newer languages would avoid D's mistake. Auto-decoding is slow, unnecessary in most cases, and still only gives you partial correctness, depending on what you're trying to do. It also means that even the simplest string operations may fail, which has consequences on the complexity of the API.
- nikic 8y agoThat sounds more like an implementation issue than a design issue. If you are using UTF-8 actual decoding into Unicode code points is not necessary for most operations and Rust will not do that. It also does not imply that string operations may fail. String construction from raw bytes may fail, but otherwise the use of UTF-8 strings should not introduce additional failure conditions.
- masklinn 8y agoI have no idea what you're talking about. Rust rarely does auto- anything, and certainly does not decode (or transcode) strings without an explicit request by the developer: Rust strings are not inherently iterable. As developer, you specifically request an iterable on code units or code points (or grapheme clusters or words through https://unicode-rs.github.io/unicode-segmentation https://unicode-rs.github.io/unicode-segmentation)
- CyberShadow 8y agoI see, thanks for the clarification - looks like I mis-extrapolated from your comment.
- jcranmer 8y ago> 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) Grapheme clusters are overrated in their importance for processing. The list of times you want to iterate over grapheme clusters: 1. You want to figure out where to position the cursor when you hit left or right. 2. You want to reverse a string. (When was the last time you wanted to do that?) The list of times when you want to iterate over Unicode codepoints: 1. When you're implementing collation, grapheme cluster searching, case modification, normalization, line breaking, word breaking, or any other Unicode algorithm. 2. When you're trying to break text into separate RFC 2047 encoded-words. 3. When you're trying to display the fonts for a Unicode string. 4. When you're trying to convert between charsets. Cases where neither is appropriate: 1. When you want to break text to separate lines on the screen. 2. When you want to implement basic hashing/equality checks. (I'm not sure where "cut the string down to 5 characters because we're out of display room" falls in this list. I suspect the actual answer is "wrong question, think about the problem differently"). Grapheme clusters is relatively expensive to compute, and its utility is very circumscribed. Iterating over Unicode codepoints is much more useful and foundational and yet still very cheap.
- masklinn 8y agoSo? Rust's stdlib provides iteration on code units and code points. The use cases where these are useful are covered. It does not provide iteration on grapheme clusters, the use cases where this is useful are not covered (and require an external dependency). At no point am I requesting replacing codepoints-wise iteration by clusters-wise iteration.
- zbentley 8y ago> Grapheme clusters are overrated in their importance for processing. The list of times you want to iterate over grapheme clusters: > 1. You want to figure out where to position the cursor when you hit left or right. > 2. You want to reverse a string. (When was the last time you wanted to do that?) You missed the big one: 3. You want to determine the logical (and often visual) length of a string. Sure, there are some languages where logical-length is less meaningful as a concept, but there are many, many languages in which it's a useful concept, and can only be easily derived by iterating grapheme clusters.