9 ms·
String types in Rust
- stcredzero 11y agoIn this post I’m going to explain the organization of Rust’s string types with String and str as examples, then get into the lesser-used string types—CString, CStr, OsString, OsStr, PathBuf, and Path Ouch. So one thing that Rust is not going to succeed at is not having the chaos around string types that C++ has. Why is it that some languages can get away with just having 1 unicode string type with libraries to translate to other representations as needed, but not others?
- Etzos 11y ago> Why is it that some languages can get away with just having 1 unicode string type with libraries to translate to other representations as needed, but not others? Actually that's almost exactly what Rust is doing. Most of the types of strings you mentioned are added from the FFI library which is used to create bindings to other languages. They are a necessity to convert between the representations nicely.
- jdmichal 11y agoTo be fair, Java and C# -- two languages that you would probably consider as having one unicode string type -- also effectively have two string types between `String` and `StringBuilder`. [0] Java also has `java.nio.file.Path`. [1] [0] Java also manages to conflate synchronization into the mix with `StringBuffer`. [1] C# (thankfully) just has static functions to manipulate strings as paths.
- llogiq 11y agoIf you want to add IO into Java's mix, you can also use StringWriter. Also I personally don't consider paths as strings, nor URLs – both are types for specific purposes with specific invariants, which are distinct from strings'.
- Pxtl 11y agoAgreed, as long as the language provides quick and fast go/from string mechanism, great them as their own objects. What drives me batty is when I know something is just a string under the hood but getting my string into that object is a nightmare and inconsistent.
- jdmichal 11y agoI suppose it depends on your definition of a string. If it's an ordered collection of glyphs, than paths and URLs are definitely strings with invariant restrictions. But, of course, that's hardly a useful definition when you need to take a URL as input, is it? So I agree that treating them as separate types with an external string representation is a very useful tool to have, even if I don't agree that it should always be applied! Having worked with the Path class in Java, I personally find it an impediment to understandable code, but I'd also be willing to accept that that particular implementation and my intuition simply don't mesh well.
- lomnakkus 11y ago> I suppose it depends on your definition of a string. If it's an ordered collection of glyphs, than paths and URLs are definitely strings with invariant restrictions. That's not quite true -- on POSIX paths are strings of bytes. POSIX doesn't specify any particular encoding[1]. AFAIR the only restrictions on paths are around the NUL (0) byte and '/' (47). [1] I'm sure we'd all be happier if Unicode (encoded as UTF-8) were specified as the way to do paths, but POSIX was invented quite a bit before Unicode & UTF-8, so here we are.
- Gankro 11y agoAnd on Windows it's "potentially ill-formed UTF-16": https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Basically if your language/std-lib claims to be cross platform and considers paths to be definitely-utf8, it's busted. I think, maybe, plan9 is a case where this claim holds?
- 11y ago
- Pxtl 11y agoHonestly, having coded a lot of C# path code, I'd rather an object that properly encapsulates this, as long as there's easy APIs to get a string copy from it and vice versa - I mean, you're going to be throwing away oodles of string refs anyways since they're immutable, so one more for the path conversion wouldn't hurt and the path API would be a lot cleaner.
- deleted 11y ago[deleted]
- tatterdemalion 11y agoString handling in Rust is definitely not as accessible as some other languages, but none of the complexity is incidental. The String/str split is actually fundamental and pervasive, but internally all string operations are performed on these two types. Path represents a file system path, so it has a totally different API, and CString and OsString are for translating to other representations in external systems (libc and the OS, of course), not for use outside of that context. The String/str divide is the unavoidable consequence of not having garbage collection. A String is owned, a str is borrowed. This blog post goes into more information. This is nowhere near the chaos that C++ has.
- eridius 11y agoYou can avoid the split without garbage collection, e.g. Swift has a single String type which is a retain-counted copy-on-write type. But this does mean that working with strings in Swift can have hidden costs (e.g. copies where you might not expect them), plus the cost of atomic reference counting. So this solution isn't appropriate for a language like Rust.
- tatterdemalion 11y agoYou're right, I meant "with Rust's memory management story," which excludes universal reference counting. People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics.
- pcwalton 11y ago> People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics. Which is pretty unfortunate, because the memory management field has always treated reference counting as a form of garbage collection—because it is [1] [2]. The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) This is not just a semantic quibble, because there are successful garbage collection techniques that combine tracing with reference counting (e.g. ulterior reference counting [3]) to achieve GC, so treating tracing as the only way to do GC makes no sense. In fact, David Bacon observed in 2004 that reference counting and tracing garbage collection are best viewed as just two extremes of a continuum [4], and many "optimizations" that you apply to one scheme or the other are really just moving the scheme you choose toward the opposite pole. [1]: http://www.memorymanagement.org/glossary/g.html#term-garbage-collection http://www.memorymanagement.org/glossary/g.html#term-garbage... [2]: http://researcher.watson.ibm.com/researcher/files/us-bacon/Bacon03Pure.pdf http://researcher.watson.ibm.com/researcher/files/us-bacon/B... [3]: http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-2003.pdf http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-20... [4]: https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf
- geodel 11y agoI think low level systems programing language need that.
- gnuvince 11y agoString is the heap-allocated sequence of characters, str is a view into a String. Strings in Rust are guaranteed to be valid UTF-8; file paths in Windows are not, so you cannot use String for those, so you need another type. Repeat for different "stringy" types that do not have the same semantics as Strings and you end up with a number of string types.
- steveklabnik 11y ago> String is the heap-allocated sequence of characters, To be clear, String is a heap-allocated sequence of u8 which form valid unicode scalar values. "character" in Rust usually means "char", and String is _not_ a list of chars.
- twoodfin 11y agoIn my experience, it is absolutely vital for performance to have both "owning" and "slice"/"view" string types, as well as sensible conversions between them. This is actually an area where the C++ standard library has been lacking until recently. For example, you'd typically want a map with string keys to own them, but without severe contortions that meant map readers had to allocate a string key (expensive!) to perform a lookup.
- pcwalton 11y agoThree reasons why Rust cannot adopt Go's solution: 1. Rust is not garbage collected. This means that a separation between strings-that-own-their-data and strings-that-reference-their-data is needed. 2. Rust cares about first-class FFI support; in particular, copying strings at FFI boundaries is right out. That is why Rust needs C strings to be a separate type. 3. Rust wants to do pathnames right in a cross-platform way. In particular, it should be impossible for other apps to create valid pathnames that cause Rust apps manipulating those paths to crash or malfunction. UTF-8 paths fail this criterion.
- PopsiclePete 11y agoPoint 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. Very cool to show off to your geek programmer friends and impress them, but I really question the day-to-day usefulness when considering the added complexity. I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. In a way, I'm a little sad that modern computer languages are going from C's extreme of "always trust the programmer" to the other of "never trust the programmer". Bugs are bad, but language usability, ease-of-use and simplicity are also worth fighting for. Having said all that, I am really happy to see Rust being awesome in the general case and appreciate all the work you and the rest of the team are putting forth.
- oldmanjay 11y agoI definitely fall on the other side of the fence, philosophically. Perhaps I've just fixed too many of those "trust me" bugs that programmers are constantly adding into code I have to maintain. I'm especially over the thinking that sufficient vigilance is something we should rely on humans to achieve when the computer is way, way better at it.
- vessenes 11y ago> 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I think that's precisely the point; at scale, and with today's modular software culture (like last week's left-pad news cycle demonstrates), 99% correct software can mean millions of installations of bad code, all vulnerable to attack, or even just 1%-type bugs. One way to think about the last 10 years of language design is an ongoing dialogue on how much or little to trust engineer intuition about which tools are safe for which jobs. The pendulum is swinging towards the belief that in general engineers should be given much more safety-oriented tools.
- burntsushi 11y agoAs the author of Rust's regex crate (and various other stringy things), Rust's string handling is easily the best I've ever used. The usage of String/&str completely dwarfs usage of the other string types, which are a bit more specialized.
- agumonkey 11y agoHaving recently read about Pascal string types http://wiki.freepascal.org/Character_and_string_types http://wiki.freepascal.org/Character_and_string_types this feels very interesting.
- thrownaway2424 11y agoYeah if nothing else the naming here could be better. If someone writes a blog post where the correspondence between String and ownership has to be repeated fifty times, it might have been better to name them OwnedString and StringView, or something like that.
- Manishearth 11y agoNote that everything but String and str are not used that often, and when they are, its straightforward. You'll probably use them in cases like `File::new(Path::new(my_string))` with some error handling if necessary, and forget that the intermediate existed. In fact, many of these APIs use std::convert trait magic so that you don't need to manually convert to the intermediate type at all. You also see these string types used when they are read, stored, and reused later. I haven't seen many cases of string-type-specific gymnastics being done, except sometimes with paths (which you can treat as a list of characters as well as a list of segments).
- nieksand 11y agoMy favorite bit of learning with Rust strings was the WTF-8 format: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Implemented: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/common/wtf8.rs https://github.com/rust-lang/rust/blob/master/src/libstd/sys...
- kibwen 11y agoWTF-8 is great, and lest anyone think that it's intended to be a joke it's worth mentioning that it's actually a very useful thing: "WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired." Though note that it's only suitable for internal usage (as in, you wouldn't expose an API that uses WTF-8), so it's not something that you need to actually think about when learning Rust. :P
- derefr 11y agoI don't really get the point of it; I'd think that, properly, what you'd want is: 1. a plain "buffer of unparsed octets" type (like Erlang's "binary" type), to hold onto your weird invalid UTF-16 (or whatever other weird invalid stuff you like); and 2. a strict, validated-on-construction "sequence of codepoints" type. Going from #1 to #2 wouldn't be a direct cast; it'd instead be a stream-parser function that could be hooked with a callback, which would be handed parsed-and-valid, and invalid-and-left-unparsed, "segments": maybe_parsed_segment() = valid_string_of_codepoints() | binary() ...and would then respond with a valid segment, which could just be "" to discard the input segment. The default callback would presumably respond to a binary() of size N by emitting a "�" string of length N. That's pretty much what we have today, but both sturdier and more extensible.
- brinker 11y agoHey, author here! Happy to answer any questions people have, and happy to hear any feedback.
- imron 11y agoMaybe consider a word other than 'sort'. When I see 'sort' collocated with 'string' my mind immediately jumps to the verb meaning of 'sort', not the noun meaning of 'sort', so it was a bit confusing.
- rkangel 11y agoYes I had that too. I would have used "variety", as in this "variety" of string. Or "kind" maybe.
- llogiq 11y agoYeah, the word you look for isn't "sort", but "kind".
- brinker 11y agoI shied away from the word "kind" simply because that has a specific meaning in type theory that doesn't fit here. Higher-kinded types are a must-requested feature for Rust, and I didn't want to confuse readers into thinking this has some connection to Rust's string types.
- kibwen 11y agoThe word "sort" isn't free of alternate associations either, and "kind" seems less likely to confuse people, seeing as Rust doesn't have any concept called "kinds" right now and won't have HKT for years, if ever.
- lmm 11y ago"sort" also has a specific meaning in type theory - it's a class of kinds. Also the type-theoretic meaning of "kind" is accurate here, no? Owned vs slice really are two different kinds - it's just that Rust can't represent kindedness in general (it has specialized support for tracking ownedness, but you can't reuse that system to track other properties).
- dallbee 11y agoNice writeup. I'd like to see some examples illustrating appropriate usage of Clone-on-Write with Rust strings - that's something I personally have struggled with while learning rust.
- steveklabnik 11y agoCow in general is on my shortlist for "I need to improve the docs." One area where it's used effectively in the stdlib is String::from_utf8_lossy: http://doc.rust-lang.org/collections/string/struct.String.html#method.from_utf8_lossy http://doc.rust-lang.org/collections/string/struct.String.ht... If the argument is valid UTF-8, then no allocations need to be performed, and so the returned value will be a &str. But if it's not valid UTF-8, the replacement characters need to be inserted, which means allocation, which means String will be returned.
- llogiq 11y agoI blogged about that some time ago: https://llogiq.github.io/2015/07/09/cow.html https://llogiq.github.io/2015/07/09/cow.html
- tatterdemalion 11y agoI've used it when I wanted to provide a value which was statically initialized with a literal in the source, but which could possibly mutate during program execution. Instead of doing `String::from` and performing the heap allocation regardless, the heap allocation is only performed if you actually mutate it.
- deleted 11y ago[deleted]
- pixel_fcker 11y agoThanks for this is was very helpful. Honestly I think the main thing that trips people up here is the naming convention. If it was String and StringView rather than str I don't think people would have nearly as much trouble. I guess the name was chosen for brevity considering how pervasive &str is in rust code.
- steveklabnik 11y agoYeah, it's sort of historic. I like to joke that renaming String to StrBuf is my only wish for Rust 2.0. That said, if anything, it would be StrBuf and str, because str is a primitive type, hence the lower case. It almost wasn't a primitive type, but there were drawbacks, so we kept it as one.
- bjz_ 11y agoWhat were the drawbacks?
- steveklabnik 11y agohttps://github.com/rust-lang/rust/pull/19612 https://github.com/rust-lang/rust/pull/19612
- Manishearth 11y agoI believe that StrBuf or something similar was the name for a short time after the ~str removal.
- bjz_ 11y agoI would be against giving the more expensive type the shorter name. That said, I kind of agree with Steve that having a `Buf` suffix would help.
- twsted 11y agoStupid question: Why "str" instead of "Str"? (For homogeneity)
- bjz_ 11y agoOh you wouldn't believe the huge amount of bikeshedding that went into the naming of `String` and `str`. :) The end result might not be the nicest - to my eyes at least - but at least a decision was made!
- Retra 11y agoBecause primitive types are all lowercase. See: char, usize, i32, f64, etc...