14 ms·
In this post I’m going to explain the organization of Rust’s string types with String and str as examples, then get into the lesser-used string types—CString, C
by stcredzero 11y ago
In this post I’m going to explain the organization of Rust’s string types with String and str as examples, then get into the lesser-used string types—CString, CStr, OsString, OsStr, PathBuf, and Path
Ouch. So one thing that Rust is not going to succeed at is not having the chaos around string types that C++ has. Why is it that some languages can get away with just having 1 unicode string type with libraries to translate to other representations as needed, but not others?
- Etzos 11y ago> Why is it that some languages can get away with just having 1 unicode string type with libraries to translate to other representations as needed, but not others? Actually that's almost exactly what Rust is doing. Most of the types of strings you mentioned are added from the FFI library which is used to create bindings to other languages. They are a necessity to convert between the representations nicely.
- jdmichal 11y agoTo be fair, Java and C# -- two languages that you would probably consider as having one unicode string type -- also effectively have two string types between `String` and `StringBuilder`. [0] Java also has `java.nio.file.Path`. [1] [0] Java also manages to conflate synchronization into the mix with `StringBuffer`. [1] C# (thankfully) just has static functions to manipulate strings as paths.
- llogiq 11y agoIf you want to add IO into Java's mix, you can also use StringWriter. Also I personally don't consider paths as strings, nor URLs – both are types for specific purposes with specific invariants, which are distinct from strings'.
- Pxtl 11y agoAgreed, as long as the language provides quick and fast go/from string mechanism, great them as their own objects. What drives me batty is when I know something is just a string under the hood but getting my string into that object is a nightmare and inconsistent.
- jdmichal 11y agoI suppose it depends on your definition of a string. If it's an ordered collection of glyphs, than paths and URLs are definitely strings with invariant restrictions. But, of course, that's hardly a useful definition when you need to take a URL as input, is it? So I agree that treating them as separate types with an external string representation is a very useful tool to have, even if I don't agree that it should always be applied! Having worked with the Path class in Java, I personally find it an impediment to understandable code, but I'd also be willing to accept that that particular implementation and my intuition simply don't mesh well.
- lomnakkus 11y ago> I suppose it depends on your definition of a string. If it's an ordered collection of glyphs, than paths and URLs are definitely strings with invariant restrictions. That's not quite true -- on POSIX paths are strings of bytes. POSIX doesn't specify any particular encoding[1]. AFAIR the only restrictions on paths are around the NUL (0) byte and '/' (47). [1] I'm sure we'd all be happier if Unicode (encoded as UTF-8) were specified as the way to do paths, but POSIX was invented quite a bit before Unicode & UTF-8, so here we are.
- Gankro 11y agoAnd on Windows it's "potentially ill-formed UTF-16": https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Basically if your language/std-lib claims to be cross platform and considers paths to be definitely-utf8, it's busted. I think, maybe, plan9 is a case where this claim holds?
- gsnedders 11y ago> And on Windows it's "potentially ill-formed UTF-16" On Windows, and in many Windows APIs, the return value is a "sequence of UTF-16 code units" (which is essentially a fancy way of saying "a sequence of 16-bit words"). This goes back to Windows originally using UCS-2, where any sequence of 16-bit words is valid (because there wasn't any concept of a surrogate, yet alone a lone surrogate), given its support predates the Unicode consortium extending Unicode to 21-bits and along with it UTF-16's existence (1996).
- Pxtl 11y agoHonestly, having coded a lot of C# path code, I'd rather an object that properly encapsulates this, as long as there's easy APIs to get a string copy from it and vice versa - I mean, you're going to be throwing away oodles of string refs anyways since they're immutable, so one more for the path conversion wouldn't hurt and the path API would be a lot cleaner.
- deleted 11y ago[deleted]
- tatterdemalion 11y agoString handling in Rust is definitely not as accessible as some other languages, but none of the complexity is incidental. The String/str split is actually fundamental and pervasive, but internally all string operations are performed on these two types. Path represents a file system path, so it has a totally different API, and CString and OsString are for translating to other representations in external systems (libc and the OS, of course), not for use outside of that context. The String/str divide is the unavoidable consequence of not having garbage collection. A String is owned, a str is borrowed. This blog post goes into more information. This is nowhere near the chaos that C++ has.
- eridius 11y agoYou can avoid the split without garbage collection, e.g. Swift has a single String type which is a retain-counted copy-on-write type. But this does mean that working with strings in Swift can have hidden costs (e.g. copies where you might not expect them), plus the cost of atomic reference counting. So this solution isn't appropriate for a language like Rust.
- tatterdemalion 11y agoYou're right, I meant "with Rust's memory management story," which excludes universal reference counting. People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics.
- pcwalton 11y ago> People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics. Which is pretty unfortunate, because the memory management field has always treated reference counting as a form of garbage collection—because it is [1] [2]. The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) This is not just a semantic quibble, because there are successful garbage collection techniques that combine tracing with reference counting (e.g. ulterior reference counting [3]) to achieve GC, so treating tracing as the only way to do GC makes no sense. In fact, David Bacon observed in 2004 that reference counting and tracing garbage collection are best viewed as just two extremes of a continuum [4], and many "optimizations" that you apply to one scheme or the other are really just moving the scheme you choose toward the opposite pole. [1]: http://www.memorymanagement.org/glossary/g.html#term-garbage-collection http://www.memorymanagement.org/glossary/g.html#term-garbage... [2]: http://researcher.watson.ibm.com/researcher/files/us-bacon/Bacon03Pure.pdf http://researcher.watson.ibm.com/researcher/files/us-bacon/B... [3]: http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-2003.pdf http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-20... [4]: https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf
- geodel 11y agoI think low level systems programing language need that.
- gnuvince 11y agoString is the heap-allocated sequence of characters, str is a view into a String. Strings in Rust are guaranteed to be valid UTF-8; file paths in Windows are not, so you cannot use String for those, so you need another type. Repeat for different "stringy" types that do not have the same semantics as Strings and you end up with a number of string types.
- steveklabnik 11y ago> String is the heap-allocated sequence of characters, To be clear, String is a heap-allocated sequence of u8 which form valid unicode scalar values. "character" in Rust usually means "char", and String is _not_ a list of chars.
- twoodfin 11y agoIn my experience, it is absolutely vital for performance to have both "owning" and "slice"/"view" string types, as well as sensible conversions between them. This is actually an area where the C++ standard library has been lacking until recently. For example, you'd typically want a map with string keys to own them, but without severe contortions that meant map readers had to allocate a string key (expensive!) to perform a lookup.
- pcwalton 11y agoThree reasons why Rust cannot adopt Go's solution: 1. Rust is not garbage collected. This means that a separation between strings-that-own-their-data and strings-that-reference-their-data is needed. 2. Rust cares about first-class FFI support; in particular, copying strings at FFI boundaries is right out. That is why Rust needs C strings to be a separate type. 3. Rust wants to do pathnames right in a cross-platform way. In particular, it should be impossible for other apps to create valid pathnames that cause Rust apps manipulating those paths to crash or malfunction. UTF-8 paths fail this criterion.
- PopsiclePete 11y agoPoint 3 reminds me of Haskell trying to prove certain grammars correct at compile-time or C++ calculating factorials using template meta-programming. Very cool to show off to your geek programmer friends and impress them, but I really question the day-to-day usefulness when considering the added complexity. I have never ever had bugs in my code from somehow having invalid Unicode code-points in my paths. 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I realize certain classes of bugs could happen, but I'd rather take that risk, have a bug, and fix it, than burden the type system and libraries. In a way, I'm a little sad that modern computer languages are going from C's extreme of "always trust the programmer" to the other of "never trust the programmer". Bugs are bad, but language usability, ease-of-use and simplicity are also worth fighting for. Having said all that, I am really happy to see Rust being awesome in the general case and appreciate all the work you and the rest of the team are putting forth.
- oldmanjay 11y agoI definitely fall on the other side of the fence, philosophically. Perhaps I've just fixed too many of those "trust me" bugs that programmers are constantly adding into code I have to maintain. I'm especially over the thinking that sufficient vigilance is something we should rely on humans to achieve when the computer is way, way better at it.
- vessenes 11y ago> 99% of the time, replacing '/' with '\' and handling drive letters gets you there. I think that's precisely the point; at scale, and with today's modular software culture (like last week's left-pad news cycle demonstrates), 99% correct software can mean millions of installations of bad code, all vulnerable to attack, or even just 1%-type bugs. One way to think about the last 10 years of language design is an ongoing dialogue on how much or little to trust engineer intuition about which tools are safe for which jobs. The pendulum is swinging towards the belief that in general engineers should be given much more safety-oriented tools.
- burntsushi 11y agoAs the author of Rust's regex crate (and various other stringy things), Rust's string handling is easily the best I've ever used. The usage of String/&str completely dwarfs usage of the other string types, which are a bit more specialized.
- agumonkey 11y agoHaving recently read about Pascal string types http://wiki.freepascal.org/Character_and_string_types http://wiki.freepascal.org/Character_and_string_types this feels very interesting.
- thrownaway2424 11y agoYeah if nothing else the naming here could be better. If someone writes a blog post where the correspondence between String and ownership has to be repeated fifty times, it might have been better to name them OwnedString and StringView, or something like that.
- Manishearth 11y agoNote that everything but String and str are not used that often, and when they are, its straightforward. You'll probably use them in cases like `File::new(Path::new(my_string))` with some error handling if necessary, and forget that the intermediate existed. In fact, many of these APIs use std::convert trait magic so that you don't need to manually convert to the intermediate type at all. You also see these string types used when they are read, stored, and reused later. I haven't seen many cases of string-type-specific gymnastics being done, except sometimes with paths (which you can treat as a list of characters as well as a list of segments).