19 ms·

Why Rust strings seem hard

254 points by brundolf 6y ago

jeltz 6y ago
Hm, I feel Rust's strings are the easiest I have worked with on any programming language but that might be due to my knowledge of unicode and of C. The only thing which surprised me was the &str type. Why isn't it just an ordinary struct (called e.g. Str) consisting of a pointer/reference and a length?
- brundolf 6y ago
  Yeah, this article was specifically aimed at people who haven't worked with C/C++, and instead have the higher-level mental model for what strings are and how they work.
- TheCoelacanth 6y ago
  Rust's strings are one of the easiest to use correctly and one of the hardest to use incorrectly, but they vast majority of string handling is done incorrectly, so that makes Rust seem hard.
- steveklabnik 6y ago
  I linked to a PR above which implemented this idea, and was rejected.
  - steveklabnik 6y ago
    (see kimundi's answer; I misunderstood what you were asking. I'd have deleted this comment but I don't seem to be able to.)
- kimundi 6y ago
  With dynamically sized types like `str`, Rust allows to separate "what kind of pointer + metadata is this" from "what kind of data does it point at". So, for example, you can have the types `str`, `[T]` or `Path`, and can have them behind the pointer types `&T`, `Box<T>` or `Arc<T>`. If Rust had defined a special struct `Str` for `&str`, then it would have to define special structs for all the combinations possible: Str, ArcSlice, BoxPath, etc.
CJefferson 6y ago
One thing I've often wondered about Rust strings. I often hear that &str is 'a string slice'. But, Rust has a notation for slices -- &[T]. Why are strings the only thing (that I know of) that don't use the same slice notation as everything else?
- brundolf 6y ago
  That's a great question. I don't have a complete answer, but I do know that &str has lots of string-specific functionality that's really helpful. The .chars() method for example gives you an iterator over actual unicode chars, as opposed to bytes, because the former can have variable byte-widths. There may be other reasons; I'm not sure.
- littlestymaar 6y ago
  > But, Rust has a notation for slices -- &[T] &[T] is an “array-slice”, even if it's called just “slice”. See this example ([..] is the syntax to create a slice of something): https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1fe3556fb5dea35603eb7d26bd7d5863 https://play.rust-lang.org/?version=stable&mode=debug&editio...
- edflsafoiewq 6y ago
  What would they be, &[u8]? That's already a thing: it's an arbitrary byte sequence. &str is specifically UTF-8 data. &OsStr and &Path are the same way.
  - __s 6y ago
    Indeed &str has as_bytes which returns itself as &[u8] But, str is a subset of &[u8], the type's contract is that it is unsafe to have non valid UTF-8 data, hence https://doc.rust-lang.org/std/str/fn.from_utf8.html https://doc.rust-lang.org/std/str/fn.from_utf8.html can error, offering the unsafe variant https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.html https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm... This is all very different than &[char] which would be an array of 4 byte characters (or, a UCS4 string)
- 6y ago
geodel 6y ago
> It's a testament to Rust's breadth and accessibility that even people who have never done low-level programming before are giving it a try. Umm, no, I think it is testament of Rust evangelism that even when Rust is least appropriate, people start using it. So we have folks who start coding a 5 webpage worth of project in Rust because you know low level control, high performace , next generation, secure software etc.
rectang 6y ago
If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. • All the 2-byte-char languages which were designed for UCS-2 before the Unicode Consortium pulled the rug out from underneath them and obsolesced constant-width UCS-2 in favor of variable-width UTF-16. • Languages which result in silent corruption when you concatenate encoded bytes with a string type (e.g. Perl but there are many examples.) • C, where NUL-terminated strings are the rule, and the standard library is of no help and so Unicode string handling needs to be built from scratch. All those checks which you have to fight to opt into, defying both the language and other lazy programmers (either inside your org, or at an org which develops dependencies you use)? Those checks either happen automatically or are much easier to use without making mistakes in Rust.
- ajross 6y ago
  Alternatively: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Obviously the world is a big place and there is room for lots of paradigms and worldviews and we aren't supposed to judge too much. But come on. If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period.
  - spamizbad 6y ago
    It's the edge-cases that get you.
  - magicalhippo 6y ago
    > if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. How so? Delphi for example has wide character-based strings as default, what's wrong with that?
    - estebank 6y ago
      The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as if ASCII was the only encoding in existence could continue treating strings in the same way (using their length to calculate their user-visible length, indexing directly on specific characters to change them, etc). All of those practices are immediately wrong once UTF-32 came in existence and UTF-16 became a variable length encoding. But even if that hadn't happened, what you want to be operating on is not characters, but grapheme clusters, which are equivalent to a vector of chars. Otherwise you won't handle the distinction between ë and ë or emojis correctly.
jstanley 6y ago
> Then if your program later says "actually that wasn't enough, now I need Y bytes", it'll have to take the new chunk, copy everything into it from the old one, and then give the old one back to the system. This is mostly true. If you get lucky, there may already be enough unused space past the end of the existing allocation, and then realloc() can return the same address again, no copying required. But if you know you're going to be doing lots of realloc() (and you're not unusually-tightly memory-constrained) then instead of growing by 1 byte each time it's often worth starting with some sensible minimum size, and doubling the allocated size each time you need more space. That way you "waste" O(N) memory, but only spend O(N lg N) time on copying the data around instead of O(N^2).
- sumtechguy 6y ago
  I personally use a 'stride' instead of doubling. On small sizes doubling works OK. But when you get past about 8k-16k the empty memory starts to stack up.
  - koverstreet 6y ago
    Never do this - you're introducing a hidden O(n^2). Folks, this is why you take the algorithms class in CS.
    - axiosgunnar 6y ago
      Could you elaborate? This seems very interesting.
      - magicsmoke 6y ago
        Asymptotically, there's no difference between allocating a n+1 buffer and a n+k buffer before copying your old data in. You'll still get O(n^2) In reality, it depends on the data you're handling. You may never end up handling sizes where the O(n^2) asymptote is significant and end up wasting memory in a potentially memory constrained situation. At the end of the day, it all depends on the actual application instead of blind adherence to textbook answers. Program both, benchmark, and use what the data tells you about your system. If I've got a 500 MB buffer that I append some data to once every 5 minutes, I might want to reconsider before I spike my memory usage to fit a 1 GB buffer just to close the program 15 minutes later.
brink 6y ago
Nicely written! I wish I had this article when I first started Rust. Would have saved me some trouble.
alexchamberlain 6y ago
Very interesting write up. Would be interested in a comparison to a COW string too.
armSixtyFour 6y ago
This was one of the more confusing parts of Rust when I first started using it. I find that I don't necessarily run into it a lot. After a while you sort of change how you organize your code and you find that you're not fighting strings or the borrow checker very often. I'm not really sure how I got to that point however, it's more just practice than anything else.
- seoaeu 6y ago
  Yeah, in some ways it is kind of weird how much discussion there is about Rust's borrow checker compared to how much time practitioners actually spend dealing with it. I see less about string handling (this post being an exception) but that is also basically a non-issue once you get the hang of it.
  - brundolf 6y ago
    Exactly. It's not actually hard once it clicks, but I think a certain subset of newcomers spend lots of time being frustrated with the fact that their understanding doesn't seem to work, when they could just be given the key bits of information and be able to move forward with things making sense. That was the motive of this post :)
zabzonk 6y ago
> "foofoo" takes up twice as much space in memory as "foo". Not in the imaginary language you keep talking about as "C/C++".
- pornel 6y ago
  Right, in case they're heap allocated, the worst-case alignment requirement of malloc, as well as free() without a size is likely to force implementations to round it up to at least 8 or 16 bytes for both.
  - ncmncm 6y ago
    No. The result of std::string("foo") is exactly the same size as of std::string("foofoo"). They take up exactly the same number of bytes in the free store, on the stack, in a hash table, what have you. When the string is very long, it occupies two stretches of storage, one exactly the same size as std::string("foo"), plus another stretch big enough for the long sequence plus whatever overhead heap allocation adds. But that is different from the examples given. For the cases cited, there is no indirection of any kind. The bytes are right there in the std::string object.
    - pornel 6y ago
      We just interpret "space in memory" at different levels of indirection. If you try to create std::string("verylong…") with more characters than you have memory, you run out of memory, so it "takes up space in memory", even though std::string has the same size as always. Bonus fact: *"foo" and *"foofoo" in Rust dereference to 3 bytes and 6 bytes, respectively. It's neither a pointer nor the first char.
  - zabzonk 6y ago
    I actually (and pedantically, I'm afraid) meant that neither in C or C++ is it true that "foofoo" takes up double the size of "foo" - both will have a zero character at the end. The sizeof operator when applied to each will also return values that indicate that one is not double of the value of the other.
jasonhansel 6y ago
My way of thinking about this (which I think is correct) is: "&str" is like "&[u8]", except that "&str" is guaranteed to contain valid UTF-8.
sequoia 6y ago
Incredibly clear and compassionate writing (callout boxes throw a bone to readers who aren’t well versed in concepts like the heap, character arrays etc.). Big kudos to the author!
- brundolf 6y ago
  Thanks :)
rsc 6y ago
If you can indulge a non-Rust point of view, if I'd been faced with this design problem, I think I would have put a "this is a rodata-backed string" bit into String, and then the representation would be {ptr, len, rodata-bit} (or squeeze a bit out of the ptr or len if the space matters). Then "abc" could have type String, and the only difference between let abc1: String = "abc" and let a: String = "a" let abc2: String = a + "bc" would be that (assuming the compiler doesn't get cute) abc1 is pointing at the rodata bytes for "abc" and abc2 is pointing at allocated bytes. (But it can tell them apart so the deallocation is not ambiguous.) It seems like this would have avoided a lot of the ink spilled over &str vs String. I know it's too late now, but was this considered at all, and if so what were the reasons not to adopt it? Thanks.
- brundolf 6y ago
  I hadn't heard of rodata before, but based on a quick Google, I think what you're describing is similar to Cow<str>. I can't speak to the reasons why this wasn't made the default, but I believe it is at least possible.
  - cesarb 6y ago
    It's probably the same reason why IIRC the default Rust String does not have a "small string optimization": it would add a lot of unpredictable branches. And it would be even worse, since unlike the "small string optimization", even small mutations which don't change the size would have to allocate when the original was read-only.
    - pornel 6y ago
      Another problem is that `&str` is more than just "read-only". It also tracks lifetime to prevent use-after-free. A universal owned-or-readonly-borrowed String would be unsafe without also adding reference counting or GC to keep the borrowed-from string alive.
- devit 6y ago
  Cow<'static, str> is exactly what you are asking for. In general the String type is not very good, and you should use something appropriate to your use of strings for any string that is used more than a constant times in your program, which can be, for instance: - String - Box<[str]> - Cow<'static, str> - Cow<'a, str> - Interned strings - Rc<[str]> or Arc<[str]> - A rope data structure In fact I think putting String in the standard library might have been a mistake since it's almost always a suboptimal choice for anything except string builders in local variables.
StreamBright 6y ago
More articles like this and I might get started with Rust again. Well async is still a problem but at least strings are much clearer now after reading this.
irrational 6y ago
I love articles like this. I imagine this is the kind of stuff you learn if you study CS or Software Engineering in college. Maybe when I retire I will go and get a CS degree so I can learn all the things I should have learned before I began working professionally as a programmer 25 years ago.
- amilios 6y ago
  Nah, I wish, you just learn how to implement quicksort in Java and stuff like that The only languages I saw during my degree were Java, C, Python, Perl, and a little bit of Prolog. And I finished my Bach last year.
- brundolf 6y ago
  We used C++ throughout most of my CS program, and I hated it at the time and never wanted to write it ever again (and haven't!), but good lord did it help me understand how computers work. I've benefitted from that perspective ever since. I'm not sure you have to do a whole degree to get that perspective, though. Just learning C or C++ should get you a good chunk of it.
  - v8dev123 6y ago
    > understand how computers work No body understands it even at electron orbitals. Your understanding will be wrong in next decade if not wrong already. Computer Arch is proprietary. Give a try for C++17. Zero Cost Abstractions RAII It's a beast.
- Thaxll 6y ago
  Except that you forget pretty quicly everything you've learned in CS because when you work in the real world you don't need it.
alkonaut 6y ago
I wish more languages had primitives for "simple strings". Most of the time when I use a string I could live with a restriction that it's ascii only and can fit in 64bytes. "Programming string" vs "human string". For example a textual value of some symbol or a name of a resource file I can control myself or a translation lookup key (which I can make sure is always short and ascii). An XML element name or json property in an enormous file with a schema I control. It seems weird to use the same type for the "human string" e.g. user input, a name, the value looked up from the translation key and so on. For the simple strings it feels like making heap allocations, or either using two bytes per char (e.g C#) or worrying about encoding in an utf-8 string (e.g. Rust) are both wasteful.
- brundolf 6y ago
  You can pretty much do this in Rust: let simple_str: &[u8] = b"hello world"; println!("{}", simple_str[6] as char); // 'w' Though I would advise against it in most cases, because even many "non-human" formats like XML and JSON do allow for Unicode characters
  - sedatk 6y ago
    Filenames too.
    - alkonaut 6y ago
      There is no problem making filenames that are bound ascii. Just because they can be non-ascii on most OS'es, doesn't mean they are (when I myself control all input - that's the key). I think it's the problem of generality here. If I make a game that has thousands of texture files I want to store in a lookup table and it saves me x ns on the lookup if those textures are in a dictionary with 1-byte characters vs 2-byte character keys, then I could just make sure my file names are ascii. So I have a collection of 2000 ascii file names. I don't care whether the rest of the files outside my game textures are, or could be, non-ascii.
      - Narishma 6y ago
Animats 6y ago
He's working too hard. You don't need all those type declarations. let a = "hello".to_string(); let b = "world".to_string(); let c = a + " " + &b; println!("Sentence: {}", c); It's a bit confusing that you need "&b". "+" was defined as String + &str, which is somewhat confusing but convenient. If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. Yes. C is awful. C++ is still sometimes UTF-16. C#/.net is still mostly UTF-16. Windows is still partly in the UTF-16 era. So is Java. So is Javascript. Python 2 came in 2-byte and 4-byte versions of Unicode. The UTF-16 systems use a "surrogate pair" kludge to deal with characters above 2^16. CPython 3 has 1, 2, and 4 byte Unicode, and switches dynamically, although the user does not see that. Linux and Rust are pretty much UTF-8 everywhere now, but everyone else hasn't killed off the legacy stuff yet.
- brundolf 6y ago
  The type declarations are there for maximum clarity, since the inferred types may not be obvious to the target audience
  - jdmichal 6y ago
    Yes, please. One of my biggest peeves are posts that are meant to be educational, but yet don't define the types used for variables nor define the namespace / package they are from. Java posts are rife with the latter, and I'm really not looking forward to the `var` keyword making the former a thing too. And quite a few get bonus points for doing such on types that require a new dependency -- but good luck figuring out which dependency without knowing the dependency name nor the package name of the class!
  - mywittyname 6y ago
    You did it right, IMHO. The parent comment just confuses me while your article was quite clear.
eximius 6y ago
An alternative to `"string".to_owned() + "foo"` is just using a macro such as `concat!("string", "other string")`.
- brundolf 6y ago
  That actually doesn't work in the general case, because concat!() only accepts static (compile-time) literals. i.e., this is allowed: concat!("string", "other string") but this is not: let my_str: String = ...; concat!(my_str, "other string")
nikisweeting 6y ago
Why cant every literal "abc" just be instantiated as a heap String by default? You could have a separate notation like &"abc" when you want a slice, similar to python's b"abc" r"abc", etc. modifiers. Heap Strings seem much more useful in general.
- brundolf 6y ago
  It would be pretty wasteful; in any read-only context you'll need a &str anyway, and making them all Strings would cause tons of unneeded allocations. Many Rust devs care a lot about avoiding unnecessary allocations: some people even use Rust on embedded systems that don't allow allocations at all, so building allocation into a language fundamental would likely be a mistake.
  - nikisweeting 6y ago
    Ok, I can see why it's not the default, but do you know why they didn't give it some syntax sugar to at least make it less painful/verbose to create one? let a = "abc" // this is a &str let b = s"abc" // this is a String
    - estebank 6y ago
      There have been conversations about this, but none that pushed forward an RFC that answered all the potential concerns. It is indeed a wart, one that I would personally like to see fixed, but I'm also ambivalent on whether I would like to make String nicer to use on literals or instead focus our efforts to make Cow<'_, str> a much nicer (and prevalent) type. https://internals.rust-lang.org/t/pre-rfc-custom-string-literals/5037 https://internals.rust-lang.org/t/pre-rfc-custom-string-lite... https://internals.rust-lang.org/t/pre-rfc-allowing-string-literals-to-be-either-static-str-or-string-similar-to-numeric-literals/5029 https://internals.rust-lang.org/t/pre-rfc-allowing-string-li... https://internals.rust-lang.org/t/pre-rfc-string-literals-through-prefixes/2928 https://internals.rust-lang.org/t/pre-rfc-string-literals-th...
- Daishiman 6y ago
  System programming languages avoid allocations and unnecessary resource consumption. I'd say it's one of their hallmark characteristics. The programming convenience of higher-level languages comes at a very substantial cost of requiring a complex runtime, more memory for data structures, unpredictable performance, and pushing complexity where it's not visible. One philosophy favors visibility over resources, the other favors convenience of use.
Someone1234 6y ago
This article was absolutely fantastic. It kind of lost me right up until the first example code, where I did in fact have different expectations. Then how it broke down why, and gave different potential solutions was just wonderful. I learned a lot. I will add though that other languages have a distinction between const and string object too (at least under the hood), they just go to great lengths to hide it from programmers. For example a const + const call might do the same thing as Rust under the hood, but it is transparent. Rust seems like it requires the extra steps because it wants the programmer to make a choice here (or more specifically stop them from making mistakes: like having immutable data stay immutable, rather than automatic conversion to mutable sting inc. the data cost involved even, automatic conversion is more convenient but also a performance footgun). I don't think Rust is wrong, I think it is opinionated and honestly as someone that like immutability I kinda dig it.
- brundolf 6y ago
  Glad you enjoyed it :) And yeah, under the hood those other languages do all kinds of wild optimizations with their "immutable" strings like sharing substrings between different strings and pooling to reduce allocations. I intentionally left out those nuances because from the user's perspective, those are all implementation details (even if they can surface in the form of performance changes).
- ncmncm 6y ago
  It is well-enough written where it treats Rust, but almost lost me, too, for its abuse of "C/C++", treating the two very different languages as if they were trivial variations. Where string handling is concerned, as in so many other places, they are fundamentally different. This "C/C++" bad habit is very commonly used, around Rust, to slyly imply there is no effective difference, but in a way that permits an injured response to criticism that "it just means C or C++". But it doesn't, unless you are talking about object file symbols or compiler optimizers; and often enough not then. What it does is encourage sloppy thinking, and resultant falsehoods. These falsehoods show up in the article, revealed if you change "C/C++" to "C or C++" in each case. In several places, it says just "C++" yet is clearly still talking about C. It is OK not to know C++, or not like it. But things are hard enough without active falsehoods. It is not hard to get it right: if you are talking about C, write "C". That will always be correct. If you are talking about C++, say "C++". Rust stands or falls on its own. It doesn't need invidious editorializing.
dralley 6y ago
This is a great article! The only omission is that you can concatenate two &str's at compile time using concat!().
- brundolf 6y ago
  I didn't know about that! It's pretty interesting, though also fairly niche because the most common usecase is a string addition where at least one member is a variable, not a literal
ncmncm 6y ago
Amusingly, Rust strings, whether &str or String, are unable to represent filenames, which in many, many programs is the overwhelmingly most common use for character sequences that people want to call strings. The Rust people invented the wonderful "WTF-8" notion to talk about these things. It gets awkward when you want to display a filename in a box on the screen because those boxes like to hold actual strings qua strings, not these godforsaken abominations that show up in file system directory listings. Handling WTF-8 will take a whole nother article. I don't know a name for WTF-8 sequences; I have been calling them sthrings, which is hard to say, and awkward, but that is kind of appropriate to the case.
- int_19h 6y ago
  It doesn't really make things any more complicated than they already are. If you take a filename in C, and then have to display it somewhere, you're facing the same problem - except that you might not even be aware of it, because all types are the same, and you won't notice the problem unless you happen to run into an unprintable filename. Rust is doing the right thing here by forcing developers to deal with those issues explicitly, rather than swiping them under the rug. The real issue is filenames that aren't proper strings - i.e. an OS/FS design defect - but this ship has sailed long ago.
  - ncmncm 6y ago
    That sthrings are equally as awkward to handle correctly in C, C++, Python, Ruby, Javascript, and Perl as in Rust makes them no less awkward. Nobody said Rust has done anything wrong to ban them from String. Many languages make the distinction, including (probably) C++23. Filename sthrings stink in any language. With the usual solution, using "invalid character blocks", two different files with different names may show up in a list looking the same, so you are still in an awkward place. Even with fully valid Unicode filenames, you can get a similar problem, from legitimate characters that just look the same. Even if filesystems forbade creating files with ill-formed names, we would still have the second problem. Filenames are just a hard problem. It is too much to ask of a language or library to solve them. Even filesystems that forbid ill-formed filenames do not solve them all.
  - 6y ago
skybrian 6y ago
This is a great intro, but clarifying one more thing might be useful: how do you return a string?
- brundolf 6y ago
  A String can be returned like any other owned value; whether or not a &str can be returned depends on lifetimes, as it does with any other reference. Lifetimes seem out of scope for this post, and the lifetimes story for strings isn't really strings-specific enough that it felt important to cover. There are other resources out there that thoroughly cover the topic of lifetimes; in fact I wrote a short summary myself in another post :) https://www.brandons.me/blog/favorite-rust-function https://www.brandons.me/blog/favorite-rust-function
xbar 6y ago
Plug for The Rust Programming Language "book": https://doc.rust-lang.org/book/ https://doc.rust-lang.org/book/ This article is lovely. The book goes further, and is truly very readable.
Koshkin 6y ago
Once, when asked, “Why is your neck crooked?”, the Camel said, “What do I have that is not?”
jokoon 6y ago
I want rust to be a popular language, but I can see how C and C++ are so easier for beginners if you considers the basics: you first learn C, and then you go into C++ and you use what you need. I'm very curious if there is feedback of uni teachers teaching rust as a first compiled language to students, especially when dealing with borrowing etc.
- brundolf 6y ago
  I'm not sure I would call them "easier for beginners", but probably "better for beginners trying to learn underpinning concepts" But overall I agree: I probably would not teach Rust as a first programming language, maybe not even a first low-level language. Its abstractions may be zero-cost in terms of performance, and airtight in terms of correctness, but many of them are not intuitive unless you understand what's really happening underneath. Some of them feel intuitive enough at first... until they aren't, and then people hit a wall. I'm not sure what the solution is to this problem (or whether there is one). To its credit the team has done an incredible job documenting and writing guides, but I still think there's a gap here. It may simply be that at a certain point Rust is not a learning language, it's a productivity language for experienced programmers.

254 points by brundolf 6y ago

Hm, I feel Rust's strings are the easiest I have worked with on any programming language but that might be due to my knowledge of unicode and of C. The only thing which surprised me was the &str type. Why isn't it just an ordinary struct (called e.g. Str) consisting of a pointer/reference and a length?

brundolf 6y ago
Yeah, this article was specifically aimed at people who haven't worked with C/C++, and instead have the higher-level mental model for what strings are and how they work.
TheCoelacanth 6y ago
Rust's strings are one of the easiest to use correctly and one of the hardest to use incorrectly, but they vast majority of string handling is done incorrectly, so that makes Rust seem hard.
steveklabnik 6y ago
I linked to a PR above which implemented this idea, and was rejected.
- steveklabnik 6y ago
  (see kimundi's answer; I misunderstood what you were asking. I'd have deleted this comment but I don't seem to be able to.)
kimundi 6y ago
With dynamically sized types like `str`, Rust allows to separate "what kind of pointer + metadata is this" from "what kind of data does it point at". So, for example, you can have the types `str`, `[T]` or `Path`, and can have them behind the pointer types `&T`, `Box<T>` or `Arc<T>`. If Rust had defined a special struct `Str` for `&str`, then it would have to define special structs for all the combinations possible: Str, ArcSlice, BoxPath, etc.

CJefferson 6y ago

One thing I've often wondered about Rust strings. I often hear that &str is 'a string slice'. But, Rust has a notation for slices -- &[T]. Why are strings the only thing (that I know of) that don't use the same slice notation as everything else?

brundolf 6y ago
That's a great question. I don't have a complete answer, but I do know that &str has lots of string-specific functionality that's really helpful. The .chars() method for example gives you an iterator over actual unicode chars, as opposed to bytes, because the former can have variable byte-widths. There may be other reasons; I'm not sure.
littlestymaar 6y ago
> But, Rust has a notation for slices -- &[T] &[T] is an “array-slice”, even if it's called just “slice”. See this example ([..] is the syntax to create a slice of something): https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1fe3556fb5dea35603eb7d26bd7d5863 https://play.rust-lang.org/?version=stable&mode=debug&editio...
edflsafoiewq 6y ago
What would they be, &[u8]? That's already a thing: it's an arbitrary byte sequence. &str is specifically UTF-8 data. &OsStr and &Path are the same way.
- __s 6y ago
  Indeed &str has as_bytes which returns itself as &[u8] But, str is a subset of &[u8], the type's contract is that it is unsafe to have non valid UTF-8 data, hence https://doc.rust-lang.org/std/str/fn.from_utf8.html https://doc.rust-lang.org/std/str/fn.from_utf8.html can error, offering the unsafe variant https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.html https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm... This is all very different than &[char] which would be an array of 4 byte characters (or, a UCS4 string)
6y ago

geodel 6y ago

> It's a testament to Rust's breadth and accessibility that even people who have never done low-level programming before are giving it a try. Umm, no, I think it is testament of Rust evangelism that even when Rust is least appropriate, people start using it. So we have folks who start coding a 5 webpage worth of project in Rust because you know low level control, high performace , next generation, secure software etc.

rectang 6y ago

If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. • All the 2-byte-char languages which were designed for UCS-2 before the Unicode Consortium pulled the rug out from underneath them and obsolesced constant-width UCS-2 in favor of variable-width UTF-16. • Languages which result in silent corruption when you concatenate encoded bytes with a string type (e.g. Perl but there are many examples.) • C, where NUL-terminated strings are the rule, and the standard library is of no help and so Unicode string handling needs to be built from scratch. All those checks which you have to fight to opt into, defying both the language and other lazy programmers (either inside your org, or at an org which develops dependencies you use)? Those checks either happen automatically or are much easier to use without making mistakes in Rust.

ajross 6y ago
Alternatively: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Obviously the world is a big place and there is room for lots of paradigms and worldviews and we aren't supposed to judge too much. But come on. If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period.
- spamizbad 6y ago
  It's the edge-cases that get you.
- magicalhippo 6y ago
  > if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. How so? Delphi for example has wide character-based strings as default, what's wrong with that?
  - estebank 6y ago
    The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as if ASCII was the only encoding in existence could continue treating strings in the same way (using their length to calculate their user-visible length, indexing directly on specific characters to change them, etc). All of those practices are immediately wrong once UTF-32 came in existence and UTF-16 became a variable length encoding. But even if that hadn't happened, what you want to be operating on is not characters, but grapheme clusters, which are equivalent to a vector of chars. Otherwise you won't handle the distinction between ë and ë or emojis correctly.

jstanley 6y ago

> Then if your program later says "actually that wasn't enough, now I need Y bytes", it'll have to take the new chunk, copy everything into it from the old one, and then give the old one back to the system. This is mostly true. If you get lucky, there may already be enough unused space past the end of the existing allocation, and then realloc() can return the same address again, no copying required. But if you know you're going to be doing lots of realloc() (and you're not unusually-tightly memory-constrained) then instead of growing by 1 byte each time it's often worth starting with some sensible minimum size, and doubling the allocated size each time you need more space. That way you "waste" O(N) memory, but only spend O(N lg N) time on copying the data around instead of O(N^2).

sumtechguy 6y ago
I personally use a 'stride' instead of doubling. On small sizes doubling works OK. But when you get past about 8k-16k the empty memory starts to stack up.
- koverstreet 6y ago
  Never do this - you're introducing a hidden O(n^2). Folks, this is why you take the algorithms class in CS.
  - axiosgunnar 6y ago
    Could you elaborate? This seems very interesting.
    - magicsmoke 6y ago
      Asymptotically, there's no difference between allocating a n+1 buffer and a n+k buffer before copying your old data in. You'll still get O(n^2) In reality, it depends on the data you're handling. You may never end up handling sizes where the O(n^2) asymptote is significant and end up wasting memory in a potentially memory constrained situation. At the end of the day, it all depends on the actual application instead of blind adherence to textbook answers. Program both, benchmark, and use what the data tells you about your system. If I've got a 500 MB buffer that I append some data to once every 5 minutes, I might want to reconsider before I spike my memory usage to fit a 1 GB buffer just to close the program 15 minutes later.

brink 6y ago

Nicely written! I wish I had this article when I first started Rust. Would have saved me some trouble.

alexchamberlain 6y ago

Very interesting write up. Would be interested in a comparison to a COW string too.

armSixtyFour 6y ago

This was one of the more confusing parts of Rust when I first started using it. I find that I don't necessarily run into it a lot. After a while you sort of change how you organize your code and you find that you're not fighting strings or the borrow checker very often. I'm not really sure how I got to that point however, it's more just practice than anything else.

seoaeu 6y ago
Yeah, in some ways it is kind of weird how much discussion there is about Rust's borrow checker compared to how much time practitioners actually spend dealing with it. I see less about string handling (this post being an exception) but that is also basically a non-issue once you get the hang of it.
- brundolf 6y ago
  Exactly. It's not actually hard once it clicks, but I think a certain subset of newcomers spend lots of time being frustrated with the fact that their understanding doesn't seem to work, when they could just be given the key bits of information and be able to move forward with things making sense. That was the motive of this post :)

zabzonk 6y ago

> "foofoo" takes up twice as much space in memory as "foo". Not in the imaginary language you keep talking about as "C/C++".

pornel 6y ago
Right, in case they're heap allocated, the worst-case alignment requirement of malloc, as well as free() without a size is likely to force implementations to round it up to at least 8 or 16 bytes for both.
- ncmncm 6y ago
  No. The result of std::string("foo") is exactly the same size as of std::string("foofoo"). They take up exactly the same number of bytes in the free store, on the stack, in a hash table, what have you. When the string is very long, it occupies two stretches of storage, one exactly the same size as std::string("foo"), plus another stretch big enough for the long sequence plus whatever overhead heap allocation adds. But that is different from the examples given. For the cases cited, there is no indirection of any kind. The bytes are right there in the std::string object.
  - pornel 6y ago
    We just interpret "space in memory" at different levels of indirection. If you try to create std::string("verylong…") with more characters than you have memory, you run out of memory, so it "takes up space in memory", even though std::string has the same size as always. Bonus fact: *"foo" and *"foofoo" in Rust dereference to 3 bytes and 6 bytes, respectively. It's neither a pointer nor the first char.
- zabzonk 6y ago
  I actually (and pedantically, I'm afraid) meant that neither in C or C++ is it true that "foofoo" takes up double the size of "foo" - both will have a zero character at the end. The sizeof operator when applied to each will also return values that indicate that one is not double of the value of the other.

jasonhansel 6y ago

My way of thinking about this (which I think is correct) is: "&str" is like "&[u8]", except that "&str" is guaranteed to contain valid UTF-8.

sequoia 6y ago

Incredibly clear and compassionate writing (callout boxes throw a bone to readers who aren’t well versed in concepts like the heap, character arrays etc.). Big kudos to the author!

brundolf 6y ago
Thanks :)

rsc 6y ago

If you can indulge a non-Rust point of view, if I'd been faced with this design problem, I think I would have put a "this is a rodata-backed string" bit into String, and then the representation would be {ptr, len, rodata-bit} (or squeeze a bit out of the ptr or len if the space matters). Then "abc" could have type String, and the only difference between let abc1: String = "abc" and let a: String = "a" let abc2: String = a + "bc" would be that (assuming the compiler doesn't get cute) abc1 is pointing at the rodata bytes for "abc" and abc2 is pointing at allocated bytes. (But it can tell them apart so the deallocation is not ambiguous.) It seems like this would have avoided a lot of the ink spilled over &str vs String. I know it's too late now, but was this considered at all, and if so what were the reasons not to adopt it? Thanks.

brundolf 6y ago
I hadn't heard of rodata before, but based on a quick Google, I think what you're describing is similar to Cow<str>. I can't speak to the reasons why this wasn't made the default, but I believe it is at least possible.
- cesarb 6y ago
  It's probably the same reason why IIRC the default Rust String does not have a "small string optimization": it would add a lot of unpredictable branches. And it would be even worse, since unlike the "small string optimization", even small mutations which don't change the size would have to allocate when the original was read-only.
  - pornel 6y ago
    Another problem is that `&str` is more than just "read-only". It also tracks lifetime to prevent use-after-free. A universal owned-or-readonly-borrowed String would be unsafe without also adding reference counting or GC to keep the borrowed-from string alive.
devit 6y ago
Cow<'static, str> is exactly what you are asking for. In general the String type is not very good, and you should use something appropriate to your use of strings for any string that is used more than a constant times in your program, which can be, for instance: - String - Box<[str]> - Cow<'static, str> - Cow<'a, str> - Interned strings - Rc<[str]> or Arc<[str]> - A rope data structure In fact I think putting String in the standard library might have been a mistake since it's almost always a suboptimal choice for anything except string builders in local variables.

StreamBright 6y ago

More articles like this and I might get started with Rust again. Well async is still a problem but at least strings are much clearer now after reading this.

irrational 6y ago

I love articles like this. I imagine this is the kind of stuff you learn if you study CS or Software Engineering in college. Maybe when I retire I will go and get a CS degree so I can learn all the things I should have learned before I began working professionally as a programmer 25 years ago.

amilios 6y ago
Nah, I wish, you just learn how to implement quicksort in Java and stuff like that The only languages I saw during my degree were Java, C, Python, Perl, and a little bit of Prolog. And I finished my Bach last year.
brundolf 6y ago
We used C++ throughout most of my CS program, and I hated it at the time and never wanted to write it ever again (and haven't!), but good lord did it help me understand how computers work. I've benefitted from that perspective ever since. I'm not sure you have to do a whole degree to get that perspective, though. Just learning C or C++ should get you a good chunk of it.
- v8dev123 6y ago
  > understand how computers work No body understands it even at electron orbitals. Your understanding will be wrong in next decade if not wrong already. Computer Arch is proprietary. Give a try for C++17. Zero Cost Abstractions RAII It's a beast.
Thaxll 6y ago
Except that you forget pretty quicly everything you've learned in CS because when you work in the real world you don't need it.

alkonaut 6y ago

I wish more languages had primitives for "simple strings". Most of the time when I use a string I could live with a restriction that it's ascii only and can fit in 64bytes. "Programming string" vs "human string". For example a textual value of some symbol or a name of a resource file I can control myself or a translation lookup key (which I can make sure is always short and ascii). An XML element name or json property in an enormous file with a schema I control. It seems weird to use the same type for the "human string" e.g. user input, a name, the value looked up from the translation key and so on. For the simple strings it feels like making heap allocations, or either using two bytes per char (e.g C#) or worrying about encoding in an utf-8 string (e.g. Rust) are both wasteful.

brundolf 6y ago
You can pretty much do this in Rust: let simple_str: &[u8] = b"hello world"; println!("{}", simple_str[6] as char); // 'w' Though I would advise against it in most cases, because even many "non-human" formats like XML and JSON do allow for Unicode characters
- sedatk 6y ago
  Filenames too.
  - alkonaut 6y ago
    There is no problem making filenames that are bound ascii. Just because they can be non-ascii on most OS'es, doesn't mean they are (when I myself control all input - that's the key). I think it's the problem of generality here. If I make a game that has thousands of texture files I want to store in a lookup table and it saves me x ns on the lookup if those textures are in a dictionary with 1-byte characters vs 2-byte character keys, then I could just make sure my file names are ascii. So I have a collection of 2000 ascii file names. I don't care whether the rest of the files outside my game textures are, or could be, non-ascii.
    - Narishma 6y ago

Animats 6y ago

He's working too hard. You don't need all those type declarations. let a = "hello".to_string(); let b = "world".to_string(); let c = a + " " + &b; println!("Sentence: {}", c); It's a bit confusing that you need "&b". "+" was defined as String + &str, which is somewhat confusing but convenient. If you've been handling Unicode properly in other languages, then Rust strings seem easy in comparison. Yes. C is awful. C++ is still sometimes UTF-16. C#/.net is still mostly UTF-16. Windows is still partly in the UTF-16 era. So is Java. So is Javascript. Python 2 came in 2-byte and 4-byte versions of Unicode. The UTF-16 systems use a "surrogate pair" kludge to deal with characters above 2^16. CPython 3 has 1, 2, and 4 byte Unicode, and switches dynamically, although the user does not see that. Linux and Rust are pretty much UTF-8 everywhere now, but everyone else hasn't killed off the legacy stuff yet.

brundolf 6y ago
The type declarations are there for maximum clarity, since the inferred types may not be obvious to the target audience
- jdmichal 6y ago
  Yes, please. One of my biggest peeves are posts that are meant to be educational, but yet don't define the types used for variables nor define the namespace / package they are from. Java posts are rife with the latter, and I'm really not looking forward to the `var` keyword making the former a thing too. And quite a few get bonus points for doing such on types that require a new dependency -- but good luck figuring out which dependency without knowing the dependency name nor the package name of the class!
- mywittyname 6y ago
  You did it right, IMHO. The parent comment just confuses me while your article was quite clear.

eximius 6y ago

An alternative to `"string".to_owned() + "foo"` is just using a macro such as `concat!("string", "other string")`.

brundolf 6y ago
That actually doesn't work in the general case, because concat!() only accepts static (compile-time) literals. i.e., this is allowed: concat!("string", "other string") but this is not: let my_str: String = ...; concat!(my_str, "other string")

nikisweeting 6y ago

Why cant every literal "abc" just be instantiated as a heap String by default? You could have a separate notation like &"abc" when you want a slice, similar to python's b"abc" r"abc", etc. modifiers. Heap Strings seem much more useful in general.

brundolf 6y ago
It would be pretty wasteful; in any read-only context you'll need a &str anyway, and making them all Strings would cause tons of unneeded allocations. Many Rust devs care a lot about avoiding unnecessary allocations: some people even use Rust on embedded systems that don't allow allocations at all, so building allocation into a language fundamental would likely be a mistake.
- nikisweeting 6y ago
  Ok, I can see why it's not the default, but do you know why they didn't give it some syntax sugar to at least make it less painful/verbose to create one? let a = "abc" // this is a &str let b = s"abc" // this is a String
  - estebank 6y ago
    There have been conversations about this, but none that pushed forward an RFC that answered all the potential concerns. It is indeed a wart, one that I would personally like to see fixed, but I'm also ambivalent on whether I would like to make String nicer to use on literals or instead focus our efforts to make Cow<'_, str> a much nicer (and prevalent) type. https://internals.rust-lang.org/t/pre-rfc-custom-string-literals/5037 https://internals.rust-lang.org/t/pre-rfc-custom-string-lite... https://internals.rust-lang.org/t/pre-rfc-allowing-string-literals-to-be-either-static-str-or-string-similar-to-numeric-literals/5029 https://internals.rust-lang.org/t/pre-rfc-allowing-string-li... https://internals.rust-lang.org/t/pre-rfc-string-literals-through-prefixes/2928 https://internals.rust-lang.org/t/pre-rfc-string-literals-th...
Daishiman 6y ago
System programming languages avoid allocations and unnecessary resource consumption. I'd say it's one of their hallmark characteristics. The programming convenience of higher-level languages comes at a very substantial cost of requiring a complex runtime, more memory for data structures, unpredictable performance, and pushing complexity where it's not visible. One philosophy favors visibility over resources, the other favors convenience of use.

Someone1234 6y ago

This article was absolutely fantastic. It kind of lost me right up until the first example code, where I did in fact have different expectations. Then how it broke down why, and gave different potential solutions was just wonderful. I learned a lot. I will add though that other languages have a distinction between const and string object too (at least under the hood), they just go to great lengths to hide it from programmers. For example a const + const call might do the same thing as Rust under the hood, but it is transparent. Rust seems like it requires the extra steps because it wants the programmer to make a choice here (or more specifically stop them from making mistakes: like having immutable data stay immutable, rather than automatic conversion to mutable sting inc. the data cost involved even, automatic conversion is more convenient but also a performance footgun). I don't think Rust is wrong, I think it is opinionated and honestly as someone that like immutability I kinda dig it.

brundolf 6y ago
Glad you enjoyed it :) And yeah, under the hood those other languages do all kinds of wild optimizations with their "immutable" strings like sharing substrings between different strings and pooling to reduce allocations. I intentionally left out those nuances because from the user's perspective, those are all implementation details (even if they can surface in the form of performance changes).
ncmncm 6y ago
It is well-enough written where it treats Rust, but almost lost me, too, for its abuse of "C/C++", treating the two very different languages as if they were trivial variations. Where string handling is concerned, as in so many other places, they are fundamentally different. This "C/C++" bad habit is very commonly used, around Rust, to slyly imply there is no effective difference, but in a way that permits an injured response to criticism that "it just means C or C++". But it doesn't, unless you are talking about object file symbols or compiler optimizers; and often enough not then. What it does is encourage sloppy thinking, and resultant falsehoods. These falsehoods show up in the article, revealed if you change "C/C++" to "C or C++" in each case. In several places, it says just "C++" yet is clearly still talking about C. It is OK not to know C++, or not like it. But things are hard enough without active falsehoods. It is not hard to get it right: if you are talking about C, write "C". That will always be correct. If you are talking about C++, say "C++". Rust stands or falls on its own. It doesn't need invidious editorializing.

dralley 6y ago

This is a great article! The only omission is that you can concatenate two &str's at compile time using concat!().

brundolf 6y ago
I didn't know about that! It's pretty interesting, though also fairly niche because the most common usecase is a string addition where at least one member is a variable, not a literal

ncmncm 6y ago

Amusingly, Rust strings, whether &str or String, are unable to represent filenames, which in many, many programs is the overwhelmingly most common use for character sequences that people want to call strings. The Rust people invented the wonderful "WTF-8" notion to talk about these things. It gets awkward when you want to display a filename in a box on the screen because those boxes like to hold actual strings qua strings, not these godforsaken abominations that show up in file system directory listings. Handling WTF-8 will take a whole nother article. I don't know a name for WTF-8 sequences; I have been calling them sthrings, which is hard to say, and awkward, but that is kind of appropriate to the case.

int_19h 6y ago
It doesn't really make things any more complicated than they already are. If you take a filename in C, and then have to display it somewhere, you're facing the same problem - except that you might not even be aware of it, because all types are the same, and you won't notice the problem unless you happen to run into an unprintable filename. Rust is doing the right thing here by forcing developers to deal with those issues explicitly, rather than swiping them under the rug. The real issue is filenames that aren't proper strings - i.e. an OS/FS design defect - but this ship has sailed long ago.
- ncmncm 6y ago
  That sthrings are equally as awkward to handle correctly in C, C++, Python, Ruby, Javascript, and Perl as in Rust makes them no less awkward. Nobody said Rust has done anything wrong to ban them from String. Many languages make the distinction, including (probably) C++23. Filename sthrings stink in any language. With the usual solution, using "invalid character blocks", two different files with different names may show up in a list looking the same, so you are still in an awkward place. Even with fully valid Unicode filenames, you can get a similar problem, from legitimate characters that just look the same. Even if filesystems forbade creating files with ill-formed names, we would still have the second problem. Filenames are just a hard problem. It is too much to ask of a language or library to solve them. Even filesystems that forbid ill-formed filenames do not solve them all.
- 6y ago

skybrian 6y ago

This is a great intro, but clarifying one more thing might be useful: how do you return a string?

brundolf 6y ago
A String can be returned like any other owned value; whether or not a &str can be returned depends on lifetimes, as it does with any other reference. Lifetimes seem out of scope for this post, and the lifetimes story for strings isn't really strings-specific enough that it felt important to cover. There are other resources out there that thoroughly cover the topic of lifetimes; in fact I wrote a short summary myself in another post :) https://www.brandons.me/blog/favorite-rust-function https://www.brandons.me/blog/favorite-rust-function

xbar 6y ago

Plug for The Rust Programming Language "book": https://doc.rust-lang.org/book/ https://doc.rust-lang.org/book/ This article is lovely. The book goes further, and is truly very readable.

Koshkin 6y ago

Once, when asked, “Why is your neck crooked?”, the Camel said, “What do I have that is not?”

jokoon 6y ago

I want rust to be a popular language, but I can see how C and C++ are so easier for beginners if you considers the basics: you first learn C, and then you go into C++ and you use what you need. I'm very curious if there is feedback of uni teachers teaching rust as a first compiled language to students, especially when dealing with borrowing etc.

brundolf 6y ago
I'm not sure I would call them "easier for beginners", but probably "better for beginners trying to learn underpinning concepts" But overall I agree: I probably would not teach Rust as a first programming language, maybe not even a first low-level language. Its abstractions may be zero-cost in terms of performance, and airtight in terms of correctness, but many of them are not intuitive unless you understand what's really happening underneath. Some of them feel intuitive enough at first... until they aren't, and then people hit a wall. I'm not sure what the solution is to this problem (or whether there is one). To its credit the team has done an incredible job documenting and writing guides, but I still think there's a gap here. It may simply be that at a certain point Rust is not a learning language, it's a productivity language for experienced programmers.