14 ms·
My Favorite Rust Function Signature
- db48x 6y agoThat function really is a good example for explaining lifetimes in Rust.
- rob74 6y agoThe function may be a good example, but this blog post doesn't really do a good job at explaining it, at least not to people unfamiliar with Rust (which I assume it's aimed at). If I understood correctly, if you assign the lifetime 'a (or 'x, 'l, whatever) to all parameters, you tell the compiler that they all have the same lifetime? And if you wanted to have different lifetimes, you could use 'a and 'b? But the article doesn't explicitly mention that, so you could end up thinking that 'a is some kind of a magical constant that you have to use...
- brundolf 6y agoThe target audience was programmers who are at least familiar with the basic concepts of stack, heap, and pointers, including but not limited to newer Rust programmers. You make a good point about 'a; I could have made that clearer. In practice it works the same way as a generic type (the letter is arbitrary, and it's declared inside the < >), which I assumed people would be familiar with, but I can see how the connection between the two may not have been obvious.
- Sharlin 6y agoYep, you’re correct. Naming the first (especially when it’s also the only) lifetime parameter ’a is just a convention, like naming the first type parameter T.
- ragnese 6y agoThat's a good point. When I first started trying to learn Rust, I also thought that there was some semantic meaning to 'a, since everyone uses that exact same name in all the examples.
- steveklabnik 6y agoYou don't have to write the lifetimes here, this can just be fn tokenize(code: &str) -> impl Iterator<Item=&str> { but including them to be more explicit does make sense for the post.
- jdmichal 6y agoThanks for posting this. I follow Rust, but haven't used it. It seemed strange to me that tying all the lifetimes like that wasn't the default.
- estebank 6y agoIt isn't done for multiple arguments because that would make the function's signature more restrictive than what you'd normally want.
- milliams 6y agoIn that case, how does the caller know that the results will last as long as the input? Does it assume `'static` or is there some cleverness?
- deleted 6y ago[deleted]
- jen20 6y agoThis is determined by lifetime elision [1]. Specifically (quoting): - Each elided lifetime in the parameters becomes a distinct lifetime parameter. - If there is exactly one lifetime used in the parameters (elided or not), that lifetime is assigned to all elided output lifetimes. So, Since the signature has one `&str` as a parameter, the lifetime is assigned to the output lifetime also per these rules. [1]: https://doc.rust-lang.org/reference/lifetime-elision.html https://doc.rust-lang.org/reference/lifetime-elision.html (Edited to improve formatting)
- brundolf 6y ago
- vlovich123 6y agoYou can actually kind of do this with `std::shared_ptr` [0], but it's going to be nowhere near as efficient[1] as the Rust mechanism. Maybe a single-threaded implementation of it could come close but you'd have to roll it yourself or use something like `boost::local_shared_ptr`. You may also need to go to intrusive pointers to make sure there's memory alignment. Of course the Rust variant is not only cheaper (no ref counts), it's also guaranteed thread-safe (which again, C++ would have to use something like the atomic `shared_ptr` to achieve) TLDR: Even when C++ can kind of do it, you have to give up concurrency safety to get close while adding memory pressure & still not being as fast while Rust gives you guaranteed concurrency safety for free regardless of accidental misuse of your API. [0] Currently that would be constructor #8 on https://en.cppreference.com/w/cpp/memory/shared_ptr/shared_ptr https://en.cppreference.com/w/cpp/memory/shared_ptr/shared_p... [1] Not only are you always paying for atomic ref counting even when you don't need it, aliased `shared_ptr` will be as expensive to take a ref for as normally constructing it - there's possible no way to `make_shared` with aliasing. This means the aliased shared_ptr will be even more expensive every time you need to access the ref count since the control block loses cache coherency. On the other hand if you structure it more carefully maybe you could vend `const std::shared_ptr<std::string>&` in your iterator so that in the single-threaded case your only overhead is constructing the aliased shared_ptr once & you leave it to the user to take a copy when transferring to another thread.
- boris 6y agoNobody in their right mind will use `shared_ptr` in this context. It will be `string_view` in and out with lifetime management left as an exercise to the programmer.
- steveklabnik 6y agoI read your parent as saying "assume we start from needing this to be checked," and then saying that this is the best that can be done. You're absolutely 1000% right that nobody would write that ever and would use string_view and not have it statically checked.
- 6y ago
- csears 6y agoReally enjoyed seeing this broken down and explained. Is a function like that considered idiomatic Rust?
- steveklabnik 6y agoYep! See my comment about the lifetimes, though, you may not actually write them out in the real code, but conceptually, this is Rust's bread and butter.
- stack_underflow 6y agoI'm learning Rust for the first time and found that The Rust Book does a pretty good job of covering language concepts. I actually just finished reading the section on lifetimes last night: https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html My experience so far with Rust has been a huge reduction in mental burden/bookkeeping compared to when I write modern C++.
- wwright 6y agoIt'll be fun when we get first-class generators and you can write a simple, imperative function and no helper types and still use this type signature :)
- steveklabnik 6y agoUntil then, https://doc.rust-lang.org/stable/std/iter/fn.from_fn.html https://doc.rust-lang.org/stable/std/iter/fn.from_fn.html can handle many cases.
- brundolf 6y agoYeah, in my case I did code.chars().filter_map(...), where it returns a Some() at the end of each complete token. But generators would definitely be a little cleaner!
- mrweasel 6y agoI really don’t like usage of ‘ in the function signature, it simply looks wrong. In generally that is my main complaint as I am trying to learn Rust, the syntax seems to be all over the place. Maybe that feeling will fade away as I get more use to the language.
- steveklabnik 6y agoAlmost none of it is unique to Rust, but it does draw inspiration from a wide variety of influences, so it can feel like that for sure. This syntax was taken from OCaml. Incidentally, nobody really loves it, but nobody could come up with anything clearly better either.
- dagmx 6y agoIn most cases you don't need to specify lifetimes. The compiler is smart enough to infer it for a lot of cases. As noted elsewhere by Steve, it's not even required for this signature.
- ashish01 6y agoI also used to feel the same way. But listening to [0]Crust of Rust: Lifetime Annotations by Jon Gjengset helped me understand its need. [0] https://www.youtube.com/watch?v=rAl-9HwD858 https://www.youtube.com/watch?v=rAl-9HwD858 really helped me understand it.
- ragnese 6y agoIf that's your main complaint, then good for Rust!
- deleted 6y ago[deleted]
- haskellandchill 6y ago> This is something that you cannot do both a) this safely and b) this efficiently in any other language. This is wrong. Pretty sure you could do this in ATS for example.
- brundolf 6y agoI may have overstated that. What I really meant was "any other language that I'm passingly familiar with", which includes the majority of mainstream languages.
- jrpelkonen 6y agoSubstrings in older versions of Java used to reuse the character array of the original string instead making a copy. The flip side is that the original string must be retained in memory until all the substrings have “become garbage”. So there is a trade-off.
- ameliaquining 6y agoThe Rust version has that same tradeoff; the string can't be deallocated until all the tokens' lifetimes have ended. (In this particular use case, that's not really a cost, since you probably need to keep the string available the whole time anyway.) Go, which is garbage-collected, also does the slice-by-reference trick (but fails criterion #1 because it doesn't really have iterators).
- brundolf 6y agoYeah. But there's also a bigger tradeoff which is having garbage collection in the first place. Rust lets you depend on immutability without having garbage collection to clean up the values for you, which is a really special thing.
- ridiculous_fish 6y agoTokenizers usually need to retain some information about where each token came from, like the range in the source. In C++ I might do it like this: struct token { string::iterator start; string::iterator end; }; and then the range in the original source can be recovered via: size_t offset = tok.start - src.begin(); What is the preferred way to do this in Rust? Is there something better than using indexes? Rust doesn't appear to have a way to say "give me the location of this &str in that &str" (even though it seems like it would be safe).
- steveklabnik 6y agoMy understanding of rust-analyzer's internals are limited, but I believe https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/parser/src/parser.rs#L163 https://github.com/rust-analyzer/rust-analyzer/blob/master/c... is what it does here; that is, it keeps track of the start and ending positions as it goes https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/parser/src/parser.rs#L297 https://github.com/rust-analyzer/rust-analyzer/blob/master/c...
- masklinn 6y ago> What is the preferred way to do this in Rust? Is there something better than using indexes? I don't think so. > Rust doesn't appear to have a way to say "give me the location of this &str in that &str" (even though it seems like it would be safe). 'course there is: str::find (https://doc.rust-lang.org/std/primitive.str.html#method.find https://doc.rust-lang.org/std/primitive.str.html#method.find) It looks a bit abstract because it takes a Pattern rather than an `&str`, but a char, a slice of chars, an &str, or a predicate function, are all patterns.
- couchand 6y agoPresumably the parent wasn't asking about searching for one during slice in another, but rather locating the original offset of a string slice derived from another.
- 6y ago
- quietbritishjim 6y ago> The downside is that, if you perform this as a separate pass, your parser now has to iterate over all of the source code twice. This may not be the end of the world: tokenizing isn't the most expensive operation. But it isn't ideal, so some parsers combine the two passes into a single one, saving cycles at the expense of readability. This paragraph is a bit muddled: * If you keep the two passes separate, then the second pass is over the token stream. So it's irrelevant that "tokenizing isn't the most expensive operation" because you're never going to do the actual tokenising a second time. * If you combine the two passes then you're not "saving cycles" because you're still doing both of the loops - it's just that they're now happening in parrallel rather than in sequence. This confusion is repeated later when he says the combined one is "without the runtime overhead of a second loop". * What you are saving is memory, because you don't have to keep the whole token stream in memory at once. But really this is a nitpick on a small note. The overall article is great.
- Rotten194 6y agoNitpick^2: You are probably saving a few cycles due to loop overhead around pointer offsets and bounds checking, + not having to load the items into cache twice, if the sequence is too long to fit into cache.
- quietbritishjim 6y agoAgreed! I definitely wasn't telling the whole truth by saying the amount of cycles spent on looping is ultimately the same. Although the difference is complex enough that I could imagine it going either way (but you're right that the cache effects would probably be the clincher).
- bananabreakfast 6y agoI don't think this is a nitpick, this is a valid point that actually plays even more into the concept of Rust's fearless concurrency. You're not avoiding the runtime overhead of a second loop but being able to operate on both streams in parallel would be a far more complicated concept in C/C++. This actually reminds me of that influential Rob Pike talk about writing a concurrent parser in Go.
- jchook 6y agoThis might be a good thread to mention nom[1], a parser combinator in Rust. 1. https://github.com/Geal/nom https://github.com/Geal/nom
- timvisee 6y agoYou might find generics useful as well, to support any parameter that can be referenced as a `&str`, such as a `String`: fn tokenize<S: AsRef<str>>(code: S) -> ... code.as_ref() Edit: whoops, was too quick, apparently this isn't a great idea here.
- ridiculous_fish 6y agoWhat would be the ... in this case?
- Arnavion 6y agoIt's a bad suggestion for this case [1]. The ... would have to be `impl Iterator<Item = String>`, because the function would not have anything to return a borrow from. That is exactly the problem the original article was trying to avoid in the first place. [1]: And arguably a bad suggestion in general. If a fn needs a String, it should take a String. If it needs a &str, it should take a &str, and if the caller has a String they can call the fn with a borrow. fns that are generic on AsRef<str> are usually only found where the ergonomics make it worth it. Eg the standard library's filesystem functions often take AsRef<Path> parameters because string literals are AsRef<Path>, which means you can write fs::read_to_string("/etc/passwd") instead of a more cumbersome fs::read_to_string(Path::new(OsStr::new("/etc/passwd")))
- timvisee 6y agoYou're right! Didn't think about it much before posting.
- alilleybrinker 6y agoIn cases like this, it can be useful to have the generic function call a separate, private function which does the implementation and is _not_ generic. Generic functions are monomorphized, meaning copies are generated for each concrete type they're called with. Separating them out in this way can reduce the size of the generated code after monomorphization.
- noncoml 6y agowhy do we need the lifetime after the function name? i.e. `tokenize<'a>`
- dpbriggs 6y agoTo make the function generic over some lifetime 'a, similar to how generic functions can be generic over some type T (e.g. foobar<T>). This is usually not necessary to write as it is elided in common use cases.
- noncoml 6y agothanks! I think what was confusing me was that in some examples i see it omitted and in some not. Now it's clear.
- brundolf 6y agoYou can think of it like a generic type parameter. The <'a> is essentially saying "for some lifetime 'a...", and then we refer to that lifetime elsewhere to bind those usages together
- noncoml 6y agoThanks!