13 ms·

Understanding Rust futures by going way too deep

475 points by tempodox 5y ago

yodelshady 5y ago
I am surprised Rust doesn't lean into the type system more for declaring async, as I really like that way of framing it. Admittedly, I'm probably not writing ```impl Future<Output = Result<(), Report>> + 'a``` all that much in practice. I like fasterthanlime's writeups but it does have one particular bugbear for me, that seems particularly common for Rust (and Python, to be fair) feature writeups: environment-based yak shaving because the author likes the mostly-unrelated library <X>. The very first thing that happens if I try to reproduce this code is a build error for cargo-edit, which is totally unrelated to the feature being taught, yet if I wanted to follow the writeup, I'd have to fix it. I do really have this bugbear, particularly with Rust tutorials
- richardwhiuk 5y ago
  This article takes ages to get to futures because the writer seems to want to go on some tour of Rust crates first.
  - cormacrelf 5y ago
    IMO if your article is long enough that it crashes MobileSafari, it should have a table of contents. Or it should be a book chapter. It’s a really obvious criticism of all of fasterthanlime’s posts that scarcely bears repeating at this point but they’re not getting any shorter. I get that he has a lot to say but I just don’t have time to wade through all of that. What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? It’s ok to teach both things but why together?!
    - fasterthanlime 5y ago
      Ok I promise I'm going to stop replying to those comments (because as you pointed out they show up on every post), but! Rust is not the only language with a decent type system out there. I can definitely see some ML/Haskell/Scala/etc. folks who would quickly pick up on all the Future stuff, drawing parallels with their usual language of choice, yet being completely unaware of all the nice tooling around Rust. It's one of its main selling points for me! Re cargo-edit specifically, it also lets me stay in flow while writing these: instead of having to repeat "we'll just edit Cargo.toml, make sure to have that under [dependencies], uh this time we need a feature so the right-hand-side needs to be an object, not a string" - instead I can just copy into the article the actual commands I run on my side, and if a quick paragraph about where that subcommand comes from unblocks even one person then it's worth it.
      - nindalf 5y ago
        I wouldn’t worry too much about it. You have your style, and it clearly works for the people who like your work. Everyone likes different things, and it doesn’t make sense to force homogeneity. It’s ok to cater to niches.
      - WJW 5y ago
        FWIW, I fall exactly into that niche. I've never written a line of Rust in my life but am quite competent with Haskell and Ruby and have dabbled in at least half a dozen others. Seeing everyday `cargo` use made me think of bundler and the Future/Result type signatures were easy to map onto Async/Either and the like. So yes, it's a long article but I didn't mind that in the least. Keep it up!
      - kelnos 5y ago
        Ignore the naysayers in this thread; I don't really get why people think it's ok nitpick how you've decided to present a particular topic that you publish to the world free of charge. It's... kinda rude, IMO. I really enjoyed reading this piece, and had never heard of cargo-edit, so thank you for that. As a Scala person, I've actually struggled with Rust's Future, because it's quite different and requires a new mental model of how things work, which I think you did a great job providing.
    - myrrlyn 5y ago
      > What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? My colleagues. Many people are skilled programmers jumping into obscure problems with only a baseline level of familiarity with the greater environment.
      - Filligree 5y ago
        Or for example me. cargo-edit looks useful. None of the tutorials I've skimmed have mentioned it.
    - kelnos 5y ago
      > What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? raises hand I'm not sure how (or why) I'd google cargo-edit, because it's one of those things that you don't even think about until someone tells you about it, and then you're happy they did. While I certainly wouldn't call myself a Rust expert, I'm no noob either, and am intensely interested in diving deep into topics like this. (Can we also dispense with the gatekeeping put-downs like "noob", please? It's unnecessary and rude.) > It’s ok to teach both things but why together?! Because the author felt like it, and since we're consuming his work for free, we're not entitled to tell him how to write?
  - fasterthanlime 5y ago
    Not to worry, we also spend some time messing with strace and GDB. I see it more as "giving a working knowledge of Rust" than a distraction. Some of these crates/tools are lifechangers, I wouldn't want folks to miss out on them. I'm sorry yodelshady has run into a build error for cargo-edit (was it OpenSSL? screw OpenSSL) but also: the more folks run into these issues and report them, the sooner they're fixed. Also! I think cargo-edit should just be adopted by the default cargo distribution because it's all-around really good.
    - PudgePacket 5y ago
      I only learnt about the logging and tracing crates existed from this blog post and I've already started using them, so thank you!
      - hmfrh 5y ago
        The Zero to Rust[1] series also has a fairly comprehensive look into these. [1]: https://www.lpalmieri.com/posts/2020-09-27-zero-to-production-4-are-we-observable-yet/ https://www.lpalmieri.com/posts/2020-09-27-zero-to-productio...
    - mikro2nd 5y ago
      I have to agree with the first comment: Speaking as someone who was a tech trainer and writer of course materials, tutorial, etc. for a couple of decades, if ``carge-edit`` is not core to what you're trying to teach, leave it out. Get your own desires (ego) out of the way, and stick to teaching what you need to teach. Leave the distractions/side-tracks/proselytising to a separate lesson. You think of the lesson as "Giving a Working Knowledge of Rust"? Great! Then title it accordingly and don't dress it up in "futures" camouflage -- you're doing your readers/learners a disservice. Many, many tutorials make this mistake.
      - kbenson 5y ago
        > Then title it accordingly and don't dress it up in "futures" camouflage -- you're doing your readers/learners a disservice. In their defense, the title is "Understanding Rust futures by going way too deep", and I think advice and tangents on best practices modules is easily covered by "going way too deep". > you're doing your readers/learners a disservice. Not every tutorial needs to be the same, cater to the same type of person, or try to explain things in the same manner. I think the bigger disservice to readers/learners would be to homogenize tutorials into what you think is best. That might work best for a certain type of person at a certain skill level, but that doesn't mean everyone will be served well by that. And as the comments here illustrate, it's not like there's a dearth of Rust futures tutorials and explanations. There's definitely room for a rambling, informal, irreverent, meandering take (it's the only thing that kept me reading until the end, or even past the first page or two).
- ohazi 5y ago
  > I am surprised Rust doesn't lean into the type system more for declaring async, as I really like that way of framing it. Admittedly, I'm probably not writing ```impl Future<Output = Result<(), Report>> + 'a``` all that much in practice. Futures was stabilized before the async syntax, so this used to be the only way you could do it. But those were definitely the bad old days. A handful of (brave) developers and library writers wrote async code like this back then when they really needed to, but the vast majority waited for async syntax to stabilize. Those state machines get ugly real fast.
  - majewsky 5y ago
    > Futures was stabilized before the async syntax, so this used to be the only way you could do it. While that's technically true (std::future was in 1.36 and async/await syntax was in 1.39), that's a very short timespan of only a few months. You are probably referring to the multiple years where futures only existed as addon libraries where their design was iterated on before the final design of std::future was settled on.
- j1elo 5y ago
  I stand by my opinion that any kind of documentation or tutorial should pass a validation step where the author does copy/paste of all their commands into a clean Docker container or VM image. If any error happens, your work is not done yet. If you need to instruct users to install a compiler or an extra tool, so be it. It just must work as-is on a clean system (for which using Docker is easiest for me, but a clean VM or any other alternative is fine too).
  - gmac 5y ago
    Preferably the author doesn't copy and paste: preferably there's a documentation build step where everything gets run, relevant outputs get inserted into the docs, and any errors get caught. This is how my docs for Zapatos[1] work, for instance. :) [1] https://jawj.github.io/zapatos/ https://jawj.github.io/zapatos/
    - j1elo 5y ago
      It is probably too much to ask everyone follow the same approach, so I'll be happy if people just test their own commands by hand. But your doc build step that validates code and catches errors? That's the ideal setup, for sure. Great job!
  - chromatin 5y ago
    > I stand by my opinion that any kind of documentation or tutorial should pass a validation step where the author does copy/paste of all their commands into a clean Docker container or VM image. If any error happens, your work is not done yet. Very much this. We often write scientific software with (command-line) instructions for nontechnical or sort-of-technical users and have learned from experience to not assume anything about the user's environment, or background knowledge, and to try to replicate our documentation's instructions line-by-line!
- Zababa 5y ago
  On the other hand, I really like that way of giving "best practice" advice, as it's really useful advice. You end up with a deeper understanding of futures, but also concrete advice that you can apply easily.
deleted 5y ago
[deleted]
the_duke 5y ago
For an excellent, in depth introduction to async in Rust I recommend this post [1], part of the "Writing an OS in Rust" series. This seems to be more of a hands on guide for the ecosystem. [1] https://os.phil-opp.com/async-await/ https://os.phil-opp.com/async-await/
- tikkabhuna 5y ago
  Another hands on guide I really rate is Tokio's [1]. You create a very minimal Redis clone progressively by swapping out synchronous for async. [1] https://tokio.rs/tokio/tutorial https://tokio.rs/tokio/tutorial
MrBuddyCasino 5y ago
Gaze upon our work, and rejoice: pub fn try_join<A, B, AR, BR, E>(a: A, b: B) -> impl Future<Output = Result<(AR, BR), E>> where A: Future<Output = Result<AR, E>>, B: Future<Output = Result<BR, E>> I get why its like that, but I really wished this wasn't a typical Rust generics signature.
- jcelerier 5y ago
  Generics are about encoding supplemental bits of information in the type system. That information has to be spelled out somewhere
  - cormacrelf 5y ago
    Yes, however in my experience you typically have to write out constraints like these a bunch of times. Refactoring a generic trait with 10 implementations is a nightmare. A struct with a couple of variations on the constraints for its impl blocks is also painful beyond like one generic param. It has to be spelled out somewhere, but then at minimum somewhere else, and in general about 3-4 more times. I see why Haskell folks like their type level functions.
    - tinco 5y ago
      In Haskell I used to do this trick where if I wanted to neatly type annotate, but my function had some crazy signature, I'd compile the program, but passing the function into another function in an intentional incorrect way. The compiler would complain "expected A but got B". And I would simply copy B and make it the type annotation :P Nowadays the editor would help you do that. When I'm coding Rust the editor shows the types inline in a small font, which is very helpful.
    - myrrlyn 5y ago
      yeah, bounds written on the type decl shouldn't be repeated on the impl. the current typeck is not smart enough to do that but the ever-"nearly there we promise" replacement is. it'll land one day. they promise
- 5y ago
zwirbl 5y ago
Haven't read this article yet, but I've enjoyed most of the previous ones from amos
DiabloD3 5y ago
Woah, he uses Iosevka for his mono font. :D Edit: Whoever is downvoting, Iosevka is one of the favorites of the HN crowd, and also my own daily driver. It is worth taking for a spin if you haven't yet.
perryizgr8 5y ago
> 107 minute read Hmmm... that is truly way too deep.
DixieDev 5y ago
Not related to async in Rust, but a 26 sec clean build time for a project with 69 dependencies and no code of its own just to get to the point where we can "do something useful" feels a little wasteful.
- kungito 5y ago
  I'd say it's pretty unfair to take clean build time to consider doing something useful. 98% of my rust builds are incremental builds which take 10-15 seconds for a project I have been working on for 9 months full time and I think my dependency tree has easily 1000 dependencies.
  - DixieDev 5y ago
    True, it's not a real problem in most scenarios if incremental compile times are good. I still feel uneasy depending on so many other crates, but this seems to be a level of paranoia that others in the community don't share. Having 1000 dependencies sounds crazy to me! If there's a bug in even one of them that affects you then there's gonna be a lot of digging to figure out the cause, and possibly multiple pull requests to get it fixed. I think my CPU would spend a bit longer than 10-15 secs on the linker step, too.
    - fasterthanlime 5y ago
      Compile/link times are definitely a discussion topic. Here's some quick tips: * Incremental compilation helps locally, not so much in CI, unless you save/restore the whole cache which gets large quickly and evicting the out-of-date objects is non-trivial. * In CI, sccache helps, but not as much as I'd like: there's a bunch of things that are non-cacheable, notably crates that pull in & compile C/C++ code (I'd trade 2 c-bindings crates against 12 Rust-only crates any day for that reason alone) * Splitting stuff across different crates helps parallelizing the build (and caching it better), which may be one of the reasons some projects end up having "1000 dependencies" (although I've rarely seen upwards of 600) * For incremental builds, linking does become the bottleneck. Full LTO is especially slow, Thin LTO is better. Switching to LLD improves thing. I'm hopeful that mold will improve things some more. * Re multiple PRs: thankfully crate families tend to live in the same repository, so fixing something across both tracing / tracing-subscriber could be a single PR, for example. I wish the Rust community invested more in build caching, but even with the current state of things, there's often steps you can take to make things better.
pictur 5y ago
As someone who has never played a pass, I have a question. Every article I see about rust says it's all very simple. this is something that can change from person to person yes, but is it like that for you? I wonder what people who play with rust think about it. (this is just a question please don't take it as an offensive comment)
- fasterthanlime 5y ago
  "Simple" can mean many things. For me Rust is "simple" because it has great tooling / great diagnostics, so when I get something wrong, most of the time it can tell me exactly what it is (and how to fix it!), and only occasionally does it send me down a rabbit hole. Rust is also "simple" because it's memory-safe - if I can get something to compile (without unsafe code), then I know there's no use-after-free, double-free, buffer overflows, etc. I can concentrate my mind on the business logic (where bugs still can and do happen). Finally, Rust is "simple" because it lets me build abstractions so I can see more clearly what I'm doing. At work we had a Go codebase full of goroutines and channels - I got tired of it and did a proof of concept in Rust: now it's just one async Stream, you can see the data flow clearly and worry about each stage of the pipeline in isolation, instead of having to jump across many different source files. But no, it's not all very simple. There's plenty to learn, and some of the core concepts (ownership / lifetimes) are fairly hard to wrap your mind around, even if you've encountered something similar in other languages. My take is that it's worth it - you'll be very slow at first, but as you get more comfortable with it, so will you get faster.
- hmfrh 5y ago
  > this is something that can change from person to person yes, but is it like that for you? In my opinion it really depends on your background. For example, Rust uses the `i32` type as the standard integer. If you come from a C/C++ background this is probably not that weird, you probably know how many bits there are in 4 bytes off hand, there's `int32_t` standard types and you probably have experienced not wanting an integer with a variable size that depends on the platform. If however you come from Python, Javascript, or maybe even Java, this might range from a little weird to very weird. Python only has the `int` type, no unsigned types and you might not know how many bits make up a "standard" integer, or even exactly how bits, signedness and integer sizes fit together. Java doesn't have unsigned types and doesn't have issues with sizes depending on the exact platform, so they might not understand why you would want the amount of bits right there in the type name. This is just for the standard integer type. If you consider how many design decisions are a direct result of working on low level, high correctness required C++ code, the "simplicity" could range from completely simple to very much not simple.
beltsazar 5y ago
I've read many articles explaining Rust async and I think the best one is the async chapter of Programming Rust (2nd edition). It's not too daunting for someone new to async concepts, and yet comprehensive enough for someone curious about how Rust async works (future, waker, pinning). Amazon link: https://www.amazon.com/dp/1492052590 https://www.amazon.com/dp/1492052590 Direct link to the chapter (if you subscribe to O'Reilly/Safari): https://learning.oreilly.com/library/view/programming-rust-2nd/9781492052586/ch20.html#asynchronous-programming https://learning.oreilly.com/library/view/programming-rust-2...
thrwyoilarticle 5y ago
>When deciding which article to read during their coffee break, people usually open several websites at the exact same moment, and read whichever article loads first. >And that's a fact. You can quote me on that because, well, who's going to go and verify that? That sounds like a lot of work. Just trust me on this. Sure, but I click away when it hurts to read.
Walther 5y ago
Thank you once again for writing such an excellent article <3 In contrast to some other comments here, I love the narrative style. After all, these are half tutorials half entertainment - and the entertainment keeps it enticing. I would not have spent nearly two hours reading about the nitty-gritty of futures implementation details today if it weren't for the way you write about things. To put it the other way - if I wanted to read a super condensed, concise reference of how these things work, I would, you know, read a reference. The narrative style of building things on your own, solving problems as you face them, picking tools and libraries on the way, feels relatable as the everyday lived experience of software development. It feels appropriate for the medium - it's a blog after all! Besides, blogs are the perfect place to mention the tiny useful tools and libraries people might not have heard of. References and books might err on the side of being "neutral" and not making opinionated recommendations, but this can also lead to unknown unknowns. People don't know what they are missing out on! Luckily, blogs have more freedom here. Mentioning things like `cargo-edit`, `serde` and others can probably make some readers feel like "one of today's lucky ten thousand" - enjoying the newly-gained quality of life improvements.
- deleted 5y ago
  [deleted]
- shantanugoel 5y ago
  +1 I enjoyed the writing style thoroughly. It made understanding the concepts super simple and fun to go through.
- ordu 5y ago
  Yeah! This article not just explains how async works, the author show research methods to learn how async works. How something works is transient knowledge, how to learn how something works is timeless priceless knowledge. The methods include strategically inserted panic!, some nice crates with examples how to use them... I cannot say I've learnt a lot about async, but I got few insights on technicalities of self-learning.
masklinn 5y ago
> I don't know! Why do they truncate stuff like that? Because Linux limits thread names to 15 bytes (not including the required terminal nul). Other unices are better though not great, macOS is 63 I think. By comparison Windows allows 32… kilobytes. Though “runtime” is probably what should be shortened here, “rt” is literally the name of the tokio features for configuring the runtime, so that would likely be more understandable than the trailing “-w”.
layoutIfNeeded 5y ago
What an obnoxious style of writing.
- masklinn 5y ago
  Don’t read it? Some people enjoy amos’ style, others don’t. If that’s not your cup of tea you can just gravitate other presentation styles, it’s not like that’s in short supply.
- libria 5y ago
  > What an obnoxious style of writing. It's good to see counter viewpoints but share what you or others would be looking for instead. "This sucks" isn't helpful. "This part sucks but would be awesome if it {did this}" is.
- codezero 5y ago
  I usually can't keep my focus in an article that is so long, but the narrative style helped keep me hooked through the entire thing.
Subsentient 5y ago
The problem with analyzing stuff like this about Rust, is that even Rust itself has no idea how it works, or what syntax and semantics actually mean. Rust doesn't merely lack a language standard, its reference and advanced documentation is also a very unfunny joke. You can learn just enough to have some idea of what you're doing, and then you're stuck huffing unicorn queefs because nobody including the compiler devs knows the exact rules on how syntax is supposed to work. Magical shortcuts, syntax, and borrow checker tricks are added and removed at random with no rhyme or reason, and Rust severely punishes the programmer for wanting to know how the language actually works, since there are usually no answers, or worse, wrong answers, for them to find. I still use Rust because the memory safety without a GC is wonderful, but please, don't even try to make it look like Rust has some official documentation or behavior. It most definitely does not, and what really disturbs me is how so many of the Rust community don't even want a standard. So OP, everything you've just done, will be rendered obsolete and woefully wrong in a couple years when Rust serves up another opaque unicorn-burger.
- deleted 5y ago
  [deleted]
- nynx 5y ago
  What? Are you sure you're looking at the right language? Or maybe you've confused stable rust and nightly rust?
  - Subsentient 5y ago
    No, I'm definitely on the right language. For example, lots of magical implicit reborrows with fuck all in the documentation, pattern matches and specifiers with no formal behavior defined, lints that change constantly. I do use nightly generally, simply because stable is crippled in features, but the same issues apply to stable. I should mention that even when they adjust these things in a stable release, they never fix the documentation to reflect it. And in the case of many of these things, they aren't documented at all, except perhaps a couple sentences about their existence.
    - 5y ago
the__alchemist 5y ago
What are y'all's thoughts on Async in embedded? A portion of the Rust embedded OSS community really likes it as an abstraction over interrupts, critical sections etc. I've been burned by over-complexity and coloring problems with it before, eg in web programming. Do you think it's a good fit? Use-case-dependent? Maybe for things like RF?
- steveklabnik 5y ago
  We don't need it, so we don't use it. I think it's fantastic that it's possible, and I think that if you do need it, it's great to have it.
- floatboth 5y ago
  There is an executor project: https://github.com/embassy-rs/embassy https://github.com/embassy-rs/embassy I'm very excited about this for one simple stupid reason: sleep(). Awaiting a timer delay deep inside some code is gonna be amazing. With typical sync code, you basically have to split your code and queue up the next work somewhere that would be picked up by the timer interrupt… basically doing the whole state-keeping that async would do for you.
  - the__alchemist 5y ago
    That sound like a nice advantage. I'd like to give this a shot on a future project. Its creator and I share some API design ideas, like using less typestate programming than other conventions, but I haven't used Embassy due to being cautious of Async. (The Typestate thing is a tangent; short explanation is it lets you check for misconfigured pins etc at compile time, but makes syntax significantly more verbose and harder to refactor)
ArchieMaclean 5y ago
Based on my understanding from the article... Iterators change over space, Futures change over time. In iterator, every time you call 'next' you get a value until the end when you get nothing. In Future, every time you call 'poll' you get nothing until the end when you get a value.
jokethrowaway 5y ago
This is a pretty great article but I don't get the hate for futures not executing immediately and requiring an .await. It's great to have pure futures-returning functions and being able to isolate side effects easily. I find it much easier to reason with (coming from mainly JS, where everything is executed as soon as a promise is created).
agolliver 5y ago
Not sure a better place to ask this, but we're getting to the point in most OSs where basically anything you want can finally be done truly async (non-blocking). Except that DNS apparently still blocks[0], so usually things farm DNS requests off to their own blocking thread pools (the author ends up disabling DNS just so they can "prove" everything works with just a single thread) What's so fundamentally difficult about writing an async/non-blocking DNS resolver? Is it just a lack of a real need for it? [0] (quote: People have been trying to build asynchronous DNS resolvers for decades without success. Can it be done? Yes. Has it been done? No. ) https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c#gistcomment-3819393 https://gist.github.com/djspiewak/46b543800958cf61af6efa8e07...
- otabdeveloper4 5y ago
  On the OS level there is no difference between "sync" and "async". They become different in programming language runtimes.
  - agolliver 5y ago
    There is a difference between blocking and non-blocking, which is what I am asking about. Language runtimes apparently cannot provide non-blocking DNS (as evidenced by this article, and the other one I linked which was on HN a week or so ago). DNS seems like it perfectly fits the model of "give me a buffer (to hold the resolved IP), and I will wake you up when it's filled". Maybe io_uring (/ IOCP?) already can support this, and these articles are just mistaken about the current (or soon-to-be) state-of-the-art? Or is there some fundamental reason about DNS that make writing a non-blocking resolver very very difficult? It's just very odd to me that userspace apps are creating their own blocking thread pools just to run DNS stuff, when they can do seemingly everything else with just a single thread if they wanted to. (edited a few times, sorry if I caught someone who was mid-reply)
    - otabdeveloper4 5y ago
      > There is a difference between blocking and non-blocking Not really. "Blocking" means "tell the scheduler to mark the process idle until operation finishes", while "non-blocking" means "don't mark the process but flip a semaphore bit when operation finishes". The point of view of the OS the difference is just which bit to flip.

475 points by tempodox 5y ago

I am surprised Rust doesn't lean into the type system more for declaring async, as I really like that way of framing it. Admittedly, I'm probably not writing ```impl Future<Output = Result<(), Report>> + 'a``` all that much in practice. I like fasterthanlime's writeups but it does have one particular bugbear for me, that seems particularly common for Rust (and Python, to be fair) feature writeups: environment-based yak shaving because the author likes the mostly-unrelated library <X>. The very first thing that happens if I try to reproduce this code is a build error for cargo-edit, which is totally unrelated to the feature being taught, yet if I wanted to follow the writeup, I'd have to fix it. I do really have this bugbear, particularly with Rust tutorials

richardwhiuk 5y ago
This article takes ages to get to futures because the writer seems to want to go on some tour of Rust crates first.
- cormacrelf 5y ago
  IMO if your article is long enough that it crashes MobileSafari, it should have a table of contents. Or it should be a book chapter. It’s a really obvious criticism of all of fasterthanlime’s posts that scarcely bears repeating at this point but they’re not getting any shorter. I get that he has a lot to say but I just don’t have time to wade through all of that. What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? It’s ok to teach both things but why together?!
  - fasterthanlime 5y ago
    Ok I promise I'm going to stop replying to those comments (because as you pointed out they show up on every post), but! Rust is not the only language with a decent type system out there. I can definitely see some ML/Haskell/Scala/etc. folks who would quickly pick up on all the Future stuff, drawing parallels with their usual language of choice, yet being completely unaware of all the nice tooling around Rust. It's one of its main selling points for me! Re cargo-edit specifically, it also lets me stay in flow while writing these: instead of having to repeat "we'll just edit Cargo.toml, make sure to have that under [dependencies], uh this time we need a feature so the right-hand-side needs to be an object, not a string" - instead I can just copy into the article the actual commands I run on my side, and if a quick paragraph about where that subcommand comes from unblocks even one person then it's worth it.
    - nindalf 5y ago
      I wouldn’t worry too much about it. You have your style, and it clearly works for the people who like your work. Everyone likes different things, and it doesn’t make sense to force homogeneity. It’s ok to cater to niches.
    - WJW 5y ago
      FWIW, I fall exactly into that niche. I've never written a line of Rust in my life but am quite competent with Haskell and Ruby and have dabbled in at least half a dozen others. Seeing everyday `cargo` use made me think of bundler and the Future/Result type signatures were easy to map onto Async/Either and the like. So yes, it's a long article but I didn't mind that in the least. Keep it up!
    - kelnos 5y ago
      Ignore the naysayers in this thread; I don't really get why people think it's ok nitpick how you've decided to present a particular topic that you publish to the world free of charge. It's... kinda rude, IMO. I really enjoyed reading this piece, and had never heard of cargo-edit, so thank you for that. As a Scala person, I've actually struggled with Rust's Future, because it's quite different and requires a new mental model of how things work, which I think you did a great job providing.
  - myrrlyn 5y ago
    > What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? My colleagues. Many people are skilled programmers jumping into obscure problems with only a baseline level of familiarity with the greater environment.
    - Filligree 5y ago
      Or for example me. cargo-edit looks useful. None of the tutorials I've skimmed have mentioned it.
  - kelnos 5y ago
    > What slice of Rust users are both such noobs they haven’t heard of cargo-edit and can’t Google it, but also so advanced they are interested in implementing Future manually to deepen their understanding? raises hand I'm not sure how (or why) I'd google cargo-edit, because it's one of those things that you don't even think about until someone tells you about it, and then you're happy they did. While I certainly wouldn't call myself a Rust expert, I'm no noob either, and am intensely interested in diving deep into topics like this. (Can we also dispense with the gatekeeping put-downs like "noob", please? It's unnecessary and rude.) > It’s ok to teach both things but why together?! Because the author felt like it, and since we're consuming his work for free, we're not entitled to tell him how to write?
- fasterthanlime 5y ago
  Not to worry, we also spend some time messing with strace and GDB. I see it more as "giving a working knowledge of Rust" than a distraction. Some of these crates/tools are lifechangers, I wouldn't want folks to miss out on them. I'm sorry yodelshady has run into a build error for cargo-edit (was it OpenSSL? screw OpenSSL) but also: the more folks run into these issues and report them, the sooner they're fixed. Also! I think cargo-edit should just be adopted by the default cargo distribution because it's all-around really good.
  - PudgePacket 5y ago
    I only learnt about the logging and tracing crates existed from this blog post and I've already started using them, so thank you!
    - hmfrh 5y ago
      The Zero to Rust[1] series also has a fairly comprehensive look into these. [1]: https://www.lpalmieri.com/posts/2020-09-27-zero-to-production-4-are-we-observable-yet/ https://www.lpalmieri.com/posts/2020-09-27-zero-to-productio...
  - mikro2nd 5y ago
    I have to agree with the first comment: Speaking as someone who was a tech trainer and writer of course materials, tutorial, etc. for a couple of decades, if ``carge-edit`` is not core to what you're trying to teach, leave it out. Get your own desires (ego) out of the way, and stick to teaching what you need to teach. Leave the distractions/side-tracks/proselytising to a separate lesson. You think of the lesson as "Giving a Working Knowledge of Rust"? Great! Then title it accordingly and don't dress it up in "futures" camouflage -- you're doing your readers/learners a disservice. Many, many tutorials make this mistake.
    - kbenson 5y ago
      > Then title it accordingly and don't dress it up in "futures" camouflage -- you're doing your readers/learners a disservice. In their defense, the title is "Understanding Rust futures by going way too deep", and I think advice and tangents on best practices modules is easily covered by "going way too deep". > you're doing your readers/learners a disservice. Not every tutorial needs to be the same, cater to the same type of person, or try to explain things in the same manner. I think the bigger disservice to readers/learners would be to homogenize tutorials into what you think is best. That might work best for a certain type of person at a certain skill level, but that doesn't mean everyone will be served well by that. And as the comments here illustrate, it's not like there's a dearth of Rust futures tutorials and explanations. There's definitely room for a rambling, informal, irreverent, meandering take (it's the only thing that kept me reading until the end, or even past the first page or two).
ohazi 5y ago
> I am surprised Rust doesn't lean into the type system more for declaring async, as I really like that way of framing it. Admittedly, I'm probably not writing ```impl Future<Output = Result<(), Report>> + 'a``` all that much in practice. Futures was stabilized before the async syntax, so this used to be the only way you could do it. But those were definitely the bad old days. A handful of (brave) developers and library writers wrote async code like this back then when they really needed to, but the vast majority waited for async syntax to stabilize. Those state machines get ugly real fast.
- majewsky 5y ago
  > Futures was stabilized before the async syntax, so this used to be the only way you could do it. While that's technically true (std::future was in 1.36 and async/await syntax was in 1.39), that's a very short timespan of only a few months. You are probably referring to the multiple years where futures only existed as addon libraries where their design was iterated on before the final design of std::future was settled on.
j1elo 5y ago
I stand by my opinion that any kind of documentation or tutorial should pass a validation step where the author does copy/paste of all their commands into a clean Docker container or VM image. If any error happens, your work is not done yet. If you need to instruct users to install a compiler or an extra tool, so be it. It just must work as-is on a clean system (for which using Docker is easiest for me, but a clean VM or any other alternative is fine too).
- gmac 5y ago
  Preferably the author doesn't copy and paste: preferably there's a documentation build step where everything gets run, relevant outputs get inserted into the docs, and any errors get caught. This is how my docs for Zapatos[1] work, for instance. :) [1] https://jawj.github.io/zapatos/ https://jawj.github.io/zapatos/
  - j1elo 5y ago
    It is probably too much to ask everyone follow the same approach, so I'll be happy if people just test their own commands by hand. But your doc build step that validates code and catches errors? That's the ideal setup, for sure. Great job!
- chromatin 5y ago
  > I stand by my opinion that any kind of documentation or tutorial should pass a validation step where the author does copy/paste of all their commands into a clean Docker container or VM image. If any error happens, your work is not done yet. Very much this. We often write scientific software with (command-line) instructions for nontechnical or sort-of-technical users and have learned from experience to not assume anything about the user's environment, or background knowledge, and to try to replicate our documentation's instructions line-by-line!
Zababa 5y ago
On the other hand, I really like that way of giving "best practice" advice, as it's really useful advice. You end up with a deeper understanding of futures, but also concrete advice that you can apply easily.

deleted 5y ago

[deleted]

the_duke 5y ago

For an excellent, in depth introduction to async in Rust I recommend this post [1], part of the "Writing an OS in Rust" series. This seems to be more of a hands on guide for the ecosystem. [1] https://os.phil-opp.com/async-await/ https://os.phil-opp.com/async-await/

tikkabhuna 5y ago
Another hands on guide I really rate is Tokio's [1]. You create a very minimal Redis clone progressively by swapping out synchronous for async. [1] https://tokio.rs/tokio/tutorial https://tokio.rs/tokio/tutorial

MrBuddyCasino 5y ago

Gaze upon our work, and rejoice: pub fn try_join<A, B, AR, BR, E>(a: A, b: B) -> impl Future<Output = Result<(AR, BR), E>> where A: Future<Output = Result<AR, E>>, B: Future<Output = Result<BR, E>> I get why its like that, but I really wished this wasn't a typical Rust generics signature.

jcelerier 5y ago
Generics are about encoding supplemental bits of information in the type system. That information has to be spelled out somewhere
- cormacrelf 5y ago
  Yes, however in my experience you typically have to write out constraints like these a bunch of times. Refactoring a generic trait with 10 implementations is a nightmare. A struct with a couple of variations on the constraints for its impl blocks is also painful beyond like one generic param. It has to be spelled out somewhere, but then at minimum somewhere else, and in general about 3-4 more times. I see why Haskell folks like their type level functions.
  - tinco 5y ago
    In Haskell I used to do this trick where if I wanted to neatly type annotate, but my function had some crazy signature, I'd compile the program, but passing the function into another function in an intentional incorrect way. The compiler would complain "expected A but got B". And I would simply copy B and make it the type annotation :P Nowadays the editor would help you do that. When I'm coding Rust the editor shows the types inline in a small font, which is very helpful.
  - myrrlyn 5y ago
    yeah, bounds written on the type decl shouldn't be repeated on the impl. the current typeck is not smart enough to do that but the ever-"nearly there we promise" replacement is. it'll land one day. they promise
5y ago

zwirbl 5y ago

Haven't read this article yet, but I've enjoyed most of the previous ones from amos

DiabloD3 5y ago

Woah, he uses Iosevka for his mono font. :D Edit: Whoever is downvoting, Iosevka is one of the favorites of the HN crowd, and also my own daily driver. It is worth taking for a spin if you haven't yet.

perryizgr8 5y ago

> 107 minute read Hmmm... that is truly way too deep.

DixieDev 5y ago

Not related to async in Rust, but a 26 sec clean build time for a project with 69 dependencies and no code of its own just to get to the point where we can "do something useful" feels a little wasteful.

kungito 5y ago
I'd say it's pretty unfair to take clean build time to consider doing something useful. 98% of my rust builds are incremental builds which take 10-15 seconds for a project I have been working on for 9 months full time and I think my dependency tree has easily 1000 dependencies.
- DixieDev 5y ago
  True, it's not a real problem in most scenarios if incremental compile times are good. I still feel uneasy depending on so many other crates, but this seems to be a level of paranoia that others in the community don't share. Having 1000 dependencies sounds crazy to me! If there's a bug in even one of them that affects you then there's gonna be a lot of digging to figure out the cause, and possibly multiple pull requests to get it fixed. I think my CPU would spend a bit longer than 10-15 secs on the linker step, too.
  - fasterthanlime 5y ago
    Compile/link times are definitely a discussion topic. Here's some quick tips: * Incremental compilation helps locally, not so much in CI, unless you save/restore the whole cache which gets large quickly and evicting the out-of-date objects is non-trivial. * In CI, sccache helps, but not as much as I'd like: there's a bunch of things that are non-cacheable, notably crates that pull in & compile C/C++ code (I'd trade 2 c-bindings crates against 12 Rust-only crates any day for that reason alone) * Splitting stuff across different crates helps parallelizing the build (and caching it better), which may be one of the reasons some projects end up having "1000 dependencies" (although I've rarely seen upwards of 600) * For incremental builds, linking does become the bottleneck. Full LTO is especially slow, Thin LTO is better. Switching to LLD improves thing. I'm hopeful that mold will improve things some more. * Re multiple PRs: thankfully crate families tend to live in the same repository, so fixing something across both tracing / tracing-subscriber could be a single PR, for example. I wish the Rust community invested more in build caching, but even with the current state of things, there's often steps you can take to make things better.

pictur 5y ago

As someone who has never played a pass, I have a question. Every article I see about rust says it's all very simple. this is something that can change from person to person yes, but is it like that for you? I wonder what people who play with rust think about it. (this is just a question please don't take it as an offensive comment)

fasterthanlime 5y ago
"Simple" can mean many things. For me Rust is "simple" because it has great tooling / great diagnostics, so when I get something wrong, most of the time it can tell me exactly what it is (and how to fix it!), and only occasionally does it send me down a rabbit hole. Rust is also "simple" because it's memory-safe - if I can get something to compile (without unsafe code), then I know there's no use-after-free, double-free, buffer overflows, etc. I can concentrate my mind on the business logic (where bugs still can and do happen). Finally, Rust is "simple" because it lets me build abstractions so I can see more clearly what I'm doing. At work we had a Go codebase full of goroutines and channels - I got tired of it and did a proof of concept in Rust: now it's just one async Stream, you can see the data flow clearly and worry about each stage of the pipeline in isolation, instead of having to jump across many different source files. But no, it's not all very simple. There's plenty to learn, and some of the core concepts (ownership / lifetimes) are fairly hard to wrap your mind around, even if you've encountered something similar in other languages. My take is that it's worth it - you'll be very slow at first, but as you get more comfortable with it, so will you get faster.
hmfrh 5y ago
> this is something that can change from person to person yes, but is it like that for you? In my opinion it really depends on your background. For example, Rust uses the `i32` type as the standard integer. If you come from a C/C++ background this is probably not that weird, you probably know how many bits there are in 4 bytes off hand, there's `int32_t` standard types and you probably have experienced not wanting an integer with a variable size that depends on the platform. If however you come from Python, Javascript, or maybe even Java, this might range from a little weird to very weird. Python only has the `int` type, no unsigned types and you might not know how many bits make up a "standard" integer, or even exactly how bits, signedness and integer sizes fit together. Java doesn't have unsigned types and doesn't have issues with sizes depending on the exact platform, so they might not understand why you would want the amount of bits right there in the type name. This is just for the standard integer type. If you consider how many design decisions are a direct result of working on low level, high correctness required C++ code, the "simplicity" could range from completely simple to very much not simple.

beltsazar 5y ago

I've read many articles explaining Rust async and I think the best one is the async chapter of Programming Rust (2nd edition). It's not too daunting for someone new to async concepts, and yet comprehensive enough for someone curious about how Rust async works (future, waker, pinning). Amazon link: https://www.amazon.com/dp/1492052590 https://www.amazon.com/dp/1492052590 Direct link to the chapter (if you subscribe to O'Reilly/Safari): https://learning.oreilly.com/library/view/programming-rust-2nd/9781492052586/ch20.html#asynchronous-programming https://learning.oreilly.com/library/view/programming-rust-2...

thrwyoilarticle 5y ago

>When deciding which article to read during their coffee break, people usually open several websites at the exact same moment, and read whichever article loads first. >And that's a fact. You can quote me on that because, well, who's going to go and verify that? That sounds like a lot of work. Just trust me on this. Sure, but I click away when it hurts to read.

Walther 5y ago

Thank you once again for writing such an excellent article <3 In contrast to some other comments here, I love the narrative style. After all, these are half tutorials half entertainment - and the entertainment keeps it enticing. I would not have spent nearly two hours reading about the nitty-gritty of futures implementation details today if it weren't for the way you write about things. To put it the other way - if I wanted to read a super condensed, concise reference of how these things work, I would, you know, read a reference. The narrative style of building things on your own, solving problems as you face them, picking tools and libraries on the way, feels relatable as the everyday lived experience of software development. It feels appropriate for the medium - it's a blog after all! Besides, blogs are the perfect place to mention the tiny useful tools and libraries people might not have heard of. References and books might err on the side of being "neutral" and not making opinionated recommendations, but this can also lead to unknown unknowns. People don't know what they are missing out on! Luckily, blogs have more freedom here. Mentioning things like `cargo-edit`, `serde` and others can probably make some readers feel like "one of today's lucky ten thousand" - enjoying the newly-gained quality of life improvements.

deleted 5y ago
[deleted]
shantanugoel 5y ago
+1 I enjoyed the writing style thoroughly. It made understanding the concepts super simple and fun to go through.
ordu 5y ago
Yeah! This article not just explains how async works, the author show research methods to learn how async works. How something works is transient knowledge, how to learn how something works is timeless priceless knowledge. The methods include strategically inserted panic!, some nice crates with examples how to use them... I cannot say I've learnt a lot about async, but I got few insights on technicalities of self-learning.

masklinn 5y ago

> I don't know! Why do they truncate stuff like that? Because Linux limits thread names to 15 bytes (not including the required terminal nul). Other unices are better though not great, macOS is 63 I think. By comparison Windows allows 32… kilobytes. Though “runtime” is probably what should be shortened here, “rt” is literally the name of the tokio features for configuring the runtime, so that would likely be more understandable than the trailing “-w”.

layoutIfNeeded 5y ago

What an obnoxious style of writing.

masklinn 5y ago
Don’t read it? Some people enjoy amos’ style, others don’t. If that’s not your cup of tea you can just gravitate other presentation styles, it’s not like that’s in short supply.
libria 5y ago
> What an obnoxious style of writing. It's good to see counter viewpoints but share what you or others would be looking for instead. "This sucks" isn't helpful. "This part sucks but would be awesome if it {did this}" is.
codezero 5y ago
I usually can't keep my focus in an article that is so long, but the narrative style helped keep me hooked through the entire thing.

Subsentient 5y ago

The problem with analyzing stuff like this about Rust, is that even Rust itself has no idea how it works, or what syntax and semantics actually mean. Rust doesn't merely lack a language standard, its reference and advanced documentation is also a very unfunny joke. You can learn just enough to have some idea of what you're doing, and then you're stuck huffing unicorn queefs because nobody including the compiler devs knows the exact rules on how syntax is supposed to work. Magical shortcuts, syntax, and borrow checker tricks are added and removed at random with no rhyme or reason, and Rust severely punishes the programmer for wanting to know how the language actually works, since there are usually no answers, or worse, wrong answers, for them to find. I still use Rust because the memory safety without a GC is wonderful, but please, don't even try to make it look like Rust has some official documentation or behavior. It most definitely does not, and what really disturbs me is how so many of the Rust community don't even want a standard. So OP, everything you've just done, will be rendered obsolete and woefully wrong in a couple years when Rust serves up another opaque unicorn-burger.

deleted 5y ago
[deleted]
nynx 5y ago
What? Are you sure you're looking at the right language? Or maybe you've confused stable rust and nightly rust?
- Subsentient 5y ago
  No, I'm definitely on the right language. For example, lots of magical implicit reborrows with fuck all in the documentation, pattern matches and specifiers with no formal behavior defined, lints that change constantly. I do use nightly generally, simply because stable is crippled in features, but the same issues apply to stable. I should mention that even when they adjust these things in a stable release, they never fix the documentation to reflect it. And in the case of many of these things, they aren't documented at all, except perhaps a couple sentences about their existence.
  - 5y ago

the__alchemist 5y ago

What are y'all's thoughts on Async in embedded? A portion of the Rust embedded OSS community really likes it as an abstraction over interrupts, critical sections etc. I've been burned by over-complexity and coloring problems with it before, eg in web programming. Do you think it's a good fit? Use-case-dependent? Maybe for things like RF?

steveklabnik 5y ago
We don't need it, so we don't use it. I think it's fantastic that it's possible, and I think that if you do need it, it's great to have it.
floatboth 5y ago
There is an executor project: https://github.com/embassy-rs/embassy https://github.com/embassy-rs/embassy I'm very excited about this for one simple stupid reason: sleep(). Awaiting a timer delay deep inside some code is gonna be amazing. With typical sync code, you basically have to split your code and queue up the next work somewhere that would be picked up by the timer interrupt… basically doing the whole state-keeping that async would do for you.
- the__alchemist 5y ago
  That sound like a nice advantage. I'd like to give this a shot on a future project. Its creator and I share some API design ideas, like using less typestate programming than other conventions, but I haven't used Embassy due to being cautious of Async. (The Typestate thing is a tangent; short explanation is it lets you check for misconfigured pins etc at compile time, but makes syntax significantly more verbose and harder to refactor)

ArchieMaclean 5y ago

Based on my understanding from the article... Iterators change over space, Futures change over time. In iterator, every time you call 'next' you get a value until the end when you get nothing. In Future, every time you call 'poll' you get nothing until the end when you get a value.

jokethrowaway 5y ago

This is a pretty great article but I don't get the hate for futures not executing immediately and requiring an .await. It's great to have pure futures-returning functions and being able to isolate side effects easily. I find it much easier to reason with (coming from mainly JS, where everything is executed as soon as a promise is created).

agolliver 5y ago

Not sure a better place to ask this, but we're getting to the point in most OSs where basically anything you want can finally be done truly async (non-blocking). Except that DNS apparently still blocks[0], so usually things farm DNS requests off to their own blocking thread pools (the author ends up disabling DNS just so they can "prove" everything works with just a single thread) What's so fundamentally difficult about writing an async/non-blocking DNS resolver? Is it just a lack of a real need for it? [0] (quote: People have been trying to build asynchronous DNS resolvers for decades without success. Can it be done? Yes. Has it been done? No. ) https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c#gistcomment-3819393 https://gist.github.com/djspiewak/46b543800958cf61af6efa8e07...

otabdeveloper4 5y ago
On the OS level there is no difference between "sync" and "async". They become different in programming language runtimes.
- agolliver 5y ago
  There is a difference between blocking and non-blocking, which is what I am asking about. Language runtimes apparently cannot provide non-blocking DNS (as evidenced by this article, and the other one I linked which was on HN a week or so ago). DNS seems like it perfectly fits the model of "give me a buffer (to hold the resolved IP), and I will wake you up when it's filled". Maybe io_uring (/ IOCP?) already can support this, and these articles are just mistaken about the current (or soon-to-be) state-of-the-art? Or is there some fundamental reason about DNS that make writing a non-blocking resolver very very difficult? It's just very odd to me that userspace apps are creating their own blocking thread pools just to run DNS stuff, when they can do seemingly everything else with just a single thread if they wanted to. (edited a few times, sorry if I caught someone who was mid-reply)
  - otabdeveloper4 5y ago
    > There is a difference between blocking and non-blocking Not really. "Blocking" means "tell the scheduler to mark the process idle until operation finishes", while "non-blocking" means "don't mark the process but flip a semaphore bit when operation finishes". The point of view of the OS the difference is just which bit to flip.