5 ms·

Cancellations in async Rust

240 points by todsacerdoti 1y ago

CaptainOfCoit 1y ago
Less clickbaity title: Cancellations in async Rust. It's really not about "cancelling async Rust" which is what I expected, even if it didn't make much sense.
- happytoexplain 1y ago
  As in the pop-culture concept of cancelling? That's what you assumed the topic "cancelling async <language name>" was going to be about?? Or am I missing context?
  - deleted 1y ago
    [deleted]
  - hansvm 1y ago
    That's what I assumed. I know languages have to handle cancellations in async code, but Rust has had a fair amount of drama over the years, and I assumed the title was accurate and reflected that some drama was happening.
    - sunshowers 1y ago
      Appreciate the feedback here — definitely don't want the title to overshadow the work itself. Will keep this in mind for next time.
  - thehamkercat 1y ago
    they probably assumed something like some_running_async_task.cancel()
  - fulafel 1y ago
    Async in Rust isn't exactly universally loved, partly because async Rust is perceived to spread progressively to Rust libraries making it less optional to use. See eg https://bitbashing.io/async-rust.html https://bitbashing.io/async-rust.html, "Async Rust Is A Bad Language"
- sunshowers 1y ago
  As the author of the talk/blog post, I was definitely going for a bit of a moral valence in the title, in the sense that future cancellations are very hard to reason about and what I call the least Rusty part of Rust. But it admittedly is a bit clickbaity too.
  - acedTrex 1y ago
    I initially skipped reading it because i thought it was another drama post about maintainers a la all the nixos stuff lately.
    - bigiain 1y ago
      To balance the universe a bit, I read it expecting a drama post - then read it right through because it was at least as interesting as the drama post I'd expected. I also discovered Oxide through this, which looks interesting, except for the complete lack of pricing I can find on their site - which puts in my head as probably in the "If you need to ask the price you can't afford it" category...
      - jgord 1y ago
        I think Oxide should be renting out time on their hardware racks, as well as selling them to big orgs. Oxide looks to be superb engineering up and down the whole stack, and if it drives more rust code into linux all the better. Now that linode has been consumed by Akamai, we need an alternative.
- binary132 1y ago
  if only
  - zackmorris 1y ago
    +1 this. IMHO async is an anti-pattern, and probably the final straw that will prevent me from ever finishing learning Rust. Once one learns pass-by-value and copy-on-write semantics (Clojure, PHP arrays), the world starts looking like a spreadsheet instead of spaghetti code. I feel that a Rust-like language could be built with no borrow checker, simply by allocating twice the memory. Since that gets ever-less expensive, I'm just not willing to die on the hill of efficiency anymore. I predict that someday Rust will be relegated to porting scripting languages to a bare-metal runtime, but will not be recommended for new work. That said, I think that Rust would make a great teaching tool in an academic setting, as the epitome of imperative languages. Maybe something great will come of it, like Swift from Objective-C or Kotlin from Java. And having grown up on C++, I have a soft spot in my heart for solving the hard problems in the fastest way possible. Maybe a voxel game in Rust, I dunno.
    - sunshowers 1y ago
      Author here -- I'd recommend reading my blog post about how cargo-nextest uses Tokio + async Rust to handle very complex state machines: https://sunshowers.io/posts/nextest-and-tokio/ https://sunshowers.io/posts/nextest-and-tokio/
      - zackmorris 1y ago
        Ah cool, a couple of kudos to you: 1) I learned about pin in Rust to prevent values from moving in memory. 2) I learned about the html <summary> tag (the turndown arrows in your article that work with Javascript disabled) hah. I can see how dealing with stream and resource cleanup in async code could be a chore. It sounds like you were able to do that in a fairly declarative manner, which is what I always strive for as well. I think my hesitation with async is that I already went down that road early in my programming life with cooperative threads/multitasking on Mac OS 9 and earlier. There always seems to be yet another brittle edge case to deal with, so it can feel infuriating playing whack-a-mole until they're all nailed down. For example, pinning memory looks a lot like locking handles in Mac OS. Handles were pointers to pointers, so it was a bare hands way to implement a memory defragmenter before runtimes were smart enough to handle it. If apps used handles, then blocks of data could be unlocked, moved somewhere else in memory, and then re-locked. Code had to do an extra hop through each handle to get to the original pointer, which was a frequent source of bugs because one async process might be working on a block, yield, and then have another async process move the handle out from under it. The lock's state was stored in a flag in the memory manager, basically a small bit of metadata. I haven't investigated, but I suspect that Rust may be able to handle locking more efficiently, perhaps more like reference counting or the borrow checker where it can infer whether a pointer is locked without storing that flag somewhere (but I could be wrong). Apple abandoned handles when it migrated to OS 10 and Darwin inherited protected memory and better virtual memory from FreeBSD. Although now that I write this out, I'm not sure that they solved in-process fragmentation. I think they just give apps the full 32 or 64 bit address space so that effectively there is always another region available for the next allocation, and let the virtual memory subsystem consolidate 4k memory blocks into contiguous strips internally. The dereferencing of memory step became implicit rather than explicit, as well as hidden from apps, so that whole classes of bugs became unreachable. Anyway, that's why I prefer the runtime to handle more of this. I want strong guarantees that I can terminate a process and all locks inside it will get freed as well. I can pretty much rely on that even in hacky languages like PHP. My frustration with all of this is that we could/should have demanded better runtimes. We could have had realtime unixes where task switching and memory allocation were effectively free. Unfortunately the powers that be (Mac OS and Windows) had runtimes that were too entrenched with too many users relying on quirks and so they dragged their feet and never did better. Languages like Rust were forced to get very clever and go to the ends of the earth to work around that. Then when companies like Google and Facebook won the internet lottery, they pulled the ladder up behind them by unilaterally handing down decrees from on high that developers should use bare hands techniques, rather than putting real resources into reforming the fundamentals so that we wouldn't have to. What I'm trying to say is that your solution is clever and solves a common pattern in about the simplest way possible, but is not as simple as synchronous-blocking unix pipes to child processes in shell scripts. That's in no way a criticism. I have similar feelings about stuff like Docker and Kubernetes after reading about Podman. If we could magically go back and see the initial assumptions that led us down the road we're on, we might have tried different approaches. It's all of those roads not taken that haunt me, because they represent so much of my workload each day.
- deleted 1y ago
  [deleted]
- benatkin 1y ago
  It made sense to me, because I imagine a thread or coroutine as something that runs code as though it were interpreting something like psuedocode, whether it's doing that or not. So from my point of view an instance of async Rust is being cancelled - not the feature of the Rust project, but instances of code. This abstraction has served me well and facilitates stepping through code in a debugger, though I jump out of thinking it at that level when I need to think of it at a lower level.
bryanlarsen 1y ago
Timely! Was grumbling about this today as I added a "this function is cancel safe" to a new function's doc comment. I really hope we get async drop soon.
- 0x1ceb00da 1y ago
  I'm curious. Can you talk a little about that function?
  - bryanlarsen 1y ago
    Most common scenario in article: select!. I split out a "wait for X to be ready" from "X" so that the former could be on the left side of a select ARM, and the rest on the right side.
Panzerschrek 1y ago
One should always keep in mind that await is always a potential return point. So, using await between two actions which always should be performed together should be avoided.
- Spivak 1y ago
  That… seems bad? Like I guess it is what it is and you just have to deal with it but what if your "critical section" has two await calls? The code can be paused between them but it's such that it must eventually resume. Say making a change in the database and emitting an audit edit for that change. Is your only option to either not do that or put a big do not cancel sign on the function docs?
  - sunshowers 1y ago
    In general I think people end up gravitating towards using message passing or the actor model for this.
  - diarrhea 1y ago
    A DB action and audit emission have to run transactionally anyway. So on cancellation, the transaction times out and nothing is written. Bad but safe. The problem is the same on other platforms. For example, what if writing to the DB throws an exception if you’re on Python? Your app just dies, the transaction times out. Unfortunate but safe. If it does not run transactionally you have a problem in any execution scenario.
    - sunshowers 1y ago
      So, regarding transactions, absolutely you can throw them away on cancellation. But there's an interesting wrinkle here: if you use a connection pool like most users, and you were going to do the ROLLBACK at the end of your future on error, then that ROLLBACK wouldn't run it the future is cancelled! Then future operations reusing the same connection would be stuck in transaction la-la land. (This is related to the fact that Rust doesn't have async drop — you can't run async code on drop, other than spawning a new task to do the cleanup.) This is prong 3 of my cancel correctness framework (that the cancellation violates a system property, in this case a cleanup property.) The solution here is to ensure the connection is in a pristine state before handing it out the next time it's used.
CodeBrad 1y ago
This was one of my favorite talks from RustConf this year! The distinction between cancel safety and cancel correctness is really helpful. Glad to see it converted to a blog post. Talks are great, but blogs are much easier to share and reference.
- sunshowers 1y ago
  Thanks! I definitely prefer reading blog posts over watching talks as well.
- pornel 1y ago
  "Cancel correctness" makes a lot of sense, because it puts the cancellation in some context. I don't like the "cancel safety" term. Not only it's unrelated to the Rust's concept of safety, it's also unnecessarily judgemental. Safe/unsafe implies there's a better or worse behavior, but what is desirable for cancellation to do is highly context-dependent. Futures awaiting spawned tasks are called "cancellation safe", because they won't stop the task when dropped. But that's not an inherently safe behavior – leaving tasks running after their spawner has been cancelled could be a bug: piling up work that won't be used, and even interfering with the rest of the program by keeping locks locked or ports used. OTOH a spawn handle that stops the task when dropped would be called "cancellation unsafe", despite being a very useful construct specifically for propagating cleanup to dependent tasks.
Matthias247 1y ago
Some other material that has been written by me on that topic: - Proposal from 2020 about async functions which are forced to run to completion (and thereby would use graceful cancellation if necessary). Quite old, but I still feel that no better idea has come up so far. https://github.com/Matthias247/rfcs/pull/1 https://github.com/Matthias247/rfcs/pull/1 - Proposal for unified cancellation between sync and async Rust ("A case for CancellationTokens" - https://gist.github.com/Matthias247/354941ebcc4d2270d07ff0c6bf066c64 https://gist.github.com/Matthias247/354941ebcc4d2270d07ff0c6...) - Exploration of an implementation of the above: https://github.com/Matthias247/min_cancel_token https://github.com/Matthias247/min_cancel_token
ossopite 1y ago
I think the send/recv with a timeout example is very interesting, because in a language where futures start running immediately without being polled, I think the situation is likely to be the opposite way around. send with a timeout is probably safe (you may still send if the timeout happened, which you might be sad about, but the message isn't lost), while recv with a timeout is probably unsafe, because you might read the message out of the channel but then discard it because you selected the timeout completion instead. And the fix is similar, you want to select either the timeout or 'something is available' from the channel, and if you select the latter you can peek to get the available data.
- sunshowers 1y ago
  Thanks, that is a great point.
- lionkor 1y ago
  Isn't this exactly what cancellation-safety is all about?
Animats 1y ago
In the initial example, it's not clear what the desired behavior is. If the queue is full, the basic options are drop something, block and wait, or panic. Timing out on a block is usually deadlock detection. He writes "It turns out that this code is often incorrect, because not all messages make their way to the channel." Well, yes. You're out of resources. Now what? What's he trying to do? Get a clean program shutdown? That's moderately difficult in threaded programs, and async has problems, too. The use case here is unclear. The real use cases involve when you're sending messages back and forth to a remote site, and the remote site goes away. Now you need to dispose of the state on your end.
- AceJohnny2 1y ago
  > He writes They go by they/she https://sunshowers.io/about/ https://sunshowers.io/about/
  - SV_BubbleTime 1y ago
    [flagged]
- sunshowers 1y ago
  Ideally, what you would like to do is buffer up the message until there's space in the channel. I cover this later in the talk under "What can be done".
  - Animats 1y ago
    The double-loop thing effectively creates a blocking operation. Something you can do directly. Why all the complexity?
    - sunshowers 1y ago
      Agreed that in the narrow case of a timeout it doesn't buy you much (and things like network sockets often let you do timeouts in synchronous code). But often you do want the power to do selects and more complex state machines. I wrote a blog post a couple years ago talking about why a project I'm the author of, cargo-nextest, switched from sync Rust to async. https://sunshowers.io/posts/nextest-and-tokio/ https://sunshowers.io/posts/nextest-and-tokio/ To this day I'm not aware of a better way to express what's become a set of increasingly complex state machines (the most recent improvement being to make the state machines responsive to user input). Nextest's runner loop is structured mostly like a GUI event loop, but without explicit state machines. It's quite nice being able to write code that's this complex in a bug-free manner.
arifalkner 1y ago
Great talk! One thing that would have been nice to call out for n00bs like myself is how in SOP Futures can't cancelled. I knew that .await took ownership of the future so that drop() could not be called on it, so given how futures are lazy, it wasn't clear to me how to cancel a future after .await had been called. I later researched how select! and Abortable() did this, but could be nice to include a callout in the beginning of your talk if you ever do it again. Otherwise, nice work!
- sunshowers 1y ago
  Thanks! What does SOP mean in this context?
alembic_fumes 1y ago
I'm not understanding what the supposed problem with these futures getting cancelled is. Since futures are not tasks, as the post itself acknowledges, does it not logically follow that one should not expect futures to complete if the future is not driven to completion, for one reason or another? What else could even be expected to happen? The examples presented for "cancel unsafe" futures seem to me like the root of the problem is some sort of misalignment of expectations to the reality: Example 1: one future cancelled on error in the other let res = tokio::try_join!( do_stuff_async(), more_async_work(), ); Example 2: data not written out on cancellation let buffer: &[u8] = /* ... */; writer.write_all(buffer)?; Both of these cases are claimed to not be cancel-safe, because the work gets interrupted and so not driven to completion. But again, what else is supposed to happen? If you want the work to finish regardless of the async context being cancelled, then don't put it in the same async context but spawn a task instead. I feel like I must be missing something obvious that keeps me from understanding the author's issue here. I thought work getting dropped on cancellation is exactly how futures are supposed to work. What's the nuance that I'm missing?
- sunshowers 1y ago
  You're absolutely right! The problem is that this has introduced many bugs in our experience at Oxide. If you've already fully internalized the idea that futures are passive and can be cancelled at any await point, the talk is just a bunch of details.
  - alembic_fumes 1y ago
    I see. Do you suppose that the origin of these bugs is more about the difficulty of reasoning about the execution of deep async stacks, or does it come down to the developers holding an incorrect mental model of the Rust futures in their minds? I am asking because I've noticed that many developers with previous experience from "task-based" languages (specifically the JS/TS world) tend to grasp the basics of Rust async quickly enough, but then run into expectation-misalignment related problems similar to the examples that you used in your post. That in turn has made want to understand whether it is the Rust futures that are themselves difficult or strange, or whether it's a case of the Rust futures appearing simple and familiar, even though they are completely different in very subtle ways. I suppose that it's a combination of both.
foota 1y ago
https://sunshowers.io/posts/cancelling-async-rust/#the-pain-of-tokio-mutexes https://sunshowers.io/posts/cancelling-async-rust/#the-pain-... was the most interesting part of this for me, as I can totally see making mistakes like this.
- schmichael 1y ago
  I'm a Go developer and this was still useful for me! Obviously Rust devs are more accustomed to more assistance from their tools than Go devs, but just about every gotcha listed is something that can happen in Go with goroutines, channels, select, and other shared concurrency primitives.
  - jcgrillo 1y ago
    !m schmichael
nijaar 1y ago
for a sec i thought DEI was going too far is the title like that on purpose?
nofriend 1y ago
doesn't rust have raii?
- sunshowers 1y ago
  It does, but you can only run synchronous code on drop. This is what "async drop" is supposed to handle — things like issuing ROLLBACK statements to the database on cancellation. It also wouldn't help when you have no valid state to restore to, as in the mutex example in the post.
  - 0x1ceb00da 1y ago
    tokio-postgres handles this by just dispatching the "ROLLBACK" command in impl Drop and ignoring the response. https://github.com/rust-postgres/rust-postgres/blob/a7a49a90734c05664ed56356b61aa937cc2a6e6b/tokio-postgres/src/client.rs#L606 https://github.com/rust-postgres/rust-postgres/blob/a7a49a90... Is this not enough? What could go wrong? If the network connection dies or the task is cancelled, I'm assuming the database server cleans up the connection state and does a rollback automatically. And adding async Drop will probably add a whole new set of footguns.
    - _davide_ 1y ago
      > What could go wrong? LoL, an insane amount of things. TCP connections are an illusion of safely, for the purpose of database commits use UDP packets as a model instead, it'll be much closer to reality.
      - 0x1ceb00da 1y ago
        > an insane amount of things List a couple > TCP connections are an illusion of safely Why?
- deleted 1y ago
  [deleted]
tison 1y ago
Rust's Future is somehow like move semantics in C++, where you may leave a Future in an invalid state after it finishes. Besides, Rust adopts a stackless coroutine design, so you need to maintain the state in your struct if you would like to implement a poll-based async structure manually. These are all common traps. And now cancellations in async Rust are a new complement to state management in async Rust (Futures). When I'm developing the mea (Make Easy Async) [1] library, I document the cancel safety attribute when it's non-trivial. Additionally, I recall [2] an instance where a thoughtless async cancellation can disrupt the IO stack. [1] https://github.com/fast/mea https://github.com/fast/mea [2] https://www.reddit.com/r/rust/comments/1gfi5r1/comment/luidor3/ https://www.reddit.com/r/rust/comments/1gfi5r1/comment/luido...
beanjuiceII 1y ago
i am honestly glad i don't write rust anymore
dxxvi 1y ago
My Rust knowledge is too low to understand. However, thank you very much for the article. I hope AI will learn it and explain to me in a few years :-)
mleonhard 1y ago
I think that async in Rust has a significant devex/velocity cost. Unfortunately, nearly all of the effort in Rust libraries has gone into async code, so the async libraries have outpaced the threaded libraries. There was only one threaded web server, https://lib.rs/crates/rouille https://lib.rs/crates/rouille . It has 1.1M lines of code (including deps). Its hello-world example reaches only 26Krps on my machine (Apple M4 Pro). It also has a bug that makes it problematic to use in production: https://github.com/tiny-http/tiny-http/issues/221 https://github.com/tiny-http/tiny-http/issues/221 . I wrote https://lib.rs/crates/servlin https://lib.rs/crates/servlin threaded web server. It uses async internally. It has 221K lines of code. Its hello-world example reaches 102Krps on my machine. https://lib.rs/crates/ehttpd https://lib.rs/crates/ehttpd is another one but it has no tests and it seems abandoned. It does an impressive 113Krps without async, using only 8K lines of code. For comparison, the popular Axum async web server has 4.3M lines of code and its hello-world example reaches 190Krps on my machine. The popular threaded Postgres client uses Tokio internally and has 1M lines of code: http://lib.rs/postgres http://lib.rs/postgres . Recently a threaded Postgres client was released. It has 500K lines of code: https://lib.rs/crates/postgres_sync https://lib.rs/crates/postgres_sync . There was no ergonomic way to signal cancellation to threads, so I wrote one: https://crates.io/crates/permit https://crates.io/crates/permit . Rust's threaded libraries are starting to catch up to the async libraries! --- I measured lines of code with `rm -rf deps.filtered && cargo vendor-filterer --platform=aarch64-apple-darwin --exclude-crate-path='*#tests' deps.filtered && tokei deps.filtered`. I ran web servers with `cargo run --release --example hello-world` and measured throughput with `rewrk -c 1000 -d 10s -h http://127.0.0.1:3000/ http://127.0.0.1:3000/`.

240 points by todsacerdoti 1y ago

Less clickbaity title: Cancellations in async Rust. It's really not about "cancelling async Rust" which is what I expected, even if it didn't make much sense.

happytoexplain 1y ago
As in the pop-culture concept of cancelling? That's what you assumed the topic "cancelling async <language name>" was going to be about?? Or am I missing context?
- deleted 1y ago
  [deleted]
- hansvm 1y ago
  That's what I assumed. I know languages have to handle cancellations in async code, but Rust has had a fair amount of drama over the years, and I assumed the title was accurate and reflected that some drama was happening.
  - sunshowers 1y ago
    Appreciate the feedback here — definitely don't want the title to overshadow the work itself. Will keep this in mind for next time.
- thehamkercat 1y ago
  they probably assumed something like some_running_async_task.cancel()
- fulafel 1y ago
  Async in Rust isn't exactly universally loved, partly because async Rust is perceived to spread progressively to Rust libraries making it less optional to use. See eg https://bitbashing.io/async-rust.html https://bitbashing.io/async-rust.html, "Async Rust Is A Bad Language"
sunshowers 1y ago
As the author of the talk/blog post, I was definitely going for a bit of a moral valence in the title, in the sense that future cancellations are very hard to reason about and what I call the least Rusty part of Rust. But it admittedly is a bit clickbaity too.
- acedTrex 1y ago
  I initially skipped reading it because i thought it was another drama post about maintainers a la all the nixos stuff lately.
  - bigiain 1y ago
    To balance the universe a bit, I read it expecting a drama post - then read it right through because it was at least as interesting as the drama post I'd expected. I also discovered Oxide through this, which looks interesting, except for the complete lack of pricing I can find on their site - which puts in my head as probably in the "If you need to ask the price you can't afford it" category...
    - jgord 1y ago
      I think Oxide should be renting out time on their hardware racks, as well as selling them to big orgs. Oxide looks to be superb engineering up and down the whole stack, and if it drives more rust code into linux all the better. Now that linode has been consumed by Akamai, we need an alternative.
binary132 1y ago
if only
- zackmorris 1y ago
  +1 this. IMHO async is an anti-pattern, and probably the final straw that will prevent me from ever finishing learning Rust. Once one learns pass-by-value and copy-on-write semantics (Clojure, PHP arrays), the world starts looking like a spreadsheet instead of spaghetti code. I feel that a Rust-like language could be built with no borrow checker, simply by allocating twice the memory. Since that gets ever-less expensive, I'm just not willing to die on the hill of efficiency anymore. I predict that someday Rust will be relegated to porting scripting languages to a bare-metal runtime, but will not be recommended for new work. That said, I think that Rust would make a great teaching tool in an academic setting, as the epitome of imperative languages. Maybe something great will come of it, like Swift from Objective-C or Kotlin from Java. And having grown up on C++, I have a soft spot in my heart for solving the hard problems in the fastest way possible. Maybe a voxel game in Rust, I dunno.
  - sunshowers 1y ago
    Author here -- I'd recommend reading my blog post about how cargo-nextest uses Tokio + async Rust to handle very complex state machines: https://sunshowers.io/posts/nextest-and-tokio/ https://sunshowers.io/posts/nextest-and-tokio/
    - zackmorris 1y ago
      Ah cool, a couple of kudos to you: 1) I learned about pin in Rust to prevent values from moving in memory. 2) I learned about the html <summary> tag (the turndown arrows in your article that work with Javascript disabled) hah. I can see how dealing with stream and resource cleanup in async code could be a chore. It sounds like you were able to do that in a fairly declarative manner, which is what I always strive for as well. I think my hesitation with async is that I already went down that road early in my programming life with cooperative threads/multitasking on Mac OS 9 and earlier. There always seems to be yet another brittle edge case to deal with, so it can feel infuriating playing whack-a-mole until they're all nailed down. For example, pinning memory looks a lot like locking handles in Mac OS. Handles were pointers to pointers, so it was a bare hands way to implement a memory defragmenter before runtimes were smart enough to handle it. If apps used handles, then blocks of data could be unlocked, moved somewhere else in memory, and then re-locked. Code had to do an extra hop through each handle to get to the original pointer, which was a frequent source of bugs because one async process might be working on a block, yield, and then have another async process move the handle out from under it. The lock's state was stored in a flag in the memory manager, basically a small bit of metadata. I haven't investigated, but I suspect that Rust may be able to handle locking more efficiently, perhaps more like reference counting or the borrow checker where it can infer whether a pointer is locked without storing that flag somewhere (but I could be wrong). Apple abandoned handles when it migrated to OS 10 and Darwin inherited protected memory and better virtual memory from FreeBSD. Although now that I write this out, I'm not sure that they solved in-process fragmentation. I think they just give apps the full 32 or 64 bit address space so that effectively there is always another region available for the next allocation, and let the virtual memory subsystem consolidate 4k memory blocks into contiguous strips internally. The dereferencing of memory step became implicit rather than explicit, as well as hidden from apps, so that whole classes of bugs became unreachable. Anyway, that's why I prefer the runtime to handle more of this. I want strong guarantees that I can terminate a process and all locks inside it will get freed as well. I can pretty much rely on that even in hacky languages like PHP. My frustration with all of this is that we could/should have demanded better runtimes. We could have had realtime unixes where task switching and memory allocation were effectively free. Unfortunately the powers that be (Mac OS and Windows) had runtimes that were too entrenched with too many users relying on quirks and so they dragged their feet and never did better. Languages like Rust were forced to get very clever and go to the ends of the earth to work around that. Then when companies like Google and Facebook won the internet lottery, they pulled the ladder up behind them by unilaterally handing down decrees from on high that developers should use bare hands techniques, rather than putting real resources into reforming the fundamentals so that we wouldn't have to. What I'm trying to say is that your solution is clever and solves a common pattern in about the simplest way possible, but is not as simple as synchronous-blocking unix pipes to child processes in shell scripts. That's in no way a criticism. I have similar feelings about stuff like Docker and Kubernetes after reading about Podman. If we could magically go back and see the initial assumptions that led us down the road we're on, we might have tried different approaches. It's all of those roads not taken that haunt me, because they represent so much of my workload each day.
deleted 1y ago
[deleted]
benatkin 1y ago
It made sense to me, because I imagine a thread or coroutine as something that runs code as though it were interpreting something like psuedocode, whether it's doing that or not. So from my point of view an instance of async Rust is being cancelled - not the feature of the Rust project, but instances of code. This abstraction has served me well and facilitates stepping through code in a debugger, though I jump out of thinking it at that level when I need to think of it at a lower level.

bryanlarsen 1y ago

Timely! Was grumbling about this today as I added a "this function is cancel safe" to a new function's doc comment. I really hope we get async drop soon.

0x1ceb00da 1y ago
I'm curious. Can you talk a little about that function?
- bryanlarsen 1y ago
  Most common scenario in article: select!. I split out a "wait for X to be ready" from "X" so that the former could be on the left side of a select ARM, and the rest on the right side.

Panzerschrek 1y ago

One should always keep in mind that await is always a potential return point. So, using await between two actions which always should be performed together should be avoided.

Spivak 1y ago
That… seems bad? Like I guess it is what it is and you just have to deal with it but what if your "critical section" has two await calls? The code can be paused between them but it's such that it must eventually resume. Say making a change in the database and emitting an audit edit for that change. Is your only option to either not do that or put a big do not cancel sign on the function docs?
- sunshowers 1y ago
  In general I think people end up gravitating towards using message passing or the actor model for this.
- diarrhea 1y ago
  A DB action and audit emission have to run transactionally anyway. So on cancellation, the transaction times out and nothing is written. Bad but safe. The problem is the same on other platforms. For example, what if writing to the DB throws an exception if you’re on Python? Your app just dies, the transaction times out. Unfortunate but safe. If it does not run transactionally you have a problem in any execution scenario.
  - sunshowers 1y ago
    So, regarding transactions, absolutely you can throw them away on cancellation. But there's an interesting wrinkle here: if you use a connection pool like most users, and you were going to do the ROLLBACK at the end of your future on error, then that ROLLBACK wouldn't run it the future is cancelled! Then future operations reusing the same connection would be stuck in transaction la-la land. (This is related to the fact that Rust doesn't have async drop — you can't run async code on drop, other than spawning a new task to do the cleanup.) This is prong 3 of my cancel correctness framework (that the cancellation violates a system property, in this case a cleanup property.) The solution here is to ensure the connection is in a pristine state before handing it out the next time it's used.

CodeBrad 1y ago

This was one of my favorite talks from RustConf this year! The distinction between cancel safety and cancel correctness is really helpful. Glad to see it converted to a blog post. Talks are great, but blogs are much easier to share and reference.

sunshowers 1y ago
Thanks! I definitely prefer reading blog posts over watching talks as well.
pornel 1y ago
"Cancel correctness" makes a lot of sense, because it puts the cancellation in some context. I don't like the "cancel safety" term. Not only it's unrelated to the Rust's concept of safety, it's also unnecessarily judgemental. Safe/unsafe implies there's a better or worse behavior, but what is desirable for cancellation to do is highly context-dependent. Futures awaiting spawned tasks are called "cancellation safe", because they won't stop the task when dropped. But that's not an inherently safe behavior – leaving tasks running after their spawner has been cancelled could be a bug: piling up work that won't be used, and even interfering with the rest of the program by keeping locks locked or ports used. OTOH a spawn handle that stops the task when dropped would be called "cancellation unsafe", despite being a very useful construct specifically for propagating cleanup to dependent tasks.

Matthias247 1y ago

Some other material that has been written by me on that topic: - Proposal from 2020 about async functions which are forced to run to completion (and thereby would use graceful cancellation if necessary). Quite old, but I still feel that no better idea has come up so far. https://github.com/Matthias247/rfcs/pull/1 https://github.com/Matthias247/rfcs/pull/1 - Proposal for unified cancellation between sync and async Rust ("A case for CancellationTokens" - https://gist.github.com/Matthias247/354941ebcc4d2270d07ff0c6bf066c64 https://gist.github.com/Matthias247/354941ebcc4d2270d07ff0c6...) - Exploration of an implementation of the above: https://github.com/Matthias247/min_cancel_token https://github.com/Matthias247/min_cancel_token

ossopite 1y ago

I think the send/recv with a timeout example is very interesting, because in a language where futures start running immediately without being polled, I think the situation is likely to be the opposite way around. send with a timeout is probably safe (you may still send if the timeout happened, which you might be sad about, but the message isn't lost), while recv with a timeout is probably unsafe, because you might read the message out of the channel but then discard it because you selected the timeout completion instead. And the fix is similar, you want to select either the timeout or 'something is available' from the channel, and if you select the latter you can peek to get the available data.

sunshowers 1y ago
Thanks, that is a great point.
lionkor 1y ago
Isn't this exactly what cancellation-safety is all about?

Animats 1y ago

In the initial example, it's not clear what the desired behavior is. If the queue is full, the basic options are drop something, block and wait, or panic. Timing out on a block is usually deadlock detection. He writes "It turns out that this code is often incorrect, because not all messages make their way to the channel." Well, yes. You're out of resources. Now what? What's he trying to do? Get a clean program shutdown? That's moderately difficult in threaded programs, and async has problems, too. The use case here is unclear. The real use cases involve when you're sending messages back and forth to a remote site, and the remote site goes away. Now you need to dispose of the state on your end.

AceJohnny2 1y ago
> He writes They go by they/she https://sunshowers.io/about/ https://sunshowers.io/about/
- SV_BubbleTime 1y ago
  [flagged]
sunshowers 1y ago
Ideally, what you would like to do is buffer up the message until there's space in the channel. I cover this later in the talk under "What can be done".
- Animats 1y ago
  The double-loop thing effectively creates a blocking operation. Something you can do directly. Why all the complexity?
  - sunshowers 1y ago
    Agreed that in the narrow case of a timeout it doesn't buy you much (and things like network sockets often let you do timeouts in synchronous code). But often you do want the power to do selects and more complex state machines. I wrote a blog post a couple years ago talking about why a project I'm the author of, cargo-nextest, switched from sync Rust to async. https://sunshowers.io/posts/nextest-and-tokio/ https://sunshowers.io/posts/nextest-and-tokio/ To this day I'm not aware of a better way to express what's become a set of increasingly complex state machines (the most recent improvement being to make the state machines responsive to user input). Nextest's runner loop is structured mostly like a GUI event loop, but without explicit state machines. It's quite nice being able to write code that's this complex in a bug-free manner.

arifalkner 1y ago

Great talk! One thing that would have been nice to call out for n00bs like myself is how in SOP Futures can't cancelled. I knew that .await took ownership of the future so that drop() could not be called on it, so given how futures are lazy, it wasn't clear to me how to cancel a future after .await had been called. I later researched how select! and Abortable() did this, but could be nice to include a callout in the beginning of your talk if you ever do it again. Otherwise, nice work!

sunshowers 1y ago
Thanks! What does SOP mean in this context?

alembic_fumes 1y ago

I'm not understanding what the supposed problem with these futures getting cancelled is. Since futures are not tasks, as the post itself acknowledges, does it not logically follow that one should not expect futures to complete if the future is not driven to completion, for one reason or another? What else could even be expected to happen? The examples presented for "cancel unsafe" futures seem to me like the root of the problem is some sort of misalignment of expectations to the reality: Example 1: one future cancelled on error in the other let res = tokio::try_join!( do_stuff_async(), more_async_work(), ); Example 2: data not written out on cancellation let buffer: &[u8] = /* ... */; writer.write_all(buffer)?; Both of these cases are claimed to not be cancel-safe, because the work gets interrupted and so not driven to completion. But again, what else is supposed to happen? If you want the work to finish regardless of the async context being cancelled, then don't put it in the same async context but spawn a task instead. I feel like I must be missing something obvious that keeps me from understanding the author's issue here. I thought work getting dropped on cancellation is exactly how futures are supposed to work. What's the nuance that I'm missing?

sunshowers 1y ago
You're absolutely right! The problem is that this has introduced many bugs in our experience at Oxide. If you've already fully internalized the idea that futures are passive and can be cancelled at any await point, the talk is just a bunch of details.
- alembic_fumes 1y ago
  I see. Do you suppose that the origin of these bugs is more about the difficulty of reasoning about the execution of deep async stacks, or does it come down to the developers holding an incorrect mental model of the Rust futures in their minds? I am asking because I've noticed that many developers with previous experience from "task-based" languages (specifically the JS/TS world) tend to grasp the basics of Rust async quickly enough, but then run into expectation-misalignment related problems similar to the examples that you used in your post. That in turn has made want to understand whether it is the Rust futures that are themselves difficult or strange, or whether it's a case of the Rust futures appearing simple and familiar, even though they are completely different in very subtle ways. I suppose that it's a combination of both.

foota 1y ago

https://sunshowers.io/posts/cancelling-async-rust/#the-pain-of-tokio-mutexes https://sunshowers.io/posts/cancelling-async-rust/#the-pain-... was the most interesting part of this for me, as I can totally see making mistakes like this.

schmichael 1y ago
I'm a Go developer and this was still useful for me! Obviously Rust devs are more accustomed to more assistance from their tools than Go devs, but just about every gotcha listed is something that can happen in Go with goroutines, channels, select, and other shared concurrency primitives.
- jcgrillo 1y ago
  !m schmichael

nijaar 1y ago

for a sec i thought DEI was going too far is the title like that on purpose?

nofriend 1y ago

doesn't rust have raii?

sunshowers 1y ago
It does, but you can only run synchronous code on drop. This is what "async drop" is supposed to handle — things like issuing ROLLBACK statements to the database on cancellation. It also wouldn't help when you have no valid state to restore to, as in the mutex example in the post.
- 0x1ceb00da 1y ago
  tokio-postgres handles this by just dispatching the "ROLLBACK" command in impl Drop and ignoring the response. https://github.com/rust-postgres/rust-postgres/blob/a7a49a90734c05664ed56356b61aa937cc2a6e6b/tokio-postgres/src/client.rs#L606 https://github.com/rust-postgres/rust-postgres/blob/a7a49a90... Is this not enough? What could go wrong? If the network connection dies or the task is cancelled, I'm assuming the database server cleans up the connection state and does a rollback automatically. And adding async Drop will probably add a whole new set of footguns.
  - _davide_ 1y ago
    > What could go wrong? LoL, an insane amount of things. TCP connections are an illusion of safely, for the purpose of database commits use UDP packets as a model instead, it'll be much closer to reality.
    - 0x1ceb00da 1y ago
      > an insane amount of things List a couple > TCP connections are an illusion of safely Why?
deleted 1y ago
[deleted]

tison 1y ago

Rust's Future is somehow like move semantics in C++, where you may leave a Future in an invalid state after it finishes. Besides, Rust adopts a stackless coroutine design, so you need to maintain the state in your struct if you would like to implement a poll-based async structure manually. These are all common traps. And now cancellations in async Rust are a new complement to state management in async Rust (Futures). When I'm developing the mea (Make Easy Async) [1] library, I document the cancel safety attribute when it's non-trivial. Additionally, I recall [2] an instance where a thoughtless async cancellation can disrupt the IO stack. [1] https://github.com/fast/mea https://github.com/fast/mea [2] https://www.reddit.com/r/rust/comments/1gfi5r1/comment/luidor3/ https://www.reddit.com/r/rust/comments/1gfi5r1/comment/luido...

beanjuiceII 1y ago

i am honestly glad i don't write rust anymore

dxxvi 1y ago

My Rust knowledge is too low to understand. However, thank you very much for the article. I hope AI will learn it and explain to me in a few years :-)

mleonhard 1y ago

I think that async in Rust has a significant devex/velocity cost. Unfortunately, nearly all of the effort in Rust libraries has gone into async code, so the async libraries have outpaced the threaded libraries. There was only one threaded web server, https://lib.rs/crates/rouille https://lib.rs/crates/rouille . It has 1.1M lines of code (including deps). Its hello-world example reaches only 26Krps on my machine (Apple M4 Pro). It also has a bug that makes it problematic to use in production: https://github.com/tiny-http/tiny-http/issues/221 https://github.com/tiny-http/tiny-http/issues/221 . I wrote https://lib.rs/crates/servlin https://lib.rs/crates/servlin threaded web server. It uses async internally. It has 221K lines of code. Its hello-world example reaches 102Krps on my machine. https://lib.rs/crates/ehttpd https://lib.rs/crates/ehttpd is another one but it has no tests and it seems abandoned. It does an impressive 113Krps without async, using only 8K lines of code. For comparison, the popular Axum async web server has 4.3M lines of code and its hello-world example reaches 190Krps on my machine. The popular threaded Postgres client uses Tokio internally and has 1M lines of code: http://lib.rs/postgres http://lib.rs/postgres . Recently a threaded Postgres client was released. It has 500K lines of code: https://lib.rs/crates/postgres_sync https://lib.rs/crates/postgres_sync . There was no ergonomic way to signal cancellation to threads, so I wrote one: https://crates.io/crates/permit https://crates.io/crates/permit . Rust's threaded libraries are starting to catch up to the async libraries! --- I measured lines of code with `rm -rf deps.filtered && cargo vendor-filterer --platform=aarch64-apple-darwin --exclude-crate-path='*#tests' deps.filtered && tokei deps.filtered`. I ran web servers with `cargo run --release --example hello-world` and measured throughput with `rewrk -c 1000 -d 10s -h http://127.0.0.1:3000/ http://127.0.0.1:3000/`.