22 ms·
Comparison of Rust async and Linux thread context switch time and memory use
- continuations 6y agoHow does Rust async compare to Goroutine, Erlang threads, Javascript async, Java async in performance and memory usage? Is there any benchmarks for that?
- sam0x17 6y agoAnd also Crystal fibers
- monadic3 6y agoWhich rust async runtime are you referring to?
- continuations 6y agoSay Tokio, the async runtime used in OP's article.
- redrobein 6y agoRust doesn't have a standard async runtime, so the question would be "how does tokio compare to goroutine, etc..." since that's the most popular one. Looking at the techempower benchmarks, the projects using tokio generally outperform Go, Java, so I'm guessing it's on par or better. Hypothetically, you could port goroutines exact behavior to rust and use that as your wanted to too.
- scottlamb 6y agoI'd think mostly similar. Goroutines are "stackful" coroutines, though, so their memory use will be higher. They have an interesting stack copying model, so I'm not sure if they require as many pages as POSIX threads do. (Having a "denser" memory space and no guard page requirement would mean you could use huge pages and thus have much less TLB pressure.)
- uniquefine 6y agoThere were benchmarks and a discussion on this on reddit recently comparing goroutines to tokio. If I recall correctly tokio was slower than goroutines but if you set the right settings it could be almost as fast. https://www.reddit.com/r/rust/comments/lg0a7b/benchmarking_tokio_tasks_and_goroutines/ https://www.reddit.com/r/rust/comments/lg0a7b/benchmarking_t...
- jph 6y agoQuickest summary: Rust async is >3x faster and lighter than Linux threads. This is a great accomplishment for Rust.
- mhh__ 6y agoIf you elide a bounds check from a function but still spend a billion cycles in a loop, you've made your code run ever so slightly faster but gained nothing in the big picture.
- shmerl 6y agoIt sound to me like comparing apples and oranges though. Parallelism (threads) and concurrency (aysnc in Rust) are not the same thing and can be actually used in combination.
- jashmatthews 6y agoYou can happily use pthreads for concurrency up to ~10k before reaching for async.
- andrewchambers 6y agoI read it as most people are over investing in async rust.
- rudedogg 6y agoWhat makes you think that? Looking at the summary it looks like async is better in every way.
- scottlamb 6y agoBetter or equal in all the ways measured. But some things aren't measured, maybe because they're obvious to the author or because they're harder to quantify. * Rust's async ecosystem [1] adds a lot of complexity over simple threaded code. * Rust's async ecosystem doesn't interoperate as easily with C libraries written in a simple threaded way. (And it's debatable which interoperates more easily with C libraries written with a different event loop.) * async tasks can't be preempted, so concurrency will fall off a cliff if they run on O(cpus) threads and involve long-running computations or accidental blocking. I think it's reasonable to ask if these numbers are enough better to justify all that, particularly given the disappointing "this advantage goes away if the context switch is due to I/O readiness". And to go back and argue pro-async for a moment, io_uring might eliminate that disappointing caveat. Then again, on the pro-thread side, there's Google's interesting fibers model that might solve some of these performance issues. [2] Also, "~17µs for a new kernel thread" is the wrong number, since you can avoid that cost with a simple thread pool. Personally I think some things are better written as async, but it's a mistake to impose it on everything. For example, if you're writing a web app in Rust, I think you're usually better off writing threaded request handlers and having a mechanism for them to interact with the async hyper code. The hyper code is better off as async because an Internet-facing server might have an enormous number of connections in keepalive state. [1] or maybe I should say ecosystems, plural, given the current tokio vs async-std divide. [2] https://lwn.net/Articles/826860/ https://lwn.net/Articles/826860/
- Animats 6y agoThat's a huge help. I only need about 20 threads in Rust, some of which are compute-bound. So involving "async" is totally the wrong tool for the job. Goodbye, Tokio.
- monadic3 6y agoSorry, what does tokio have to do with "async"? The default implementation uses posix threads, no?
- deleted 6y ago[deleted]
- Matthias247 6y agoIt's a runtime for running lightweight tasks (`Future`s, async functions) on top of it. What is not async about it? And of course it still needs posix threads. The executor needs to run somewhere, and the only somewhere that an OS offers is a thread.
- monadic3 6y agoSure, but I didn't think anything about async functions implied running tasks. Isn't it just syntactic sugar over futures? You certainly don't need to use the tokio runtime in order to use async functions. So, it's not clear why you'd abandon the async syntax just because you're compute bound.
- monadic3 6y agoI'll admit it's a bit disconcerting not getting a reply but I'll get over it.
- masklinn 6y ago> So involving "async" is totally the wrong tool for the job. Sadly with so many things having gone async-first (or only) it’s become difficult not to end up with an async runtime anyway, or not be forced to use an async system. I wanted to build a small web-based tool for local, didn’t really find anything which was not async.
- klodolph 6y agoI am not surprised that the cost of context switching due to I/O readiness can often be roughly equal between async tasks and kernel threads. Normal blocking I/O can be surprisingly efficient because of various factors, such as a reduced need for system calls. Think about it this way—if you have a user-space thread which wakes up due to I/O readiness, then this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time. In both scenarios, you’re calling a blocking system call and waking up the thread later. Of course, there are scenarios when epoll_wait() returns multiple events, which reduces the number of context switches. But the general result is that it’s not always easy to beat blocking I/O and kernel threads.
- khuey 6y agoio_uring will help a lot here.
- klodolph 6y agoWell, it can, but not always. Remember that if you’re waiting for an event to arrive, that generally involves a syscall, the thread being put to sleep, and then the thread being woken up. Any time you’re doing that, think, “Could I just replace this polling system with a call to read()?” What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Whether you are often on the fast path for io_uring will depend on the particulars of your application and its I/O patterns.
- quotemstr 6y agoio_uring is specifically designed so that zero system calls are necessary while the system is busy. Userspace and the kernel both update ring buffers, and ring buffers can be checked and drained without entering the kernel at all.
- amluto 6y agoI’m curious how this looks on ARM64. (Sorry, Aarch64.). x86 context switches are overcomplicated and inherently slow.
- lstamour 6y agoWell, here's what it looked like on my MacBook Pro with M1/16GB: M1-MBP async-brigade % time cargo run --release 500 tasks, 10000 iterations: mean 761.403µs per iteration, stddev 8.929µs (1.522µs per task per iter) cargo run --release 3.21s user 4.60s system 99% cpu 7.818 total M1-MBP thread-brigade % time cargo run --release 500 tasks, 10000 iterations: mean 787.149µs per iteration, stddev 67.289µs (1.574µs per task per iter) cargo run --release 0.94s user 7.19s system 100% cpu 8.081 total I ran it a few times and the numbers came up rather similar each time: async-brigade finished in 760.273µs-764.928µs while thread-brigade took 784.510µs-796.323µs. As macOS doesn't have taskset, I can't easily set affinity. I tried to use the workaround documented elsewhere to use Xcode's Instruments to reduce the number of CPU cores but it would always re-enable itself at 8 cores, so that didn't work.
- continuations 6y ago> A context switch takes around 0.2µs between async tasks, versus 1.7µs between kernel threads. But this advantage goes away if the context switch is due to I/O readiness: both converge to 1.7µs. This is a big surprise. If you look at the Techempower web benchmark [1], the performance of actix-web is about 20x higher than that of Rocket. The common explanation is that actix-web is async and hence much faster than Rocket which relies on kernel context switching. But if Rust async and kernel thread has the same switch time as shown by this benchmark, then why is actix-web so much faster than Rocket? [1] https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune&l=yyku7z-sf https://www.techempower.com/benchmarks/#section=data-r20&hw=...
- jashmatthews 6y agoRocket's low performance could well be because the threadpool and DB connection pool is undersized: https://github.com/TechEmpower/FrameworkBenchmarks/blob/b8917c6891b22ea27eb4bb591c419c721afe918d/frameworks/Rust/rocket/src/main.rs#L200 https://github.com/TechEmpower/FrameworkBenchmarks/blob/b891...
- rurdaneta 6y agofor more information on cybersecurity and how to keep your company safe, log on to demyo inc, you will find a wealth of valuable information. https://demyo.com/ https://demyo.com/
- KirillPanov 6y ago> But this advantage goes away if the context switch is due to I/O readiness This is not at all a fair comparison unless you're using io_uring.
- josephg 6y agoGood point - It’d be very interesting to see how io_uring changes those numbers if anyone has some time to make a fork / PR!
- jorangreef 6y agoNot Rust, but you may be interested in colorless async io_uring using Zig: https://news.ycombinator.com/item?id=26111847 https://news.ycombinator.com/item?id=26111847 Also (very rough) benchmarks (take with a pinch of salt) comparing various styles of fs and network IO (blocking, epoll, io_uring) for C and Zig: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...
- jsd1982 6y agoPlease don't mix nanoseconds and microseconds. This is just confusing to read. Stick to nanoseconds for everything.
- jstrong 6y agoah, a fellow traveler - godspeed. what a sane, reasonable world we could have if nanosecond timestamps ruled supreme.
- makapuf 6y agoI wouldn't be so extreme, providers of '0' keys would flourish
- mst 6y agoI occasionally wonder whether it would be easier to skim '1k us' than '1ms' though, providing everything was denominated in us. Maybe I'll try it in a blog post one day and see what percentage of the comments consist of hurled fruit.
- electricshampo1 6y agoa related analysis showing epoll vs thread per request https://www.slideshare.net/brendangregg/rxnetty-vs-tomcat-performance-results https://www.slideshare.net/brendangregg/rxnetty-vs-tomcat-pe...
- otabdeveloper4 6y agoA meaningless comparison. Linux, being a preemptively multitasking OS, switches thread contexts regardless of what you're running. So the Rust async context switch is on top of the regular Linux context switch, not instead.
- ithkuil 6y agoWhen you're in sub microsecond time scales, preemption events are relatively rare.
- otabdeveloper4 6y agoI don't see your point. "Linux thread context switch time" is a meaningless metric, since Linux will switch thread context regardless of what you choose to run on your computer. Any "async" switches are additional overhead; you don't get to not have kernel preemption just because your Rust thread is now switching contexts "asyncly". There are benefits to having an additional user-mode scheduling mechanism inside your kernel thread, but saving CPU cycles isn't one of them.
- ithkuil 6y ago> switches thread contexts regardless of what you're running my point was that thread context switches caused by preemption happen at an entirely different time scale than the rate of context switches caused by syscalls (if the system is doing any meaningful level of IO)
- bitcharmer 6y agoI work in ultra-low latency space and agree with GP. This comparison makes no sense as OS-level context switch is completely different from a task-switch within the same native thread. The Rust ones from that benchmark are essentially fibers, not threads. You will see similar performance for switching fibers if well implemented in Java, C++ or other natively compiled language. This has nothing to do with Rust.
- richardwhiuk 6y agoThe problem with threads is you need to correctly size your thread pool. That's hugely difficult if you have unknown lengths of blocking IO.
- tictac-toe 6y agoYou can use a dynamically sized thread pool. E.g. remove a thread with a certain probability once it's idle for more than X seconds.
- jstrong 6y agowhat kind of problem were you trying to solve that you found sizing a thread pool to be difficult? generally when I've worked on high performance server code I've been coding with a target machine in mind, so it's more a matter of mapping the thread pool size to the resources available on that machine. but I'm interested to hear about circumstances where it wouldn't be easy.
- imtringued 6y agoYou have a threadpool with X threads. You dispatch Y tasks. X of them run for 5 minutes. That means the remaining Y-X tasks are delayed by 5 minutes despite low CPU utilization.
- richardwhiuk 6y agoIf you have downstream nodes which may have large amounts of latency in some scenarios, then you may need a huge thread pool. If you add a huge thread pool, and then those downstreams don't have a large latency, then you end up accepting a huge amount of work and then are CPU starved. So in order to correctly size your thread pool, you need to understand all your downstream latency, and adapt to it. Compared to an async runtime, which just handles this scenario, it's very painful. Even if you get this roughly right, the scheduler is very unhappy when you have lots of threads - it tends to make incorrect scheduling decisions.
- lamontcg 6y agohttps://en.wikipedia.org/wiki/Green_threads https://en.wikipedia.org/wiki/Green_threads https://en.wikipedia.org/wiki/Thread_(computing)#Threading_models https://en.wikipedia.org/wiki/Thread_(computing)#Threading_m...
- jgilias 6y agoThe discussion should feature prominently somewhere on top that the comparison is between the Tokio async runtime and Linux threads. Reason being, people not familiar with Rust may assume that the discussion applies to Rust async in general, when it doesn't necessarily. Sure, in practice Tokio is pretty close to being the de-facto async runtime in Rust. But it's not the only one, as Rust's async language constructs allow for different runtime implementations that may be optimized for different use cases.
- deleted 6y ago[deleted]
- kohlerm 6y agoTo me it looks like the main advantage of async is memory usage, which is kind of expected because of the overhead of a thread. But if you do not need lots of thread it doesn't look like there is a huge benefit going async. Or do I miss something here?
- mikeschmatz 6y agoIt was my conclusion as well. I only found async useful in situations when a service has to deal with a large number of incoming requests, e.g. web server
- erk__ 6y agoIt could be very interesting to see similar comparisons with other operating systems like FreeBSD with kqueue or DragonflyBSD with Light Weight Kernel Threads.
- secondcoming 6y agoPerformance profiling on a laptop is largely pointless. There's too much stuff trying to conserve power by limiting performance.
- the8472 6y agoI have done benchmarking on a linux laptop. Once you disable turbo-boost you get quite consistent results for CPU-bound tasks at least.
- bitcharmer 6y agoYou have to do so much more to be able to reliably measure events on the scale of nanos. You need to lock C-states, disable P-state driver, isolate CPUs, get rid of RCUs, affinitize your tasks, enable low-tick mode, skew hr ticks, make sure you use TSC clocksource, set the cpu governor, get rid of vmstat, set correct idle driver, disable audits, and watchdogs and much, much more.
- the8472 6y agoIf you want to instrument only a handful events, yes. But for microbenchmarks which you can run for many iterations to get min/max/stdev (such as the benchmarks in the article) it's much easier. Disabling turbo often is sufficient to lower the variance far enough that old and new code are clearly distinguishable.
- bitcharmer 6y agoIt has nothing to do with instrumenting and everything to do with platform noise.
- secondcoming 6y ago+1 This guy profiles
- bzbarsky 6y agoAre your users going to be running your application on laptops? Will they have the same "conserve power by limiting performance" going on? If so, that is _exactly_ the environment you want to do performance work in, generally speaking.
- lxpz 6y ago> People often see that there's some theoretical benefit of async and then they accept far less ergonomic coding styles and the additional bug classes that only happen on async due to accidental blocking etc... despite the fact that when you consider a real-world deployed application, those "benefits" become indistinguishable from noise. However, due to the additional bug classes and worse ergonomics, there is now less energy for actually optimizing the business logic, which is where all of the cycles and resource use are anyway, so in-practice async implementations tend to be buggier and slower. I disagree with this. I feel like using Async programming is actually much more powerfull and expressive than theaded programming, especially with Rust combinators on streams of futures (for examples: futures_unordered), which allow to trivially express complex concurrency patterns (such as: wait for the first two requests to return something and discard the third request's response, and btw also cancel that request). Async programming also allows for structured programming, where each task is an owned resource of a parent tasks, which means that lifetimes of tasks can be controlled and runaway threads can't exist (if one is avoiding tokio::spawn). I've been developping [Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage https://git.deuxfleurs.fr/Deuxfleurs/garage) for some time now (a simple distributed object store that implements a subset of S3, not ready for production!), and I've been in awe about how easy it was to write these complex patterns using async Rust.
- kzrdude 6y agoThat's interesting to hear - but how much of an investment is it to climb the that mountain (or hill) that makes you comfortable with working with the async model?
- TheNorthman 6y agoNot OP but: I found it surprisingly manageable. I think the disconnect for many people is they think they'll understand it simply by using it [0]. For me, investing a short time reading some of the design articles/documents really helped it click. [0]: Which is fair, I wouldn't be surprised if this was the best way for some.
- pas 6y agoRust's async model takes very little time to grasp. It's very explicit. Nothing runs in the background (contrast that with NodeJS). You have strong static types to help you to know when you got a Future, you can decide where/when to await it. It's programming with threads, where you have a thread pool and pipes to put tasks onto that and a helper function/macro. Await does this under the hood of course.
- tarruda 6y agoIn the past couple of years I started to use a heavier functional style for my code. What I noticed is that any syntactical benefits of async/await has a lesser impact when most of your application logic lives in pure functions, since you greatly reduce the amount of code in async functions. When I started using async/await in JS 4-5 years ago I thought: "How could we have lived without this for so long?". These days I don't care much about it.
- pron 6y agoThe throughput increase in I/O scenarios with many tasks is due to the number of supported concurrent processes and Little's law; it has little to do with context switching time, which has a negligible impact on the throughput in these use-cases: https://inside.java/2020/08/07/loom-performance/ https://inside.java/2020/08/07/loom-performance/ Low context switch latency only matters when the number of tasks is very small (their data all fits in the cache), and the workload is entirely computational. Otherwise, even the fastest implementation is ~60 ns, which is the cost of a cache-miss, and the compiler can't optimise things into a simple goto because the dispatch goes through a scheduler that has a megamorphic call-site. So memory is much more important for I/O use-case throughput, and while it is true that the kernel doesn't commit the full stack memory on thread creation, it's misleading to think that you get good memory usage. For one, once the memory is committed, it's never uncommitted (although it can be paged out). For another, the granularity is that of a page, i.e. at least 4K, which can often be much higher than what a task requires. > It is hard to pin down exactly how the alleged advantages would arise. For I/O use-cases the answer is here: > the async version uses about 1/20th as much memory as the threaded version. This could translate to 20x throughput -- due to Little's law -- although usually less because there are other limits, like network saturation.
- PixelOfDeath 6y agoI hat very good experience with using buffer variables to copy-"prefetch" unpredictable costly fetches. E.g. from cache lines that get touched by several cores for communication. And I only actual use them after one iteration of whatever I do. So the core could fetch the memory content without actually having to stall because I do not use it until later. I'm not sure how realistic that is inside a kernel thread scheduler, but it sure is useful in user space for task based libraries.
- sriku 6y agoWondering how these micro benchmarks would fare on FreeBSD. (Will post here if I end up doing it over the weekend)
- gwbas1c 6y agoTLDR: Async code will have much lower CPU utilization compared to threaded code. An async version of a program might run just as fast as a threaded one, but it will overall use less system resources. The threaded version will be easier to write. You can also have lower RAM overhead per thread if you choose a smaller stack space. Many programs will run fine with a smaller stack space, BTW. ---- Years ago I had to build a load simulator in C#. The CTO looked at me and told me that it had to simulate 100,000 clients; thus it had to be async. He arranged for me to have a very powerful computer to run the load simulator. I originally wrote non-blocking code. The non-blocking code had very low load at 100,000 clients, but I hit a problem with a difficult-to-understand edge case. Because we only had a weekend to do load testing, I refactored the load simulator to be threaded. It only took me 20 minutes or so. The problem with the difficult-to-understand edge case went away, but CPU usage went up dramatically. We had to tune the .Net framework to use a much smaller stack space. In the end, I was able to have 100,000 threads to run the load simulator. CPU usage and RAM usage were very high, but the load simulator ran fine. If I had more time, I would have taken the time to understand the edge case and continue to use non-blocking code. Then the program would have used much less system resources, but ran just as fast.
- scottlamb 6y ago> You can also have lower RAM overhead per thread if you choose a smaller stack space. No, this is about as low as it gets. As the author explained, "the kernel only allocates physical memory to a stack as the thread touches its pages, so the initial memory consumption of a thread in user space is actually only around 8kiB." The smallest possible page size (on x86-64) is 4 KiB, and you can't share pages between thread stacks, [1] so you can't go below 4 KiB of physical memory usage per thread. I'm not exactly sure how the author got to 8 KiB; maybe they meant "for each userspace thread" rather than "memory used in userspace" and are counting kernel memory too. I'm pretty sure the kernel uses at least 4 KiB per userspace thread (for a stack of its own, among other overhead). Green threads won't take you below 4 KiB either, for the same reason. [1] Without some custom ABI that guards against stack overflow in a different way. Golang has a custom ABI (I'm not sure exactly if this is why), and interoperability with C suffers, so this isn't an approach I'd love for Rust.
- eptcyka 6y agoI think the async benchmark could be faster still when pinned to a single core if a single threaded runtime was used, and possibly if a single-thread channel implementations were used, but then it's becoming a bit academic. Really, what async gives you is a programming style that's very similar to using blocking sockets, but allows one to achieve select() like performance when doing I/O. That, and it allows one to not have to have a special thread for timers or, even worse, a thread per timer, as that's hidden away by the async runtime implementation and just works.
- juancampa 6y agoThis is very illuminating. Thank you. I'd love to see a third "column": processes, i.e. fork().
- kentonv 6y agoI feel like this analysis is missing some more nuanced points about stack memory. Yes, pages will only be allocated for a thread's stack when the thread actually uses them. However, the thread does not release said memory afterwards. The memory can only be reused by the same thread. If a thread ever once does something that temporarily allocates a bunch of stack space, then it forever consumes that space going forwards even when no longer needing it. If you have 10,000 threads and each one of them happens to, at some point in its lifetime, use 1MB of stack space and frees it, then you are now using 10GB of RAM on mostly-unused pages. Now you might say "what on Earth would ever use 1MB of stack???", but the problem is, in normal programs with few threads, there's no problem with a function temporarily using a ton of stack, and so random things feel free to do so. Maybe some library call you make likes to allocate a temporary buffer on the stack and you don't even know it. There's also normally no problem with doing some deep recursion every now and then, so it happens. Often, stack allocation is data-dependent (e.g. recursive descend parsing). So if you try to strictly limit your stack space then you risk running into random stack overflows or maybe even security issues. And if you do find a limit that works, it's still probably much larger than the average usage, so you're still wasting a bunch of memory. IIRC, Go uses segmented stacks to avoid this problem, but C/C++/Rust do not. (I think Rust tried to at one point, but later gave up on that because of the complexity?) In contrast, async tasks only hold onto the memory they are actually using to store live data at any particular moment. If an async task invokes some deeply-nested function and uses a bunch of stack space, it doesn't really matter, because all the tasks are running on the same thread, so the next task to call that function uses the same pages rather than allocate new ones. (There's actually a similar issue regarding heap space. Memory allocators that perform reasonably with multiple threads typically maintain per-thread freelists, so if you have lots and lots of threads, you end up with a bunch of free'd memory stuck in freelists. Though, some allocators, like the new tcmalloc, are starting to use per-core freelists instead, which may avoid this problem.)