5 ms·
Async/await with one thread is simple and well-understood. That's the Javascript model. Threads let you get all those CPUs working on the problem, and Rust help
by Animats 3y ago
Async/await with one thread is simple and well-understood. That's the Javascript model. Threads let you get all those CPUs working on the problem, and Rust helps you manage the locking. Plus, you can have threads at different priorities, which may be necessary if you're compute-bound.
Multi-threaded async/await gets ugly. If you have serious compute-bound sections, the model tends to break down, because you're effectively blocking a thread that you share with others.
Compute-bound multi-threaded does not work as well in Rust as it should. Problems include:
- Futex congestion collapse. This tends to be a problem with some storage allocators. Many threads are hitting the same locks. In particular, growing a buffer can get very expensive in allocators where the recopying takes place with the entire storage allocator locked.
I've mentioned before that Wine's library allocator, in a .DLL that's emulating a Microsoft library, is badly prone to this problem. Performance drops by two orders of magnitude with all the CPU time going into spinlocks. Microsoft's own implementation does not have this problem.
- Starvation of unfair mutexes. Both the standard Mutex and crossbeam-channel channels are unfair. If you have multiple threads locking a resource, doing something, unlocking the resource, and repeating that cycle, one thread will win repeatedly and the others will get locked out.[1]
If you need fair mutexes, there's "parking-lot". But you don't get the poisoning safety on thread panic that the standard mutexes give you.
If you're not I/O bound, this gets much more complicated.
[1] https://users.rust-lang.org/t/mutex-starvation/89080 https://users.rust-lang.org/t/mutex-starvation/89080
- exfalso 3y agoYes, 100%. I've mostly only dealt with IO-bound computations, but the contention issues arise there as well. What's the point of having a million coroutines when the IO throughput is bounded again? How will coroutines save me when I immediately exhaust my size 10 DB connection pool? It won't, it just makes debugging and working around the issues harder and difficult to reason about.
- AstralStorm 3y agoThe debugging issue is bigger than what it seems. Use of async/await model in particular ends up with random hanged micro tasks in some random place in code that are very hard to trace back to the cause because they're dispersed potentially anywhere. Concurrency is also rather undefined, as are priorities most of the time. This can be partly fixed by labelling, which adds more complexity, but at least is explicit. Then the programmer needs to know what to label... Which they won't do, and Rust has no training wheels to help with concurrency. Threads, well you have well defined ingress and egress. Priorities are handled by the OS and to some degree fairness is usually ensured.
- ninkendo 3y ago> What's the point of having a million coroutines when the IO throughput is bounded again? Because if you did the same thing with a million threads, you’d have all those same problems and more, because (a) threads take up more RAM, (b) threads are more expensive to spawn and tear down, and (c) threads have more registers to save/restore when context switching than coroutines. The real answer is that you shouldn’t have a million coroutines, and you shouldn’t have a million threads. Doing such a thing is only useful if you have other useful work you can do while you wait on I/O. This is true for web servers (which want to serve other requests, so… maybe one coroutine per active request) and UI’s (which should have a dedicated thread for the UI event loop), but for other applications there’s shockingly little need to actually do async programming in the first place. Parallelism is a well-understood problem and can be used in key places (ie. doing a lot of math that you can easily split into multiple cores), but concurrency is IMO a net negative for most applications that don’t look like a web server.
- _flux 3y agoJemalloc can use separate arenas for different threads which I imagine mostly solves the futex congestion issue. Perhaps it introduces new ones?
- gpderetta 3y agoIIRC glibc default malloc doesn't use per-thread arenas as they would waste too much memory on programs spawning tens of thousands of threads, and glibc can't really make too many workload assumptions. Instead I think it uses a fixed pool of arenas and tries to minimize contention. These days on Linux, with restartable sequences, you can have true per-cpu arenas with zero contention. Not sure which allocator use them though.
- JonChesterfield 3y agohttps://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree/doc/man/rseq.2 https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree... Thank you, I didn't know about this one. An allocator that seems to use it at https://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html
- jeffbee 3y agoNot only does tcmalloc use rseq, the feature was contributed to Linux by the tcmalloc authors, for this purpose, among other purposes.
- celrod 3y ago> As pressure from thread collisions increases, additional arenas are created via mmap to relieve the pressure. The number of arenas is capped at eight times the number of CPUs in the system (unless the user specifies otherwise, see mallopt), which means a heavily threaded application will still see some contention, but the trade-off is that there will be less fragmentation. https://sourceware.org/glibc/wiki/MallocInternals https://sourceware.org/glibc/wiki/MallocInternals So glibc's malloc will use up to 8x #CPUs arenas. If you have 10_000 threads, there is likely to be contention.
- jmspring 3y agoYou focus on rust rather than generalizing... If you are IO bound, consider threads. This is almost the same as async / await. What was missing above, and the problem with how most compute education is these days, if you are compute bound you need to think about processes. If you were dealing with python concurrent.futures, you would need to consider processpooexecutor vs. threadpoolexecutor. Threadpoolexecutor gives you the same as the above. With multiprocessor executor, you will have multiple processes executing independently but you have to copy a memory space. Which people don't consider. In python DS work - multiprocessor workloads need to determine memory space considerations. It's kinda f'd up how JS doesn't have engineers think about their workloads.
- baq 3y agoBackend JS just spins up another container and/or lambda and if it's too slow and requires multiple CPUs in a single deployment, oh well, too bad.
- zelphirkalt 3y agoThat is of course a huge overhead, compared to how other languages solve the problem.
- fennecfoxy 3y agoBackend JS does whatever the hecky you _want_ it to do. The Cluster module has been around for a long time: https://nodejs.org/api/cluster.html https://nodejs.org/api/cluster.html Tbf a lot of the time you're running it in a container and you allocate 1 vcpu there, only downside is maybe a little extra memory overhead. And for most lambdas I think they're suited to being single threaded (imo).
- danbruc 3y ago[...] if you are compute bound you need to think about processes. How would that help? Running several processes instead of several threads will not speed anything up [1] and might actually slow you down because of additional inter-process communication overhead. [1] Unless we are talking about running processes across multiple machines to make use of additional processors.
- neonsunset 3y agoI keep having to repeat it on this unfortunate website: it's an implementation detail, a multi-threaded executor of async/await can cope with starvation perfectly well as demonstrated by .NET's implementation that can shrug off some really badly written code that interleaves blocking calls with asynchronous. https://news.ycombinator.com/item?id=39530435 https://news.ycombinator.com/item?id=39530435 https://news.ycombinator.com/item?id=39786142 https://news.ycombinator.com/item?id=39786142 https://news.ycombinator.com/item?id=39721626 https://news.ycombinator.com/item?id=39721626
- AstralStorm 3y agoSaid badly with code will still execute with poor performance and the mix can be actively hard to spot.
- neonsunset 3y agoYou would be surprised. It ultimately regresses to "thread per request but with extra steps". I remember truly atrocious codebases that were spamming task.Result everywhere and yet there were performing tolerably even back on .NET Framework 4.6.1. The performance has (and often literally) improved ten-fold since then, with threadpool being rewritten, its hill-climbing algorithm receiving further tuning and it getting proactive blocked workers detection that can inject threads immediately without going through hill-climbing.
- maccard 3y agoBadly written threaded code will have the same problem, unfortunately
- temporarely 3y agoJust morning bathroom musings based on your posts (yep /g) and this got me thinking maybe the robust solution (once and for all for all languages) may require a rethink at the hardware level. The CPU bound issue comes down to systemic interrupt/resume I think; if this can be done fairly for n wip thread-of-execution with efficient queued context swaps (say maybe a cpu with n wip contexts) then the problem becomes a resource allocation issue. Your thoughts?
- binary132 3y agoWhat you said sounded in my head more like you’re describing a cooperatively scheduled OS rather than a novel hardware architecture.
- temporarely 3y ago(This has been a very low priority background thread in my head this morning so cut me some slack on hand waving.) Historically, the H/W folks addressed (pi) memory related architectural changes, such as when multicore came around and we got level caches. Imagine if we had to deal at software level with memory coherence in different cores [down to the fundamental level of invalidating Lx bytes]. There would be NUMA like libraries and various hacks to make it happen. Arguably you could say "all that is in principle OS responsibility even memory coherence across cores" and we're done. Or you would agree that "thank God the H/W people took care of this" and ask can they do the same for processing? The CPU model afaik hasn't changed that much in terms of granularity of execution steps whereas the H/W people could realize that d'oh an execution granularity in conjunction with hot context switching mechanism, could really help the poor unwashed coders in efficiently executing multiple competing sequences of code (which is all they know about at H/W level). If your CPU's architecture specs n+/-e clock ticks per context iteration, then you compile for that and you design languages around that. CPU bound now becomes heavy CPU usage but is not a disaster for any other process sharing the machine with you. It becomes a matter of provisioning instead of programming ad-hoc provisioning ..
- binary132 3y ago
- klysm 3y agoI find this has worked well for me when I can easily state what thread pool work gets executed on.
- mmis1000 3y agoI think its kotlin's model. The language don't have a default co-routine executor set, you need to provide your own or spawn one with the std. It can be thread-pooled, single thread, or custom one but there isn't a default one set. If you use single threaded executor, then race condition won't happen. If you choose pooled, then you obviously should realize there can be a race condition. It's all about choice you made.
- HarHarVeryFunny 3y agoI'm not sure what you mean by "multi-threaded async/wait"... Isn't the article considering async/await as an alternative to threads (i.e. coroutines vs threads)? I'm a C++ programmer, and still using C++17 at work, so no coroutines, but don't futures provide a similar API? Useful for writing async code in serialized fashion that may be easier (vs threads) to think about and debug. Of course there are still all the potential pitfalls that you enumerate, so it's no magic bullet for sure, but still a useful style of programming on occasion.
- lights0123 3y agoasync/await tasks can be run in parallel on multiple threads, usually no more threads than there are hardware threads. This allows using the full capabilities of the machine, not just one core's worth. In a server environment with languages that support async/await but don't have the ability to execute on multiple cores like Node.js and Python, this is usually done by spawning many duplicate processes and distributing incoming connections round-robin between them.
- zarzavat 3y agoThey mean async/await running over multiple OS threads compared to over one OS thread. You can also have threads running on one OS thread (Python) or running on multiple OS threads (everything else). Every language’s concurrency model is determined by both a concurrency interface (callbacks, promises, async await, threads, etc), and an implementation (single-threaded, multiple OS threads, multiple OS processes).
- miohtama 3y agoAlso there is the extra software development and maintenance cost due to coloured functions that async/await causes https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... Unless you are doing high scalability software, async might not be worth of the trade offs.
- nine_k 3y agoIf you expect to color your functions async by default, it's really easy to turn a sync functuion into a near-zero-cost async, a Future that has already been resolved at construction time by calling the sync function. This way, JS / TS becomes pretty comfortable. Except for stacktraces, of course.
- yencabulator 3y agoExcept when your sync function calls a function that makes the async runtime deadlock. No, there's no compiler error or lint for that. Good luck!
- ngrilly 3y agoUseful stacktraces is one of the reasons why I use Go instead of JS/TS on the server-side.
- RReverser 3y agoJS has proper async stacktraces as well, or do you mean something else?
- ngrilly 3y agoYou're right. But it was not the case years ago when my team evaluated Go and Node.js. I should have clarified.
- laggyluke 3y ago> Useful stacktraces is one of the reasons why I use Go I'm not sure if this is sarcasm or not.
- MuffinFlavored 3y ago> Async/await with one thread is simple and well-understood. That's the Javascript model. Bit of nuance (I'm not an authority of this and I don't know the current up-to-date answer across 'all' of the Javascript runtimes these days): Isn't it technically "the developer is exposed the concept of just one thread without having to worry about order of execution (other than callbacks/that sort of pre-emption)" but under the hood it can actually be using as many threads as it wants (and often does)? It's just "abstracted" away from the user.
- recursive 3y agoI don't think so. Two threads updating the same variable would be observable in code. There's no way around it.
- MuffinFlavored 3y agoI'm referring to something like this: https://stackoverflow.com/questions/7018093/is-nodejs-really-single-threaded https://stackoverflow.com/questions/7018093/is-nodejs-really... It's like a pedantic technical behind the scenes point I think, just trying to learn "what's true"
- recursive 3y agoLooks like the spec refers to the thing-that-has-the-thread as an "agent". https://tc39.es/ecma262/#sec-agents https://tc39.es/ecma262/#sec-agents I don't know the details about how any implementation of a javascript execution environment allows for the creation of new agents.
- kaba0 3y agoI mean, yeah, if you go deep enough your OS may decide to schedule your browser thread to a different core as well. I don’t think it has any relevance here - semantically, it is executed on a single thread, which is very different from multi-threading.
- 3y ago
- sporkland 3y agoI don't know how the challenges with cooperative (no-preemptive) multitasking keep needing to get rediscovered. Even golang, which I consider a very responsibly designed language, went with cooperative at first until they were forced to switch to preemptive. Not saying cooperative multitasking doesn't have its place, just that it's gotta have a warning sticker or even better disallow certain types of code from executing statically. Also great time to plug a related post, What color is your function: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...