9 ms·

Thread-per-core

271 points by ingve 3y ago

insanitybit 3y ago
> Some Rust users are unhappy with this decision, so unhappy that they use language I would characterize as melodramatic: Seriously. I've said this before, the way people talk about these problems is so dramatic. I've written 10s of thousands of LOC in Rust and you'd think, from these blog posts, that I must be miserable. I am not. I am quite pleased with it. It's so funny to me that Thread Per Core is seen as the holy land. Like it's just objectively better. It is not. TPC is quite tricky, indeed. Scylla had a great post recently talking about how they were suffering performance penalties due to accidentally sharing some memory - this is the sort of thing you have to be super careful about when leaning into TPC. "Hot partitions" is another one. I'm going to refrain from commenting more (this article looks excellent and I want to read the citations) but I'm very very happy to see this post. I think far to little credit is given to the effort put into getting a novel language like Rust to support this so well.
- Waterluvian 3y ago
  This kind of attitude exists everywhere, but good gravy does it exist strongly in Software Development. I feel like every software developer could benefit from a summer tarring rooftops to mature their idea of what "killing the joy" of doing work feels like.
  - insanitybit 3y ago
    A big problem with software dev is the lack of rigor, imo. > It’s always off-putting to me that claims written this way can be taken seriously as a technical criticism, but our industry is rather unserious. From the article, this resonates so strongly with me. Instead of people reading papers, talking seriously, it's so hand wavy and weak.
    - Waterluvian 3y ago
      My general feeling is that I think people lean on how they feel about these problems because building an experiment and quantifying the problem to make a solid case for it actually being a problem worthy of resources and remediation is... tedious and unfun? Not to say these people don't exist. But I think they exist in lesser numbers. Even I'm guilty of this. I know it... so I do it... but it's very unfun and tedious and... well I'm paid to do it so I can't complain.
      - insanitybit 3y ago
        I'm going to avoid derailing into a "tech has these problems", respectfully, as I think the core of the post is really worthwhile and I could talk about tangentially related things for too long and be too distracting. I think we are likely, overall, in agreement.
      - aeonik 3y ago
        I think it's more then tedium and fun that blocks the rigor. Though, those two properties definitely hold a fair number of people back. I'll use myself as a research subject: I really like the scientific method, and don't mind a long grind. But the main barriers I run into with software fall into at least two major categories: 1. Compatible datasets or software: it takes a LOT of manual effort to collate good data sets, and because I am a software engineer, I want to automate these things, then I end up in tarpits and rabbit holes, building bridges rather than testing hypotheses. 2. Abstraction fatigue, There is so much vocabulary that we use, and we switch layers so quickly and in different contexts that I find it very time consuming and opaque to harmonize all the concepts. Real life pressures don't always afford the ability to really understand an entire stack, which I posit is necessary for a certain level of Rigor. For example, right now I'm trying to Grok Clojure Transducers, and some folks are saying they are the same things as Monads. Are transducers a better pattern? Are they the same? Different? How much do they overlap? I have feelings relating to dynamic types vs static types. I have feelings about wrapping implementations. I have feelings about the JVM vs the Haskell compiler vs the V8 JavaScript engine. Also, I get feelings related to polymorphism, metaprogramming, dependent types, homoiconicty. But nothing fully baked. To call my feelings even 10% baked would be charitable. I can't currently describe all the relationships, or why my brain is currently zeroing in here in full fidelity. Only parts of it, and I remain in a state perplexity for now. The formality of Category Theory and Abstract algebra sounds like it would be really nice here, but it's going to take me years to get to to speed in those domains, and I'm not even sure they have the theorums to compare these two patterns!
  - taeric 3y ago
    I was going to say the same thing. My gut is it exists as strongly in other places, but that we don't have direct exposure to their social zones to see it. And it isn't like rust is particularly hit by it, either. You'd think people writing PHP or Java hate life, if you only went by what you are likely to see in our social sites.
    - safety1st 3y ago
      The PHP developers I speak to these days seem pretty happy to be using PHP. The ones who fly into a rage over it seem to be people who don't actually use it!
      - cesarb 3y ago
        > The ones who fly into a rage over it seem to be people who don't actually use it! Well, that does make some sense: if something bothers you so much that you'd "fly into a rage over it", you would avoid using it as much as possible (and PHP is not like JavaScript which has no real alternative).
        taeric 3y ago
        The annoying thing is more that a ton of the ones that fly into a rage are also ones that have never used it. There is a ton of dog-piling in what people complain about.
        cesarb 3y ago
        > The annoying thing is more that a ton of the ones that fly into a rage are also ones that have never used it. Even that does make some sense: if, on a first look, you see something you deeply dislike, you probably will also avoid using it in the first place. As a personal example, I never learned Go because, when I first looked at it (IIRC, it was when I had to use it to run a Heartbleed detector), I deeply disliked the way it required developers to organize their source code (a single per-language directory mixing all projects together, instead of the per-project directories I have always used); I understand this might have changed later, but in the meantime I invested my time in learning another programming language, and so far haven't found a need to look at Go again.
  - mcronce 3y ago
    Your analogy really hits. I don't mind doing construction in general (I quite like doing it once in a while), but roofing absolutely sucks no matter how you cut it
- api 3y ago
  I'm pleased with it too. Seriously. When I learned it it took me a bit to get lifetimes and borrows and stuff but once I did I got it and I rarely have to "fight" it. Running in an IDE hooked to rust-analyzer helps a lot too since you get pretty instant feedback if you get something wrong most of the time. I have this suspicion that at least a few of the people who hate a lot of languages with more complex type systems or large stdlibs (e.g. Go, Java, C#) are trying to edit them in a plain vanilla text editor without these features. This would require you to memorize way too much shit. Why? Modern machines have gigabytes of RAM. Run a language server.
- VirusNewbie 3y ago
  I love this stuff, and while I consider myself informed, i'm not an expert. However, I've bantered a bit with Scylla's CTO and others about how I would regularly run into issues with Scylla because they didn't use TPC, and I'd see hot keys that would suffer performance because they only let one core manager a subset of keys... So, while I can't make an authoritative statement, I think TPC is better overall...
- efficax 3y ago
  i've written about 30k lines of async rust this year alone and haven't found the horror of Send + Sync + 'static to impact me much at all. You just have to think about things for a second sometimes.
  - insanitybit 3y ago
    TBH most of the time you don't have to think at all. Compiler says "you need those annotations" and you copy/paste them and then you move on. It's really incredible to me that people are acting like this is some extremely onerous process, like writing a dozen characters causes some sort of incredible pain.
- deleted 3y ago
  [deleted]
cmrdporcupine 3y ago
There is no right answer on this front, and it's all about different use cases. It comes down to I/O-bound vs CPU bound workloads, and to how negatively things like cache evictions and lock contention might affect you. If your thing is an HTTP server talking to an external database with some mild business logic inbetween, and hosted on a shared virtual server, then, yeah, work-stealing and re-using threads at least intuitively makes sense (tho you should always benchmark.) If you're building a database or similar type of system where high concurrency under load with lots of context switches is going to lead to cache evictions and contention all over the place -- you're going to have a bad time. Thread per core makes immense sense. An async framework itself may not make any sense at all. But there is no right, dogmatic answer on what "is better." Profile your application. I've said it before, but I feel like the focus of Rust as a whole is being distorted by a massive influx of web-service type development. I remain unconvinced that Rust is the right language for that kind of work, but it seems to do ok for them, so whatever. But the kind of emphasis it puts on popular discussion of the language, and the kind of crates that get pushed to the forefront right now on the whole reflect this bias. (Which is also the bias of most people employed as SWEs on this forum, etc.)
- anonymousDan 3y ago
  Yeah I agree about the webdev influx. It would be a shame if Rusts utility for systems programming is ruined as a result.
- ndriscoll 3y ago
  Funny, I was thinking a web app is ideal for thread per core. The application itself generally has very little state outside of a request (other than for the socket listener and database connections, which can be segmented per thread), and what state it has is probably mostly static across requests (so caches don't invalidate often). It should be easy to deal with ownership of shared state because there isn't any.
  - cmrdporcupine 3y ago
    It all depends on if you think the operating systems scheduler is better than your async framework's. There's a long tradition of not wanting to "waste" threads by blocking on I/O.
DarkNova6 3y ago
Good writeup I recommend to read more than just the headline. My favorite line: > I have a hard time believing that someone who’s biggest complaint is adding “Send” bounds to some generics is engaged in that kind of engineering. Edit: I fully agree with the comment of "duped". I was not aware of the greater context of this discussion. As such, I might have quoted this sentence prematurely.
Pxtl 3y ago
I've never touched Rust, but I can see the complaint -- if I have to write my code in a special way so that its state can be marshalled across threads to redistribute load in a way that I do not need and will make the actual end-to-end latency of a single request actually slower in cases where I've got oodles of CPU head-room, I would find that infuriating. I could see this approach making sense in a platform where transferrable state was the default and is rarely broken, but it doesn't sound like that's the case in Rust? edit: I'm curious, what's the ergonomics like for this? Is it just "oh, your code won't compile if you don't add this magic incantation to say it has `Send`?" Or is it "your code will fail in intermittent, hard-to-debug ways as state gets mangled during work-steals if you don't do this right"?
- cmrdporcupine 3y ago
  On the whole the compiler detects when things are 'Send' & 'Sync'. If you write your program thread-safe, you won't have issues. And that's the crux of the matter: people are griping on the whole because maybe Tokio async is hard when it puts Send&Sync demands all over the placer, but the reality is that writing safe concurrent code of any kind is hard. It's not intuitive, and the problem is that async makes them feel like it's happening automagically and just "taking care of it" -- but it's really not. You need to know what you're doing, and the compiler is just helping you here. Yes, you can hide that by going thread-per-core and that could eliminate the need for Send in some circumstances (not all). But it might come back to bite you, architecturally, in the long run.
- creata 3y ago
  > I'm curious, what's the ergonomics like for this? Your code won't compile unless everything is Send/Sync as appropriate, and (I might be wrong, but) the lazy path to achieving this is usually wrapping things that might be shared in Arcs and/or Mutexes.
- ko27 3y ago
  I think the argument against thread-per-core as the default can be made simply: - if you are CPU bound, work stealing will be better for most cases - if you are IO bound, thread-per-core might work better, but again, you have enough CPU headroom that the performance doesn't really matter IMO, work stealing is a better default to encode into language API
samsquire 3y ago
Thank you for this blog post. I like the thread per core design because kernel context switches are expensive. But userspace context switching scheduling (such as an unbuffered golang channel) for a 64 bits of data at a time makes me uncomfortable too unless it represents a large amount of work backed by a pointer to work data. I think I like to multiplex sockets over threads and multiplex IO over threads so that you can do CPU work while IO is going on and you can process multiple clients per thread. You cannot scale memory mutation by adding threads. You want ideally one thread to own the data and be safe to mutate it uncontendedly. When you send data to another thread, don't refer to that data again. Transfer ownership to that thread. If you can divide your request into phases of expansion (map) and contraction (synchronization) you can do intrarequest parallisation. I've been looking into runqueues of go and tokio where you have a local runqueue without a mutex and a global runqueue with a mutex. I've been trying to think how IO threads that run liburing or epoll can wake up a Coroutine or an async task on a worker thread without the mutex. The worker thread is looking for tasks to resume that are unblocked and needs to be notified when there is IO finished. I think you can have a Coroutine that is always runnable to read from a lock free ringbuffer. You can have that Coroutine that checks for finished IO's ringbuffer yield if its contended by writes by the IO thread that is trying to make work available to the worker threads.
- ori_b 3y ago
  > I like the thread per core design because kernel context switches are expensive You can do a millions to tens of millions kernel entries/exits per second per core. They're on the same order of magnitude as a single digit number of full cache misses. So, expensive, but a lot less expensive than many people assume. Edit: for some (outdated, 2018) measurements, https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ https://eli.thegreenplace.net/2018/measuring-context-switchi...
  - cmrdporcupine 3y ago
    And all our assumptions get blown up with every new hardware generation, too. Hardware engineers are building systems for the applications of today. And the kernel developers are optimizing as well. It's all a moving target. Dogma has no place here.
pornel 3y ago
Everyone who has worked with sharding at scale knows the pain when there's this one terrible key (your biggest customer, hottest key, largest blob) that just ruins load-balancing of the shards.
- scottlamb 3y ago
  100% true, but in fairness work stealing doesn't completely solve that either. You often end up locking the shard anyway because lockless data structures are limited and hard. And if you're doing something like a key/value store, you're probably sharding to decide which server to direct traffic to, and so you have the same problem at that layer. (There are things that can help, of course, e.g. <https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/ https://aws.amazon.com/blogs/architecture/shuffle-sharding-m...>.)
- IshKebab 3y ago
  Yeah maybe, but how many people are running these high performance async servers on 2 core machines?
asd4 3y ago
"What they mean by IO bound is actually that their system doesn’t use enough work to saturate a single core when written in Rust: if that’s the case, of course write a single threaded system." Many of the applications I write are like this, a daemon sitting in the background reacting to events. Making them single threaded means I can get rid of all the Arc and Mutex overhead (which is mostly syntactic at that point, but makes debugging and maintenance easier). Being able to do this is one of the things I love about Rust: only pay for what you need. The article that this one is responding to calls out tokio and other async libraries for making it harder to get back to a simple single threaded architecture. Sure there is some hyperbole but I generally agree with the criticism. Making everything more complex by default because its better for high throughput applications seems to be opposite of Rust's ideals.
- Kinrany 3y ago
  I was going to comment on the same quote. The problem is that one may still want concurrency even when a single thread on a single CPU is enough.
- amluto 3y ago
  I’ve written services like this, and I would never have called them IO bound. They’re not throughput-bound at all. They mostly sit idle, then they do work and try to get it done quickly to minimize use of system resources. Unless they sometimes get huge bursts of work and something else cares quite a lot about latency during those bursts, using more than one thread adds complexity and overhead for no gain.
  - __alexs 3y ago
    In an era of 10Gb NICs in every server very few things are really IO bound.
    - kosolam 3y ago
      10gb nics and their respective connections are quite expensive. Not many servers have these at all.
MrBuddyCasino 3y ago
> This appears in Rust’s APIs as futures needing to be Send, which can be difficult for people with a poor view of their system’s state to figure out the best way to ensure. This is why work-stealing is said to be “harder.” Is it just me or does this come across as rather arrogant? The problem of 'static lifetimes and send/sync constraints resonates among developers, and my impression is not that those were morons.
- Guvante 3y ago
  Harder is in quotes because it isn't necessarily harder. If you would need to do it anyway it isn't harder. Less "people are worked up over doing a little work" more "async makes you solve problems earlier you were going to solve anyway". Similar vibe to the borrow checker. Sometimes it is overly restrictive othertimes you didn't actually consider the corner cases when you assumed everything would be fine.
- withoutboats3 3y ago
  I'm just referring back to my earlier point: people say not doing work stealing would be both easier and faster. My claim is that its one or the other, because to get share-nothing to be faster you need to architect your code in a way that is not easier than making a shared-state architecture thread-safe. There is a parallel sentence with "slower" in the next paragraph. I don't think people who struggle to get parallel & concurrent Rust to compile are morons, though I don't like when they act like the APIs we built for them are ruining their lives.
  - MrBuddyCasino 3y ago
    Fair enough, the technical point is argued well. Thanks for the clarification.
- eminence32 3y ago
  From my own personal experience, I definitely struggle sometimes to understand if my state is Send or not. So the line you quoted from the article resonates with me.
duped 3y ago
Personally I feel like this post misses the forest for the trees. The debate isn't about thread-per-core work stealing executors, it's whether async/await is a good abstraction for it in Rust. And the more async code I write the more I feel that it's leaky and hard to program against. The alternative concurrency model people want is structured concurrency via stackful coroutines and channels on top of a work stealing executor. Until someone does the work to demo that and compare it to async/await with futures I don't think there's any productive discussion to be had. People who don't like async are going to avoid it and people who don't care about making sure everything and its mother is Send + Sync + 'static are going to keep on doing it.
- withoutboats3 3y ago
  In fact there are multiple things in the world that people disagree about. This post addresses a different debate than the one you wanted me to write about, that's all.
  - duped 3y ago
    The post you quoted and whose author you insulted at the end was talking about this issue.
    - withoutboats3 3y ago
      No it wasn't. Anyone can follow the link and see that it was about preferring non-work-stealing executors, none of the things you complained about.
- zozbot234 3y ago
  AIUI, nothing's stopping you from using channels over Rust's existing async support. Stackful coroutines are kinda pointless, since at that point you might as well be using separate threads.
  - packetlost 3y ago
    Greenthreads have their benefits though, especially for massive concurrency.
newpavlov 3y ago
I believe that having the Send bound as a requirement to allow migration of tasks between executor threads is a clear deficiency of the Rust async system by itself, together with the fundamental issues around async Drop, which prevent implementation of scoped APIs. Similarly to threads it should be sufficient to have Send bound on functions like spawning and sending data through channels. The share-nothing approach is usually used as nothing more than a workaround to hide this deficiency.. Selectively pinning tasks to a thread/core has advantages and can be really useful in some circumstances, but it's a finer discussion, which has little to do with async users dissatisfaction related to Send.
scottlamb 3y ago
Regarding the quote: > The Original Sin of Rust async programming is making it multi-threaded by default. If premature optimization is the root of all evil, this is the mother of all premature optimizations, and it curses all your code with the unholy Send + 'static, or worse yet Send + Sync + 'static, which just kills all the joy of actually writing Rust. Agree about the melodramatic tone. I also don't think removing the Send + Sync really makes that big a difference. It's the 'static that bothers me the most, and that's not there because of work stealing. I want scoped concurrency. Something like <https://github.com/tokio-rs/tokio/issues/2596 https://github.com/tokio-rs/tokio/issues/2596>. Another thing I really hate about Rust async right now is the poor instrumentation. I'm having a production problem at work right now in which some tasks just get stuck. I wish I could do the equivalent of `gdb; thread apply all bt`. Looking forward to <https://github.com/tokio-rs/tokio/issues/5638 https://github.com/tokio-rs/tokio/issues/5638> landing at least. It exists right now but is experimental and in my experience sometimes panics. I'm actually writing a PR today to at least use the experimental version on SIGTERM to see what's going on, on the theory that if it crashes oh well, we're shutting down anyway. Neither of these complaints would be addressed by taking away work stealing. In fact, I could keep doing down my list, and taking away work stealing wouldn't really help with much of anything.
- teraflop 3y ago
  > I'm having a production problem at work right now in which some tasks just get stuck. I wish I could do the equivalent of `gdb; thread apply all bt` For all the hate that Java gets, this is something that has Just Worked(tm) for like 25 years, and it's enormously helpful for troubleshooting. You don't even need a debugger; you can just send the JVM a SIGQUIT, and it'll dump a stack trace of every thread to stderr (including which locks each thread is holding and/or waiting on) and keep running. I miss this feature in every other language I've worked with. You can even use it for ad-hoc profiling in production: just take a bunch of snapshots, and use grep/sed/sort/uniq to look for hotspots.
  - scottlamb 3y ago
    Go does this well, too. iirc the std library has a package for serving <http://blah/debug/pprof/goroutines http://blah/debug/pprof/goroutines> so you don't even need to ssh in to the server in question. There are libraries for doing the same for any language that just uses kernel threads. It's when you throw in async that you really need to reinvent this kind of observability, and Rust isn't there yet unfortunately.
kosolam 3y ago
Very nice write up that demonstrates the difference between theory and practice.
jacknews 3y ago
Paints the 'rust community' as quite restrictive, and not a place I want to be, IMHO.
weinzierl 3y ago
"This means that any state that is used across a yield point in that task needs to be thread-safe. This appears in Rust’s APIs as futures needing to be Send [..]" Does that mean that I have to unnecessarily use locking (e.g. ARC and Mutex) even with flavor = "current_thread" or am I misunderstanding this?
- necubi 3y ago
  No locking (which provides mutable `Sync` in rust terms) is generally required, but things must be Send (i.e., allowed to move between threads, but not shared). In my experience is mostly an issue when dealing with references, because `&T` is only Send if `T` is sync. However, `&mut T` is Send if `T` is Send. So if you can avoid aliasing your references, you're also ok.
  - weinzierl 3y ago
    Thanks! So does that mean that if I write my code with default tokio and ARC where required and then when I add flavor = "current_thread" I can replace my ARC with Rc?
    - the_white_oak 3y ago
      Rc is not Send. But you can use tokio's LocalSet to run non-Send futures. https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html
hjl22 3y ago
I haven't touched async Rust, but have used normal Rust a little bit. My understanding is that 'static is program lifetime, meaning either static compile-time data like constant strings, or it is memory leaks. How does this apply to async in practice? Do you need to leak things that you want to make async?
- scottlamb 3y ago
  It means spawned futures can't contain references that aren't 'static. But they can own memory. String for example is 'static. Likewise Vec<T>, Box<T>, Arc<T>, Rc<T>, etc. where T: 'static.
  - hjl22 3y ago
    Thanks for the explanation. I understand it now.
- pitaj 3y ago
  Most heap-allocated values like `String` are also static. Essentially it means that anything the value references is not on the stack.
  - hjl22 3y ago
    Thanks for the explanation. I understand it now.
- Georgelemental 3y ago
  `T: 'static` means that values of the type `T` own all their data (more specifically, their lifetime is not constrained by the lifetime of anything they reference). `&'static T` is a reference to something that lives as long as the program (static data or leaked memory).
  - hjl22 3y ago
    Thanks for the explanation. I understand it now.
lordnacho 3y ago
First of all, does anyone know if there's a flag that just turns off work-stealing? Seems like you could then decide for yourself how you want Tokio in particular to work. Second, the send + sync thing seems onerous, but from my POV you are encouraged to only pass little messages on channels anyway. I find it easier to just not share anything. If two things need the same structure, they can both subscribe to the ring that gets the messages, and each construct the object for themselves. I find if you have a bunch of Arc<RwLock<>> something is wrong. YMMV.
- c-cube 3y ago
  You can configure tokio with feature flags in cargo. I'm particular you can pick a single threaded scheduler.
dpc_01234 3y ago
This is missing the forest for the trees. There is no one always-right way to get best performance for every program. You can argue about it all you want. Thread-per-core benefits/downside is a typical "it depends" discussion. The problem is that using `async` in the first place is a premature optimization. 99% of Rust programs are not redis, linkerd and alikes. They are some CLI tools or Web apps that could have been written in Python or Ruby, and they would still be fast enough. So why as a community we abandoned blocking IO Rust and everything is async now, and developers are just doing `#[tokio::main]` by default on everything?
- vbezhenar 3y ago
  One reason might be: if you're fine with thread-per-core performance, you probably wouldn't want to use Rust in the first place, as there are language with better programming experience trading it for speed. Like python. And if you want to use Rust, probably you need extra performance and you can adopt less convenient style (because you already adapted less convenient language) to get better performance.
  - dpc_01234 3y ago
    Wat. No.
sakras 3y ago
> The problem with work-stealing is that it means a task can run on one thread, pause, and then be started again on another thread: that’s what it means for the work to be stolen. This means that any state that is used across a yield point in that task needs to be thread-safe. Do any existing "thread-per-core" systems actually provide yield points as a thing you can do? Most of my experience is with OpenMP (using both BSP and task parallelism) and a little bit of TBB (also task parallelism). If you want to yield in these systems, you break it up into two tasks. > if state is moved from one thread to another, this introduces synchronization costs This isn't obvious to me. If you're moving state, then there is no synchronization because only one thread is touching the state at any given point. Unless we're talking about synchronization due to starting a new task?
catern 3y ago
An important thing omitted in this post, which makes work-stealing less attractive, is that one core being idle can actually improve performance of other cores. Today's CPUs basically have a fixed energy budget, and if one core is idle that means more of that budget can go to other cores. In other words, core utilization is less relevant today - what you care about is energy utilization (which is shared across cores). Of course, there's a point at which this stops being relevant - if you have multiple sockets for example, this won't apply. But work stealing across multiple sockets is so expensive anyway that you would never want to do it. You might as well work-steal across machines at that point - something which is indeed useful sometimes, but usually niche.
- jeffbee 3y ago
  You can practice energy-aware scheduling at higher levels, too. If you have to send an RPC and you can choose between multiple peers, choose the one with the coldest CPU temperature.
- CyberDildonics 3y ago
  If a CPU is being cooled enough to not throttle, it is much more time and energy efficient to use all the cores you can rather than have another core run at a slightly higher frequency. Higher frequencies have diminishing returns and exponential heat loss. You might as well work-steal across machines at that point Shared memory is extremely fast, it crushes using local loopback networking, let alone using actual networking.
jandrewrogers 3y ago
The original problem thread-per-core was invented to solve ~15 years ago was scalability and efficiency of compute on commodity many-core servers. Contrary to what some have suggested, thread-per-core was expressly about optimizing for CPU bound workloads. It turned out to be excellent for high-throughput I/O bound workloads later, albeit requiring more sophisticated I/O handling. When I read articles like this, it looks like speed-running the many software design mistakes that were made when thread-per-core architectures were introduced. To be fair, thread-per-core computer science is poorly documented, having originated primarily in HPC. This article focuses on a vexing problem of thread-per-core architectures: balancing work across cores. There are four elementary models for this, push/pull of data/load. Work-stealing is essentially the "load pull" model. This only has low overhead if you almost never need to use it e.g. if the work is naturally balanced in a way that few real-world problems actually are. For workloads where dynamic load skew across cores is common, which is the more interesting problem, work-stealing becomes a performance bottleneck due to coordination overhead. Nonetheless, it is easy to understand so people still use work-stealing when the workload is amenable to it, it just doesn’t generalize well. There are a few rare types of workloads (not mentioned in the article) where it is probably the best choice. The model with the most gravity these days seems to be "data push", which is less intuitive but requires much less thread coordination. The "data push" model has its own caveats — there are workloads for which it is poor — but it generalizes well to most common workloads. Thread-per-core architectures are here to stay -- they cannot be beat for scalability and efficiency. However, I have observed that most software engineers have limited intuition for what a modern and idiomatic thread-per-core design looks like, made worse by the fact that there are relatively few articles or papers that go deep on this subject.
- withoutboats3 3y ago
  Thanks for this response, it's really interesting. > For workloads where dynamic load skew across cores is common, which is the more interesting problem, work-stealing becomes a performance bottleneck due to coordination overhead. Nonetheless, it is easy to understand so people still use work-stealing when the workload is amenable to it, it just doesn’t generalize well This sounds right to me. The reason Rust async frameworks use work-stealing is mainly that it's easy to enable at the framework level and will improve performance in a lot of applications, especially those that are not ideally architected. Based on your comment and your self-description on your profile, these are not the kinds of applications you work on. I would be interested in receiving links to more literature.
cryptonector 3y ago
> It’s always off-putting to me that claims written this way can be taken seriously as a technical criticism, but our industry is rather unserious. It's not the industry. It's people. > What these people advocate instead is an alternative architecture that they call “thread-per-core.” They promise that this architecture will be simultaneously more performant and easier to implement. In my view, the truth is that it may be one or the other, but not both. Thread-per-core is hands-down better for performance. That's because you can't rely on storing application state smeared on a thread/co-routine stack, so you have to instead make all your state explicit, and in the process you'll effectively compress it by no longer smearing it on a very large data structure called the call stack. Smaller state == less cache thrashing. The problem with thread-per-core is that you really need async/await or coding in continuation passing style in order to make it work. CPS is mostly not an option in most cases, so async/await it has to be. Developers should understand the concepts of thread-per-core and thread-per-client and pick the best one for their case. IMO thread-per-core is always better because if your application explodes in popularity then you'll really care to make it more efficient, but rewrites are always hard or infeasible, so if you get it right from the beginning, then you win. What I mean by "pick the best one" then has to do with the developer's skills and productivity and time constraints. Sometimes "best" is about what you can manage, not what's "best" in an objective vacuum that ignores real-world constraints.
- IshKebab 3y ago
  > It's not the industry. It's people. Yes, and this is just naive to think that other industries are somehow bastions of serious debate.
mikhailfranco 3y ago
Erlang/Elixir/BEAM emphasizes share nothing, allows (encourages) a bezillion user-space processes, then executes with a thread-per-core (by default). The actual number of schedulers (real threads) is configurable as a command line option, but it's rare, approaching unheard-of, to override the default. Virding's First Rule of Programming ... https://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html https://rvirding.blogspot.com/2008/01/virdings-first-rule-of...
- jerf 3y ago
  If Go with its green threads is a step down from Rust in performance, Erlang is two or three steps down from Go. If you step down your performance needs, a lot of these problems melt away. Most programmers should indeed do that. There's no need to bring these problems on yourself if you don't actually need them. Personally I harbor a deep suspicion a non-trivial amount of the stress in the Rust ecosystem over async and its details is coming from people who don't actually need the performance they are sacrificing for. (Obviously, there absolutely people who do need that performance and I am 100% not talking about them.) But it's hard to tell, because they don't exactly admit that's what they're doing if you ask, or, at least, not until many years and grey hairs later. But in the meantime, some language needs to actually solve these problems (better than C++), and since Rust has volunteered for that role, that means that at the limit, the fact that other languages that chose to just take a performance hit don't seem to have these problems doesn't have very many applicable lessons for Rust, at least when it is being used at these maximum performance levels.
  - mikhailfranco 3y ago
    Agree, Erlang will never win any performance benchmarks, but that is mostly due to other aspects of the language: big integers, string handling, safer-rather-than-faster floating point, etc. [Elixir is a little better, supporting binary-first for strings, rather than charlists - Erlang is very good at pattern-matching binaries.] Share-nothing and thread-per-core are good for many reasons, including performance, but they also feed into the main philosophies for Erlang development: resilience, horizontal scalability and comprehensibility. As Joe Armstrong said: “Make it work, then make it beautiful, then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful.”
davidhyde 3y ago
> “The problem with work-stealing is that it means a task can run on one thread, pause, and then be started again on another thread”. The article explains how work-stealing is a way to solve tail latency. I understand tail latency to be caused by tasks that yield (by themselves) after an unexpectedly long amount of time. This cannot be know in advance. The article explains how work-stealing results in cache misses and extra developer constraints like Send, Sync and ‘static. What if the executor only moves a task to another thread after a timeout period has expired? This should result in no latency penalty if things are running smoothly and a constant, low, extra latency when tasks need to be stolen now and again. This would mitigate cache miss issues but alas not the developer overhead of all those multithreaded constraints.
exfalso 3y ago
TLDR summary: 1. Straw man 2. Talk about an unrelated paper 3. Conclusion: I am smart and those people are dumb On a somewhat related note: I'd say 95% of the software I've written was IO bound(and if it wasn't at the beginning then the goal was to make it IO-bound), and that remaining 5% does not benefit at all from async/coroutines(used them in Haskell, Kotlin(Quasar) and Rust). I'm very curious about what real-world CPU-bound use cases people have that can benefit from async performance-wise. If you're optimizing on that microsecond-nanosecond scale then you shouldn't even have yields/blocks on your hot paths, and synchronization will at most be done using memfences, so what are we talking about? I'd also conjecture that if you place your problem domain on the IO vs CPU bound axis, the problems where async would provide performance benefits can invariably be solved by other designs that perform even better (GPU/FPGA). Yes I'm one of those people who thinks coroutines have extremely marginal actual value, and most of that value is the "feeling of how cool this is" that people experience when they first learn about the concept. If there was a way to bin the concept altogether I would do it in a heartbeat. As-is, the entire Rust ecosystem is suffering heavily because of it.
- slaymaker1907 3y ago
  A lot of the time, the benefit can be from much more controlled scheduling without having to introduce actual threads. A lot of software doesn't want to take the overhead of creating platform threads (memory and taking additional CPU time) but does want to do scheduling internally for various CPU bound workloads. I think web pages are often like this. You don't really want to hog tons of CPU, but you also want to have guarantees like not blocking rendering when running a user provided regex since said regex could take a long time to run, but in practice that regex will be fast. Even if we want to eventually offload that to a separate thread, async gives us the option to first try evaluating it in the main thread for some number of iterations before offloading it. GPU/FPGA programming is kind of irrelevant because as expensive as developing high performance code is for CPUs, costs go up by an order of magnitude for those platforms unless there is some existing library you can utilize (mainly applicable for ML/AI). These platforms are also very expensive. It's like saying just rent a helicopter if you need to get somewhere quickly instead of asking what "quickly" actually means and optimizing accordingly.
cryptonector 3y ago
Work stealing is like process migration, and as such it has significant overhead and you really need to be picky about when to rebalance by moving loads.
Perenti 3y ago
Perhaps the title should mention this is primarily about Rust, and not thread-per-core in general, which would be interesting.

271 points by ingve 3y ago

> Some Rust users are unhappy with this decision, so unhappy that they use language I would characterize as melodramatic: Seriously. I've said this before, the way people talk about these problems is so dramatic. I've written 10s of thousands of LOC in Rust and you'd think, from these blog posts, that I must be miserable. I am not. I am quite pleased with it. It's so funny to me that Thread Per Core is seen as the holy land. Like it's just objectively better. It is not. TPC is quite tricky, indeed. Scylla had a great post recently talking about how they were suffering performance penalties due to accidentally sharing some memory - this is the sort of thing you have to be super careful about when leaning into TPC. "Hot partitions" is another one. I'm going to refrain from commenting more (this article looks excellent and I want to read the citations) but I'm very very happy to see this post. I think far to little credit is given to the effort put into getting a novel language like Rust to support this so well.

Waterluvian 3y ago
This kind of attitude exists everywhere, but good gravy does it exist strongly in Software Development. I feel like every software developer could benefit from a summer tarring rooftops to mature their idea of what "killing the joy" of doing work feels like.
- insanitybit 3y ago
  A big problem with software dev is the lack of rigor, imo. > It’s always off-putting to me that claims written this way can be taken seriously as a technical criticism, but our industry is rather unserious. From the article, this resonates so strongly with me. Instead of people reading papers, talking seriously, it's so hand wavy and weak.
  - Waterluvian 3y ago
    My general feeling is that I think people lean on how they feel about these problems because building an experiment and quantifying the problem to make a solid case for it actually being a problem worthy of resources and remediation is... tedious and unfun? Not to say these people don't exist. But I think they exist in lesser numbers. Even I'm guilty of this. I know it... so I do it... but it's very unfun and tedious and... well I'm paid to do it so I can't complain.
    - insanitybit 3y ago
      I'm going to avoid derailing into a "tech has these problems", respectfully, as I think the core of the post is really worthwhile and I could talk about tangentially related things for too long and be too distracting. I think we are likely, overall, in agreement.
    - aeonik 3y ago
      I think it's more then tedium and fun that blocks the rigor. Though, those two properties definitely hold a fair number of people back. I'll use myself as a research subject: I really like the scientific method, and don't mind a long grind. But the main barriers I run into with software fall into at least two major categories: 1. Compatible datasets or software: it takes a LOT of manual effort to collate good data sets, and because I am a software engineer, I want to automate these things, then I end up in tarpits and rabbit holes, building bridges rather than testing hypotheses. 2. Abstraction fatigue, There is so much vocabulary that we use, and we switch layers so quickly and in different contexts that I find it very time consuming and opaque to harmonize all the concepts. Real life pressures don't always afford the ability to really understand an entire stack, which I posit is necessary for a certain level of Rigor. For example, right now I'm trying to Grok Clojure Transducers, and some folks are saying they are the same things as Monads. Are transducers a better pattern? Are they the same? Different? How much do they overlap? I have feelings relating to dynamic types vs static types. I have feelings about wrapping implementations. I have feelings about the JVM vs the Haskell compiler vs the V8 JavaScript engine. Also, I get feelings related to polymorphism, metaprogramming, dependent types, homoiconicty. But nothing fully baked. To call my feelings even 10% baked would be charitable. I can't currently describe all the relationships, or why my brain is currently zeroing in here in full fidelity. Only parts of it, and I remain in a state perplexity for now. The formality of Category Theory and Abstract algebra sounds like it would be really nice here, but it's going to take me years to get to to speed in those domains, and I'm not even sure they have the theorums to compare these two patterns!
- taeric 3y ago
  I was going to say the same thing. My gut is it exists as strongly in other places, but that we don't have direct exposure to their social zones to see it. And it isn't like rust is particularly hit by it, either. You'd think people writing PHP or Java hate life, if you only went by what you are likely to see in our social sites.
  - safety1st 3y ago
    The PHP developers I speak to these days seem pretty happy to be using PHP. The ones who fly into a rage over it seem to be people who don't actually use it!
    - cesarb 3y ago
      > The ones who fly into a rage over it seem to be people who don't actually use it! Well, that does make some sense: if something bothers you so much that you'd "fly into a rage over it", you would avoid using it as much as possible (and PHP is not like JavaScript which has no real alternative).
      - taeric 3y ago
        The annoying thing is more that a ton of the ones that fly into a rage are also ones that have never used it. There is a ton of dog-piling in what people complain about.
        cesarb 3y ago
        > The annoying thing is more that a ton of the ones that fly into a rage are also ones that have never used it. Even that does make some sense: if, on a first look, you see something you deeply dislike, you probably will also avoid using it in the first place. As a personal example, I never learned Go because, when I first looked at it (IIRC, it was when I had to use it to run a Heartbleed detector), I deeply disliked the way it required developers to organize their source code (a single per-language directory mixing all projects together, instead of the per-project directories I have always used); I understand this might have changed later, but in the meantime I invested my time in learning another programming language, and so far haven't found a need to look at Go again.
- mcronce 3y ago
  Your analogy really hits. I don't mind doing construction in general (I quite like doing it once in a while), but roofing absolutely sucks no matter how you cut it
api 3y ago
I'm pleased with it too. Seriously. When I learned it it took me a bit to get lifetimes and borrows and stuff but once I did I got it and I rarely have to "fight" it. Running in an IDE hooked to rust-analyzer helps a lot too since you get pretty instant feedback if you get something wrong most of the time. I have this suspicion that at least a few of the people who hate a lot of languages with more complex type systems or large stdlibs (e.g. Go, Java, C#) are trying to edit them in a plain vanilla text editor without these features. This would require you to memorize way too much shit. Why? Modern machines have gigabytes of RAM. Run a language server.
VirusNewbie 3y ago
I love this stuff, and while I consider myself informed, i'm not an expert. However, I've bantered a bit with Scylla's CTO and others about how I would regularly run into issues with Scylla because they didn't use TPC, and I'd see hot keys that would suffer performance because they only let one core manager a subset of keys... So, while I can't make an authoritative statement, I think TPC is better overall...
efficax 3y ago
i've written about 30k lines of async rust this year alone and haven't found the horror of Send + Sync + 'static to impact me much at all. You just have to think about things for a second sometimes.
- insanitybit 3y ago
  TBH most of the time you don't have to think at all. Compiler says "you need those annotations" and you copy/paste them and then you move on. It's really incredible to me that people are acting like this is some extremely onerous process, like writing a dozen characters causes some sort of incredible pain.
deleted 3y ago
[deleted]

cmrdporcupine 3y ago

There is no right answer on this front, and it's all about different use cases. It comes down to I/O-bound vs CPU bound workloads, and to how negatively things like cache evictions and lock contention might affect you. If your thing is an HTTP server talking to an external database with some mild business logic inbetween, and hosted on a shared virtual server, then, yeah, work-stealing and re-using threads at least intuitively makes sense (tho you should always benchmark.) If you're building a database or similar type of system where high concurrency under load with lots of context switches is going to lead to cache evictions and contention all over the place -- you're going to have a bad time. Thread per core makes immense sense. An async framework itself may not make any sense at all. But there is no right, dogmatic answer on what "is better." Profile your application. I've said it before, but I feel like the focus of Rust as a whole is being distorted by a massive influx of web-service type development. I remain unconvinced that Rust is the right language for that kind of work, but it seems to do ok for them, so whatever. But the kind of emphasis it puts on popular discussion of the language, and the kind of crates that get pushed to the forefront right now on the whole reflect this bias. (Which is also the bias of most people employed as SWEs on this forum, etc.)

anonymousDan 3y ago
Yeah I agree about the webdev influx. It would be a shame if Rusts utility for systems programming is ruined as a result.
ndriscoll 3y ago
Funny, I was thinking a web app is ideal for thread per core. The application itself generally has very little state outside of a request (other than for the socket listener and database connections, which can be segmented per thread), and what state it has is probably mostly static across requests (so caches don't invalidate often). It should be easy to deal with ownership of shared state because there isn't any.
- cmrdporcupine 3y ago
  It all depends on if you think the operating systems scheduler is better than your async framework's. There's a long tradition of not wanting to "waste" threads by blocking on I/O.

DarkNova6 3y ago

Good writeup I recommend to read more than just the headline. My favorite line: > I have a hard time believing that someone who’s biggest complaint is adding “Send” bounds to some generics is engaged in that kind of engineering. Edit: I fully agree with the comment of "duped". I was not aware of the greater context of this discussion. As such, I might have quoted this sentence prematurely.

Pxtl 3y ago

I've never touched Rust, but I can see the complaint -- if I have to write my code in a special way so that its state can be marshalled across threads to redistribute load in a way that I do not need and will make the actual end-to-end latency of a single request actually slower in cases where I've got oodles of CPU head-room, I would find that infuriating. I could see this approach making sense in a platform where transferrable state was the default and is rarely broken, but it doesn't sound like that's the case in Rust? edit: I'm curious, what's the ergonomics like for this? Is it just "oh, your code won't compile if you don't add this magic incantation to say it has `Send`?" Or is it "your code will fail in intermittent, hard-to-debug ways as state gets mangled during work-steals if you don't do this right"?

cmrdporcupine 3y ago
On the whole the compiler detects when things are 'Send' & 'Sync'. If you write your program thread-safe, you won't have issues. And that's the crux of the matter: people are griping on the whole because maybe Tokio async is hard when it puts Send&Sync demands all over the placer, but the reality is that writing safe concurrent code of any kind is hard. It's not intuitive, and the problem is that async makes them feel like it's happening automagically and just "taking care of it" -- but it's really not. You need to know what you're doing, and the compiler is just helping you here. Yes, you can hide that by going thread-per-core and that could eliminate the need for Send in some circumstances (not all). But it might come back to bite you, architecturally, in the long run.
creata 3y ago
> I'm curious, what's the ergonomics like for this? Your code won't compile unless everything is Send/Sync as appropriate, and (I might be wrong, but) the lazy path to achieving this is usually wrapping things that might be shared in Arcs and/or Mutexes.
ko27 3y ago
I think the argument against thread-per-core as the default can be made simply: - if you are CPU bound, work stealing will be better for most cases - if you are IO bound, thread-per-core might work better, but again, you have enough CPU headroom that the performance doesn't really matter IMO, work stealing is a better default to encode into language API

samsquire 3y ago

Thank you for this blog post. I like the thread per core design because kernel context switches are expensive. But userspace context switching scheduling (such as an unbuffered golang channel) for a 64 bits of data at a time makes me uncomfortable too unless it represents a large amount of work backed by a pointer to work data. I think I like to multiplex sockets over threads and multiplex IO over threads so that you can do CPU work while IO is going on and you can process multiple clients per thread. You cannot scale memory mutation by adding threads. You want ideally one thread to own the data and be safe to mutate it uncontendedly. When you send data to another thread, don't refer to that data again. Transfer ownership to that thread. If you can divide your request into phases of expansion (map) and contraction (synchronization) you can do intrarequest parallisation. I've been looking into runqueues of go and tokio where you have a local runqueue without a mutex and a global runqueue with a mutex. I've been trying to think how IO threads that run liburing or epoll can wake up a Coroutine or an async task on a worker thread without the mutex. The worker thread is looking for tasks to resume that are unblocked and needs to be notified when there is IO finished. I think you can have a Coroutine that is always runnable to read from a lock free ringbuffer. You can have that Coroutine that checks for finished IO's ringbuffer yield if its contended by writes by the IO thread that is trying to make work available to the worker threads.

ori_b 3y ago
> I like the thread per core design because kernel context switches are expensive You can do a millions to tens of millions kernel entries/exits per second per core. They're on the same order of magnitude as a single digit number of full cache misses. So, expensive, but a lot less expensive than many people assume. Edit: for some (outdated, 2018) measurements, https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ https://eli.thegreenplace.net/2018/measuring-context-switchi...
- cmrdporcupine 3y ago
  And all our assumptions get blown up with every new hardware generation, too. Hardware engineers are building systems for the applications of today. And the kernel developers are optimizing as well. It's all a moving target. Dogma has no place here.

pornel 3y ago

Everyone who has worked with sharding at scale knows the pain when there's this one terrible key (your biggest customer, hottest key, largest blob) that just ruins load-balancing of the shards.

scottlamb 3y ago
100% true, but in fairness work stealing doesn't completely solve that either. You often end up locking the shard anyway because lockless data structures are limited and hard. And if you're doing something like a key/value store, you're probably sharding to decide which server to direct traffic to, and so you have the same problem at that layer. (There are things that can help, of course, e.g. <https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/ https://aws.amazon.com/blogs/architecture/shuffle-sharding-m...>.)
IshKebab 3y ago
Yeah maybe, but how many people are running these high performance async servers on 2 core machines?

asd4 3y ago

"What they mean by IO bound is actually that their system doesn’t use enough work to saturate a single core when written in Rust: if that’s the case, of course write a single threaded system." Many of the applications I write are like this, a daemon sitting in the background reacting to events. Making them single threaded means I can get rid of all the Arc and Mutex overhead (which is mostly syntactic at that point, but makes debugging and maintenance easier). Being able to do this is one of the things I love about Rust: only pay for what you need. The article that this one is responding to calls out tokio and other async libraries for making it harder to get back to a simple single threaded architecture. Sure there is some hyperbole but I generally agree with the criticism. Making everything more complex by default because its better for high throughput applications seems to be opposite of Rust's ideals.

Kinrany 3y ago
I was going to comment on the same quote. The problem is that one may still want concurrency even when a single thread on a single CPU is enough.
amluto 3y ago
I’ve written services like this, and I would never have called them IO bound. They’re not throughput-bound at all. They mostly sit idle, then they do work and try to get it done quickly to minimize use of system resources. Unless they sometimes get huge bursts of work and something else cares quite a lot about latency during those bursts, using more than one thread adds complexity and overhead for no gain.
- __alexs 3y ago
  In an era of 10Gb NICs in every server very few things are really IO bound.
  - kosolam 3y ago
    10gb nics and their respective connections are quite expensive. Not many servers have these at all.

MrBuddyCasino 3y ago

> This appears in Rust’s APIs as futures needing to be Send, which can be difficult for people with a poor view of their system’s state to figure out the best way to ensure. This is why work-stealing is said to be “harder.” Is it just me or does this come across as rather arrogant? The problem of 'static lifetimes and send/sync constraints resonates among developers, and my impression is not that those were morons.

Guvante 3y ago
Harder is in quotes because it isn't necessarily harder. If you would need to do it anyway it isn't harder. Less "people are worked up over doing a little work" more "async makes you solve problems earlier you were going to solve anyway". Similar vibe to the borrow checker. Sometimes it is overly restrictive othertimes you didn't actually consider the corner cases when you assumed everything would be fine.
withoutboats3 3y ago
I'm just referring back to my earlier point: people say not doing work stealing would be both easier and faster. My claim is that its one or the other, because to get share-nothing to be faster you need to architect your code in a way that is not easier than making a shared-state architecture thread-safe. There is a parallel sentence with "slower" in the next paragraph. I don't think people who struggle to get parallel & concurrent Rust to compile are morons, though I don't like when they act like the APIs we built for them are ruining their lives.
- MrBuddyCasino 3y ago
  Fair enough, the technical point is argued well. Thanks for the clarification.
eminence32 3y ago
From my own personal experience, I definitely struggle sometimes to understand if my state is Send or not. So the line you quoted from the article resonates with me.

duped 3y ago

Personally I feel like this post misses the forest for the trees. The debate isn't about thread-per-core work stealing executors, it's whether async/await is a good abstraction for it in Rust. And the more async code I write the more I feel that it's leaky and hard to program against. The alternative concurrency model people want is structured concurrency via stackful coroutines and channels on top of a work stealing executor. Until someone does the work to demo that and compare it to async/await with futures I don't think there's any productive discussion to be had. People who don't like async are going to avoid it and people who don't care about making sure everything and its mother is Send + Sync + 'static are going to keep on doing it.

withoutboats3 3y ago
In fact there are multiple things in the world that people disagree about. This post addresses a different debate than the one you wanted me to write about, that's all.
- duped 3y ago
  The post you quoted and whose author you insulted at the end was talking about this issue.
  - withoutboats3 3y ago
    No it wasn't. Anyone can follow the link and see that it was about preferring non-work-stealing executors, none of the things you complained about.
zozbot234 3y ago
AIUI, nothing's stopping you from using channels over Rust's existing async support. Stackful coroutines are kinda pointless, since at that point you might as well be using separate threads.
- packetlost 3y ago
  Greenthreads have their benefits though, especially for massive concurrency.

newpavlov 3y ago

I believe that having the Send bound as a requirement to allow migration of tasks between executor threads is a clear deficiency of the Rust async system by itself, together with the fundamental issues around async Drop, which prevent implementation of scoped APIs. Similarly to threads it should be sufficient to have Send bound on functions like spawning and sending data through channels. The share-nothing approach is usually used as nothing more than a workaround to hide this deficiency.. Selectively pinning tasks to a thread/core has advantages and can be really useful in some circumstances, but it's a finer discussion, which has little to do with async users dissatisfaction related to Send.

scottlamb 3y ago

Regarding the quote: > The Original Sin of Rust async programming is making it multi-threaded by default. If premature optimization is the root of all evil, this is the mother of all premature optimizations, and it curses all your code with the unholy Send + 'static, or worse yet Send + Sync + 'static, which just kills all the joy of actually writing Rust. Agree about the melodramatic tone. I also don't think removing the Send + Sync really makes that big a difference. It's the 'static that bothers me the most, and that's not there because of work stealing. I want scoped concurrency. Something like <https://github.com/tokio-rs/tokio/issues/2596 https://github.com/tokio-rs/tokio/issues/2596>. Another thing I really hate about Rust async right now is the poor instrumentation. I'm having a production problem at work right now in which some tasks just get stuck. I wish I could do the equivalent of `gdb; thread apply all bt`. Looking forward to <https://github.com/tokio-rs/tokio/issues/5638 https://github.com/tokio-rs/tokio/issues/5638> landing at least. It exists right now but is experimental and in my experience sometimes panics. I'm actually writing a PR today to at least use the experimental version on SIGTERM to see what's going on, on the theory that if it crashes oh well, we're shutting down anyway. Neither of these complaints would be addressed by taking away work stealing. In fact, I could keep doing down my list, and taking away work stealing wouldn't really help with much of anything.

teraflop 3y ago
> I'm having a production problem at work right now in which some tasks just get stuck. I wish I could do the equivalent of `gdb; thread apply all bt` For all the hate that Java gets, this is something that has Just Worked(tm) for like 25 years, and it's enormously helpful for troubleshooting. You don't even need a debugger; you can just send the JVM a SIGQUIT, and it'll dump a stack trace of every thread to stderr (including which locks each thread is holding and/or waiting on) and keep running. I miss this feature in every other language I've worked with. You can even use it for ad-hoc profiling in production: just take a bunch of snapshots, and use grep/sed/sort/uniq to look for hotspots.
- scottlamb 3y ago
  Go does this well, too. iirc the std library has a package for serving <http://blah/debug/pprof/goroutines http://blah/debug/pprof/goroutines> so you don't even need to ssh in to the server in question. There are libraries for doing the same for any language that just uses kernel threads. It's when you throw in async that you really need to reinvent this kind of observability, and Rust isn't there yet unfortunately.

kosolam 3y ago

Very nice write up that demonstrates the difference between theory and practice.

jacknews 3y ago

Paints the 'rust community' as quite restrictive, and not a place I want to be, IMHO.

weinzierl 3y ago

"This means that any state that is used across a yield point in that task needs to be thread-safe. This appears in Rust’s APIs as futures needing to be Send [..]" Does that mean that I have to unnecessarily use locking (e.g. ARC and Mutex) even with flavor = "current_thread" or am I misunderstanding this?

necubi 3y ago
No locking (which provides mutable `Sync` in rust terms) is generally required, but things must be Send (i.e., allowed to move between threads, but not shared). In my experience is mostly an issue when dealing with references, because `&T` is only Send if `T` is sync. However, `&mut T` is Send if `T` is Send. So if you can avoid aliasing your references, you're also ok.
- weinzierl 3y ago
  Thanks! So does that mean that if I write my code with default tokio and ARC where required and then when I add flavor = "current_thread" I can replace my ARC with Rc?
  - the_white_oak 3y ago
    Rc is not Send. But you can use tokio's LocalSet to run non-Send futures. https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html

hjl22 3y ago

I haven't touched async Rust, but have used normal Rust a little bit. My understanding is that 'static is program lifetime, meaning either static compile-time data like constant strings, or it is memory leaks. How does this apply to async in practice? Do you need to leak things that you want to make async?

scottlamb 3y ago
It means spawned futures can't contain references that aren't 'static. But they can own memory. String for example is 'static. Likewise Vec<T>, Box<T>, Arc<T>, Rc<T>, etc. where T: 'static.
- hjl22 3y ago
  Thanks for the explanation. I understand it now.
pitaj 3y ago
Most heap-allocated values like `String` are also static. Essentially it means that anything the value references is not on the stack.
- hjl22 3y ago
  Thanks for the explanation. I understand it now.
Georgelemental 3y ago
`T: 'static` means that values of the type `T` own all their data (more specifically, their lifetime is not constrained by the lifetime of anything they reference). `&'static T` is a reference to something that lives as long as the program (static data or leaked memory).
- hjl22 3y ago
  Thanks for the explanation. I understand it now.

lordnacho 3y ago

First of all, does anyone know if there's a flag that just turns off work-stealing? Seems like you could then decide for yourself how you want Tokio in particular to work. Second, the send + sync thing seems onerous, but from my POV you are encouraged to only pass little messages on channels anyway. I find it easier to just not share anything. If two things need the same structure, they can both subscribe to the ring that gets the messages, and each construct the object for themselves. I find if you have a bunch of Arc<RwLock<>> something is wrong. YMMV.

c-cube 3y ago
You can configure tokio with feature flags in cargo. I'm particular you can pick a single threaded scheduler.

dpc_01234 3y ago

This is missing the forest for the trees. There is no one always-right way to get best performance for every program. You can argue about it all you want. Thread-per-core benefits/downside is a typical "it depends" discussion. The problem is that using `async` in the first place is a premature optimization. 99% of Rust programs are not redis, linkerd and alikes. They are some CLI tools or Web apps that could have been written in Python or Ruby, and they would still be fast enough. So why as a community we abandoned blocking IO Rust and everything is async now, and developers are just doing `#[tokio::main]` by default on everything?

vbezhenar 3y ago
One reason might be: if you're fine with thread-per-core performance, you probably wouldn't want to use Rust in the first place, as there are language with better programming experience trading it for speed. Like python. And if you want to use Rust, probably you need extra performance and you can adopt less convenient style (because you already adapted less convenient language) to get better performance.
- dpc_01234 3y ago
  Wat. No.

sakras 3y ago

> The problem with work-stealing is that it means a task can run on one thread, pause, and then be started again on another thread: that’s what it means for the work to be stolen. This means that any state that is used across a yield point in that task needs to be thread-safe. Do any existing "thread-per-core" systems actually provide yield points as a thing you can do? Most of my experience is with OpenMP (using both BSP and task parallelism) and a little bit of TBB (also task parallelism). If you want to yield in these systems, you break it up into two tasks. > if state is moved from one thread to another, this introduces synchronization costs This isn't obvious to me. If you're moving state, then there is no synchronization because only one thread is touching the state at any given point. Unless we're talking about synchronization due to starting a new task?

catern 3y ago

An important thing omitted in this post, which makes work-stealing less attractive, is that one core being idle can actually improve performance of other cores. Today's CPUs basically have a fixed energy budget, and if one core is idle that means more of that budget can go to other cores. In other words, core utilization is less relevant today - what you care about is energy utilization (which is shared across cores). Of course, there's a point at which this stops being relevant - if you have multiple sockets for example, this won't apply. But work stealing across multiple sockets is so expensive anyway that you would never want to do it. You might as well work-steal across machines at that point - something which is indeed useful sometimes, but usually niche.

jeffbee 3y ago
You can practice energy-aware scheduling at higher levels, too. If you have to send an RPC and you can choose between multiple peers, choose the one with the coldest CPU temperature.
CyberDildonics 3y ago
If a CPU is being cooled enough to not throttle, it is much more time and energy efficient to use all the cores you can rather than have another core run at a slightly higher frequency. Higher frequencies have diminishing returns and exponential heat loss. You might as well work-steal across machines at that point Shared memory is extremely fast, it crushes using local loopback networking, let alone using actual networking.

jandrewrogers 3y ago

The original problem thread-per-core was invented to solve ~15 years ago was scalability and efficiency of compute on commodity many-core servers. Contrary to what some have suggested, thread-per-core was expressly about optimizing for CPU bound workloads. It turned out to be excellent for high-throughput I/O bound workloads later, albeit requiring more sophisticated I/O handling. When I read articles like this, it looks like speed-running the many software design mistakes that were made when thread-per-core architectures were introduced. To be fair, thread-per-core computer science is poorly documented, having originated primarily in HPC. This article focuses on a vexing problem of thread-per-core architectures: balancing work across cores. There are four elementary models for this, push/pull of data/load. Work-stealing is essentially the "load pull" model. This only has low overhead if you almost never need to use it e.g. if the work is naturally balanced in a way that few real-world problems actually are. For workloads where dynamic load skew across cores is common, which is the more interesting problem, work-stealing becomes a performance bottleneck due to coordination overhead. Nonetheless, it is easy to understand so people still use work-stealing when the workload is amenable to it, it just doesn’t generalize well. There are a few rare types of workloads (not mentioned in the article) where it is probably the best choice. The model with the most gravity these days seems to be "data push", which is less intuitive but requires much less thread coordination. The "data push" model has its own caveats — there are workloads for which it is poor — but it generalizes well to most common workloads. Thread-per-core architectures are here to stay -- they cannot be beat for scalability and efficiency. However, I have observed that most software engineers have limited intuition for what a modern and idiomatic thread-per-core design looks like, made worse by the fact that there are relatively few articles or papers that go deep on this subject.

withoutboats3 3y ago
Thanks for this response, it's really interesting. > For workloads where dynamic load skew across cores is common, which is the more interesting problem, work-stealing becomes a performance bottleneck due to coordination overhead. Nonetheless, it is easy to understand so people still use work-stealing when the workload is amenable to it, it just doesn’t generalize well This sounds right to me. The reason Rust async frameworks use work-stealing is mainly that it's easy to enable at the framework level and will improve performance in a lot of applications, especially those that are not ideally architected. Based on your comment and your self-description on your profile, these are not the kinds of applications you work on. I would be interested in receiving links to more literature.

cryptonector 3y ago

> It’s always off-putting to me that claims written this way can be taken seriously as a technical criticism, but our industry is rather unserious. It's not the industry. It's people. > What these people advocate instead is an alternative architecture that they call “thread-per-core.” They promise that this architecture will be simultaneously more performant and easier to implement. In my view, the truth is that it may be one or the other, but not both. Thread-per-core is hands-down better for performance. That's because you can't rely on storing application state smeared on a thread/co-routine stack, so you have to instead make all your state explicit, and in the process you'll effectively compress it by no longer smearing it on a very large data structure called the call stack. Smaller state == less cache thrashing. The problem with thread-per-core is that you really need async/await or coding in continuation passing style in order to make it work. CPS is mostly not an option in most cases, so async/await it has to be. Developers should understand the concepts of thread-per-core and thread-per-client and pick the best one for their case. IMO thread-per-core is always better because if your application explodes in popularity then you'll really care to make it more efficient, but rewrites are always hard or infeasible, so if you get it right from the beginning, then you win. What I mean by "pick the best one" then has to do with the developer's skills and productivity and time constraints. Sometimes "best" is about what you can manage, not what's "best" in an objective vacuum that ignores real-world constraints.

IshKebab 3y ago
> It's not the industry. It's people. Yes, and this is just naive to think that other industries are somehow bastions of serious debate.

mikhailfranco 3y ago

Erlang/Elixir/BEAM emphasizes share nothing, allows (encourages) a bezillion user-space processes, then executes with a thread-per-core (by default). The actual number of schedulers (real threads) is configurable as a command line option, but it's rare, approaching unheard-of, to override the default. Virding's First Rule of Programming ... https://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html https://rvirding.blogspot.com/2008/01/virdings-first-rule-of...

jerf 3y ago
If Go with its green threads is a step down from Rust in performance, Erlang is two or three steps down from Go. If you step down your performance needs, a lot of these problems melt away. Most programmers should indeed do that. There's no need to bring these problems on yourself if you don't actually need them. Personally I harbor a deep suspicion a non-trivial amount of the stress in the Rust ecosystem over async and its details is coming from people who don't actually need the performance they are sacrificing for. (Obviously, there absolutely people who do need that performance and I am 100% not talking about them.) But it's hard to tell, because they don't exactly admit that's what they're doing if you ask, or, at least, not until many years and grey hairs later. But in the meantime, some language needs to actually solve these problems (better than C++), and since Rust has volunteered for that role, that means that at the limit, the fact that other languages that chose to just take a performance hit don't seem to have these problems doesn't have very many applicable lessons for Rust, at least when it is being used at these maximum performance levels.
- mikhailfranco 3y ago
  Agree, Erlang will never win any performance benchmarks, but that is mostly due to other aspects of the language: big integers, string handling, safer-rather-than-faster floating point, etc. [Elixir is a little better, supporting binary-first for strings, rather than charlists - Erlang is very good at pattern-matching binaries.] Share-nothing and thread-per-core are good for many reasons, including performance, but they also feed into the main philosophies for Erlang development: resilience, horizontal scalability and comprehensibility. As Joe Armstrong said: “Make it work, then make it beautiful, then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful.”

davidhyde 3y ago

> “The problem with work-stealing is that it means a task can run on one thread, pause, and then be started again on another thread”. The article explains how work-stealing is a way to solve tail latency. I understand tail latency to be caused by tasks that yield (by themselves) after an unexpectedly long amount of time. This cannot be know in advance. The article explains how work-stealing results in cache misses and extra developer constraints like Send, Sync and ‘static. What if the executor only moves a task to another thread after a timeout period has expired? This should result in no latency penalty if things are running smoothly and a constant, low, extra latency when tasks need to be stolen now and again. This would mitigate cache miss issues but alas not the developer overhead of all those multithreaded constraints.

exfalso 3y ago

TLDR summary: 1. Straw man 2. Talk about an unrelated paper 3. Conclusion: I am smart and those people are dumb On a somewhat related note: I'd say 95% of the software I've written was IO bound(and if it wasn't at the beginning then the goal was to make it IO-bound), and that remaining 5% does not benefit at all from async/coroutines(used them in Haskell, Kotlin(Quasar) and Rust). I'm very curious about what real-world CPU-bound use cases people have that can benefit from async performance-wise. If you're optimizing on that microsecond-nanosecond scale then you shouldn't even have yields/blocks on your hot paths, and synchronization will at most be done using memfences, so what are we talking about? I'd also conjecture that if you place your problem domain on the IO vs CPU bound axis, the problems where async would provide performance benefits can invariably be solved by other designs that perform even better (GPU/FPGA). Yes I'm one of those people who thinks coroutines have extremely marginal actual value, and most of that value is the "feeling of how cool this is" that people experience when they first learn about the concept. If there was a way to bin the concept altogether I would do it in a heartbeat. As-is, the entire Rust ecosystem is suffering heavily because of it.

slaymaker1907 3y ago
A lot of the time, the benefit can be from much more controlled scheduling without having to introduce actual threads. A lot of software doesn't want to take the overhead of creating platform threads (memory and taking additional CPU time) but does want to do scheduling internally for various CPU bound workloads. I think web pages are often like this. You don't really want to hog tons of CPU, but you also want to have guarantees like not blocking rendering when running a user provided regex since said regex could take a long time to run, but in practice that regex will be fast. Even if we want to eventually offload that to a separate thread, async gives us the option to first try evaluating it in the main thread for some number of iterations before offloading it. GPU/FPGA programming is kind of irrelevant because as expensive as developing high performance code is for CPUs, costs go up by an order of magnitude for those platforms unless there is some existing library you can utilize (mainly applicable for ML/AI). These platforms are also very expensive. It's like saying just rent a helicopter if you need to get somewhere quickly instead of asking what "quickly" actually means and optimizing accordingly.

cryptonector 3y ago

Work stealing is like process migration, and as such it has significant overhead and you really need to be picky about when to rebalance by moving loads.

Perenti 3y ago

Perhaps the title should mention this is primarily about Rust, and not thread-per-core in general, which would be interesting.