6 ms·

I've never touched Rust, but I can see the complaint -- if I have to write my code in a special way so that its state can be marshalled across threads to redist

by Pxtl 3y ago

I've never touched Rust, but I can see the complaint -- if I have to write my code in a special way so that its state can be marshalled across threads to redistribute load in a way that I do not need and will make the actual end-to-end latency of a single request actually slower in cases where I've got oodles of CPU head-room, I would find that infuriating. I could see this approach making sense in a platform where transferrable state was the default and is rarely broken, but it doesn't sound like that's the case in Rust? edit: I'm curious, what's the ergonomics like for this? Is it just "oh, your code won't compile if you don't add this magic incantation to say it has `Send`?" Or is it "your code will fail in intermittent, hard-to-debug ways as state gets mangled during work-steals if you don't do this right"?

cmrdporcupine 3y ago
On the whole the compiler detects when things are 'Send' & 'Sync'. If you write your program thread-safe, you won't have issues. And that's the crux of the matter: people are griping on the whole because maybe Tokio async is hard when it puts Send&Sync demands all over the placer, but the reality is that writing safe concurrent code of any kind is hard. It's not intuitive, and the problem is that async makes them feel like it's happening automagically and just "taking care of it" -- but it's really not. You need to know what you're doing, and the compiler is just helping you here. Yes, you can hide that by going thread-per-core and that could eliminate the need for Send in some circumstances (not all). But it might come back to bite you, architecturally, in the long run.
creata 3y ago
> I'm curious, what's the ergonomics like for this? Your code won't compile unless everything is Send/Sync as appropriate, and (I might be wrong, but) the lazy path to achieving this is usually wrapping things that might be shared in Arcs and/or Mutexes.
ko27 3y ago
I think the argument against thread-per-core as the default can be made simply: - if you are CPU bound, work stealing will be better for most cases - if you are IO bound, thread-per-core might work better, but again, you have enough CPU headroom that the performance doesn't really matter IMO, work stealing is a better default to encode into language API
- layer8 3y ago
  > you have enough CPU headroom that the performance doesn't really matter But it would affect power consumption? (Just trying to understand.)
  - basro 3y ago
    No, performance and power consumption should go hand in hand in this case. If you are strongly IO bound, paying for the synchronization is not really going to matter much I believe. There are cases where you can be CPU bound and using the share nothing model would work out to your advantage. There's also the case where you only have one cpu core anyway (for example if you want to get all the juice out of a cheap single core VPS)
- vlovich123 3y ago
  The main argument for work stealing is that it’s hard to achieve uniformity of work loads across all threads. The main argument for a single-threaded thread per core design is that it’s easier to code AND performs/scales way better than work stealing (including average and tail latencies, TPS etc). IMHO it’s a misconception that this is somehow tied to CPU or IO bound work. Take for example databases. You’d think that that’s the prime “I/O bound” use case. Except it’s not. There’s a talk about a DB researcher that analyzed that Postgres spends 70% of its time book keeping things within the database. And that makes sense. I/O is done in bulk with the cost amortized over a lot of transactions. That book keeping work? Extremely expensive because you have to acquire locks all over the place, do atomics, memory allocations etc. Atomics and memory allocations are extremely expensive in certain contexts and atomics also have a negative in that your scaling with number of CPUs is sub linear due to hard to remove false sharing of cache lines and CPU stalls to handle the synchronization. On the other hand, a shared nothing approach where you’re not allocating memory in your hot path is very hard to achieve and not suitable for all problems. Nor does everyone need that performance. So the work stealing approach is better in those use cases as it provides reasonable performance and the programming model is simpler in some ways since you don’t have to think about the data path as careful since everything has an Arc / Mutex in there.
duped 3y ago
> edit: I'm curious, what's the ergonomics like for this? Is it just "oh, your code won't compile if you don't add this magic incantation to say it has `Send`?" Or is it "your code will fail in intermittent, hard-to-debug ways as state gets mangled during work-steals if you don't do this right"? You write some code that looks like this. struct Server { // ... } impl Server { async fn serve(&self) { loop { let server = self.clone(); let message = read_message().await; // Each message handler is a new task, seems reasonable? spawn(async move { let result = server.handler(message).await; write_response(result); }) } } async fn handler(&self, arg:Message) -> Result<U, Error> { let this = do_this(); let that = do_that(arg).await; do_the_rest(this, that); } } And everything works fine. Then one day you change the implementation of `do_this` such that the type of `this` is no longer `Send`. You will get a nasty compile error that spawn(...) doesn't work because the type created by the anonymous scope in `async move { }` is not Send. The reason is not necessarily obvious (and the error message is unhelpful). If `this` is not `Send` then you can't hold it across the `.await` of `do_that(arg).await`, because each .await represents a point in execution where the future may yield and be scheduled onto another thread by the executor. If you can make the type Send then everything is fine. If you can't (which is entirely possible!) then you need to change the scheduling of the future to `spawn_local` (or whatever your async executor calls it). This may require adding a bunch of boilerplate to even call `spawn_local` in the first place. This is the issue with Send. It's not just adding type annotations. There are subtle ways it infects your code that may cause it to break later in non-obvious ways, because if a type implements Send is not always obvious.
- diath 3y ago
  > And everything works fine. Until the handler needs to access a resource that's shared by tasks.
  - duped 3y ago
    That's an example of how your code can subtly break

by Pxtl 3y ago

cmrdporcupine 3y ago

On the whole the compiler detects when things are 'Send' & 'Sync'. If you write your program thread-safe, you won't have issues. And that's the crux of the matter: people are griping on the whole because maybe Tokio async is hard when it puts Send&Sync demands all over the placer, but the reality is that writing safe concurrent code of any kind is hard. It's not intuitive, and the problem is that async makes them feel like it's happening automagically and just "taking care of it" -- but it's really not. You need to know what you're doing, and the compiler is just helping you here. Yes, you can hide that by going thread-per-core and that could eliminate the need for Send in some circumstances (not all). But it might come back to bite you, architecturally, in the long run.

creata 3y ago

> I'm curious, what's the ergonomics like for this? Your code won't compile unless everything is Send/Sync as appropriate, and (I might be wrong, but) the lazy path to achieving this is usually wrapping things that might be shared in Arcs and/or Mutexes.

ko27 3y ago

I think the argument against thread-per-core as the default can be made simply: - if you are CPU bound, work stealing will be better for most cases - if you are IO bound, thread-per-core might work better, but again, you have enough CPU headroom that the performance doesn't really matter IMO, work stealing is a better default to encode into language API

layer8 3y ago
> you have enough CPU headroom that the performance doesn't really matter But it would affect power consumption? (Just trying to understand.)
- basro 3y ago
  No, performance and power consumption should go hand in hand in this case. If you are strongly IO bound, paying for the synchronization is not really going to matter much I believe. There are cases where you can be CPU bound and using the share nothing model would work out to your advantage. There's also the case where you only have one cpu core anyway (for example if you want to get all the juice out of a cheap single core VPS)
vlovich123 3y ago
The main argument for work stealing is that it’s hard to achieve uniformity of work loads across all threads. The main argument for a single-threaded thread per core design is that it’s easier to code AND performs/scales way better than work stealing (including average and tail latencies, TPS etc). IMHO it’s a misconception that this is somehow tied to CPU or IO bound work. Take for example databases. You’d think that that’s the prime “I/O bound” use case. Except it’s not. There’s a talk about a DB researcher that analyzed that Postgres spends 70% of its time book keeping things within the database. And that makes sense. I/O is done in bulk with the cost amortized over a lot of transactions. That book keeping work? Extremely expensive because you have to acquire locks all over the place, do atomics, memory allocations etc. Atomics and memory allocations are extremely expensive in certain contexts and atomics also have a negative in that your scaling with number of CPUs is sub linear due to hard to remove false sharing of cache lines and CPU stalls to handle the synchronization. On the other hand, a shared nothing approach where you’re not allocating memory in your hot path is very hard to achieve and not suitable for all problems. Nor does everyone need that performance. So the work stealing approach is better in those use cases as it provides reasonable performance and the programming model is simpler in some ways since you don’t have to think about the data path as careful since everything has an Arc / Mutex in there.

duped 3y ago

> edit: I'm curious, what's the ergonomics like for this? Is it just "oh, your code won't compile if you don't add this magic incantation to say it has `Send`?" Or is it "your code will fail in intermittent, hard-to-debug ways as state gets mangled during work-steals if you don't do this right"? You write some code that looks like this. struct Server { // ... } impl Server { async fn serve(&self) { loop { let server = self.clone(); let message = read_message().await; // Each message handler is a new task, seems reasonable? spawn(async move { let result = server.handler(message).await; write_response(result); }) } } async fn handler(&self, arg:Message) -> Result<U, Error> { let this = do_this(); let that = do_that(arg).await; do_the_rest(this, that); } } And everything works fine. Then one day you change the implementation of `do_this` such that the type of `this` is no longer `Send`. You will get a nasty compile error that spawn(...) doesn't work because the type created by the anonymous scope in `async move { }` is not Send. The reason is not necessarily obvious (and the error message is unhelpful). If `this` is not `Send` then you can't hold it across the `.await` of `do_that(arg).await`, because each .await represents a point in execution where the future may yield and be scheduled onto another thread by the executor. If you can make the type Send then everything is fine. If you can't (which is entirely possible!) then you need to change the scheduling of the future to `spawn_local` (or whatever your async executor calls it). This may require adding a bunch of boilerplate to even call `spawn_local` in the first place. This is the issue with Send. It's not just adding type annotations. There are subtle ways it infects your code that may cause it to break later in non-obvious ways, because if a type implements Send is not always obvious.

diath 3y ago
> And everything works fine. Until the handler needs to access a resource that's shared by tasks.
- duped 3y ago
  That's an example of how your code can subtly break