11 ms·

Scipio: A Thread-per-Core Crate for Rust and Linux

167 points by akshaykumar90 6y ago

Horusiath 6y ago
Can we make it work in Windows by any chance?
- deleted 6y ago
  [deleted]
- fhdksjdhdkdl 6y ago
  Surely there's an extant tool to do the same thing. > We know that thread-per-core can deliver significant efficiency gains. But what is it? If someone were to write this as parody people would call it a hack job
- danielheath 6y ago
  Do it!
- wongarsu 6y ago
  In principle you could likely do it. Either call SetProcessAffinityMask on every process on the system to keep them off your dedicated cores, and add a hook to call it on every newly started process. Or simply start your threads with the highest possible priority, with affinity for one core, and never use blocking APIs. In the Windows scheduler a thread of higher priority always gets priority over threads with lower priority, so if you set your priority sufficiently high and never tell the kernel that you are waiting on something you shouldn't get interrupted.
  - jpf0 6y ago
    In practice, we've experienced 20 microsecond-100 millisecond thread interruption issues on Windows. The resources we've found are inadequate to address the questions. No such problems on Linux. Windows scheduler experts appear to be rare. We wonder if Windows is simply not amenable to certain types of high-performance applications.
- WJW 6y ago
  Probably, but you will have to rip out all the io_uring specific bits and replace them with their Windows equivalent. Probably something with I/O completion ports. If you are interested, look into the new Windows I/O scheduler that was implemented for GHC 9.0 (in Haskell, obv). There is also some preliminary work to integrate io_uring into the Haskell runtime if it' available, but the Haskell runtime philosophy is rather different from the Rust approach.
  - Arnavion 6y ago
    Worth noting that having number of IOCP worker threads == number of cores has always been the official suggestion for building IOCP applications. Eg https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-co... >The concurrency value of a completion port is specified when it is created with CreateIoCompletionPort via the NumberOfConcurrentThreads parameter. This value limits the number of runnable threads associated with the completion port. >[...] >The best overall maximum value to pick for the concurrency value is the number of CPUs on the computer.
indy 6y ago
Would a thread-per-core architecture help mitigate security vulnerabilities like spectre and meltdown?
- danielheath 6y ago
  Threads are still going to make syscalls into kernel code, so... not as much, no.
- pjmlp 6y ago
  You are better with processes per core, with no threads.
- remram 6y ago
  This prevents the application's thread from moving cores, but doesn't prevent an unrelated thread from using that core (especially in a hypervised environment). That unrelated thread could still perform those attacks against the application's thread.
MrBuddyCasino 6y ago
Finally, a Seastar clone for Rust! Really impressed by some of the work coming out from Datadog. Curious if one of the popular Rust web frameworks will choose it as I/O backend, could be a great way to push down latencies even further.
- glaubercosta 6y ago
  Seastar folks are really smart.
rwha 6y ago
With a name like Scipio I would imagine it also uses a thread-per-core of neighboring computers as well.
Cloudef 6y ago
Why not use multiple processes instead?
- gpderetta 6y ago
  for a more constructive reply than a downvote, using multiple threads allows you to post closures between executors which is a very flexible and powerful message passing model. I haven't looked deep enough into scipio, but as it is inspired on seastar I assume it allows that as well. Multiple processes works better if you have little or no sharing or communication between the threads.
  - glaubercosta 6y ago
    Exactly right. Scipio doesn't support that yet, but keep in mind that when Hannibal was out there putting everyone to shame Scipio was still a child. There is already a PR to support that that I intend to merge soon. Indeed without that, you might as well use multiple processes
- penberg 6y ago
  Excellent question! I think VoltDB, for example, uses the same application-level data partitioning approach, but with processes instead of threads. One advantage of the "thread-per-core" approach is that it allows fast communication between shards because the underlying threads share the same address space. In contrast, processes need to use a more heavy-weight approach of inter-process communication (IPC). Of course, process-based systems will have a reliability advantage, because processes cannot easily mess with each others state.
  - fulafel 6y ago
    You can still use shared memory with processes.
- quietbritishjim 6y ago
  To turn this question on its head, why use multiple processes? That's not just rhetorical, I don't understand why you'd ask that question. The main advantages of separate processes are (1) any global variables are not shared between them, which is sometimes what you want (like Python's GIL), and (2) there is memory protection. Neither of those seem relevant to a Rust data processing program. To actually answer your question a bit: Threads allow all the benefits of processes (except the two I just mentioned), with the added benefit of being lower overhead and allow sharing information more easily (but you can still use full-blown IPC to communicate between them if you wish).
di4na 6y ago
So this sounds a lot like an application of Virding's First Rule of Programming. Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang. http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-...
- unrealhoang 6y ago
  How is this in any way related to erlang?
  - tybit 6y ago
    State being assigned to a single thread removing the need to use locks is a major selling point of Erlang/BEAM.
    - unrealhoang 6y ago
      This post is about the architecture to pin multiple tasks (read erlang processes) that use the same data shard (read shared data) on to a single thread to avoid lock and furthermore, never block on that thread to avoid context switch.
    - imtringued 6y ago
      The only requirement the actor model demands is that actors only have access to their own data but how actors are scheduled is up to the runtime. It could be that Erlang/BEAM does pin actors to a single thread and then pins those threads to a single core but it is not 100% necessary.
      - dnautics 6y ago
        nonetheless, the BEAM does do that, and only revokes actors from a core under certain conditions (power saving). Also it's important to note that the BEAM is not "the actor model" so it is not bound by the requirements thereof, it just "looks like an actor if you squint at it a little bit".
BenoitP 6y ago
I'll go against the grain here and say that async/await, wether implemented by one thread-per-core like here or by stackless coroutines is not the solution. Async/await will make complexity explode because of the colored function problem [1]. The solution to expensive context switches is cheap context switches, plain and simple. User-mode lightweight threads like go's, or upcoming Java's with Loom [2] have proven that this is possible. Yes, it does mean that it can only happen in a language that controls its stack (so that you can slice it off and pop a continuation on it). I sincerely believe this is Rust's ballpark; hell they even started the project with that idea in mind. [1] https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... [2] https://jdk.java.net/loom/ https://jdk.java.net/loom/
- unrealhoang 6y ago
  Is there a mechanism to group some goroutines onto a single CPU thread to avoid locking? In the environments that have to rely on thread-per-core (sensitive to even 50us), blocking functions will become obvious in the profiling report.
  - BenoitP 6y ago
    > Is there a mechanism to group some goroutines onto a single CPU thread to avoid locking? Python's GIL comes to mind :p
- penberg 6y ago
  So thread-per-core is not just about eliminating context switches. It's also about partitioning application-level data to reduce inter-core synchronization to let speculative, out-of-order cores run independently as much as possible. Don't get me wrong, user-level threads are great, and arguably much simpler programming model than async/await. But they're not _the solution_ either.
  - BenoitP 6y ago
    They're not the solution to well-sliced data processing, but I can't see how they would hurt the performance.
faitswulff 6y ago
Scipio has seen some action on HN before in "C++ vs Rust: an async Thread-per-Core story": https://news.ycombinator.com/item?id=24444347 https://news.ycombinator.com/item?id=24444347
Luker88 6y ago
I am a fan of the thread-per-core model, but I do not share their views on the data sharding per core While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load) Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great
- mjb 6y ago
  > While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards Serial access to hot keys is a hard thing to design around, and you're right that sharding doesn't solve that problem. Worse, it exposes other keys that just happen to share the shard (or the core) to poor performance. There are a couple of well-understood solutions to this problem. The obvious one is to dynamically re-balance shards based on heat, either moving some keys or split/merge. This is the same tactic that many distributed databases use, and while it's complex to do, it's easier to do locally than distributed. Another option is stochastic re-balancing, like the Stochastic Fairness Queuing (https://ieeexplore.ieee.org/document/91316 https://ieeexplore.ieee.org/document/91316) model used in networking. Here, shards are randomly re-shuffled occasionally. Doesn't fix the noisy neighbor problem, but does mean that the noisyness moves around. That might seem silly, but it's pretty much what's going to happen under the covers of the non-sharded version of the code when the scheduler gets involved, only easier to reason about. > When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load) Lower apparent performance for neighbors of the heavy hitter, sure. Under which other circumstances does it reduce performance? > Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great I'm a bit puzzled by this. Distributed systems have exactly the same problem, and solving that problem is much harder there because the cost of contention is higher, data movement is more expensive, and you have to deal with a lot more failure cases. The statistics may be better for distributed systems because a hot tenant has to be a lot hotter to make hot box than a hot core. But that's a very specific kind of bet, and if you end up with a tenant that does cause a hot box you have an even harder problem to solve. Dynamo (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos...) solves this by moving the key space around, as do many similar kinds of systems. It's not easy, though, and filled with caveats. If you're scared of sharding, distributed sharding should be scarier than on-box sharding.
shwestrick 6y ago
Thread-per-core has been around for a while! It's how a lot of modern multicore languages work. You've probably heard of "green" or "lightweight" threads before... it's all the same idea. Typically, you'd want to use a scheduler (probably work-stealing https://en.wikipedia.org/wiki/Work_stealing https://en.wikipedia.org/wiki/Work_stealing) under the hood to dynamically assign tasks to processors, which is much more effective at load-balancing than "sharding". All of these languages/libraries use a dynamic scheduler for load-balancing: * Rayon (Rust) [https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon] * Goroutines (Go) [https://golangbyexample.com/goroutines-golang/ https://golangbyexample.com/goroutines-golang/] * OpenMP [https://www.openmp.org/ https://www.openmp.org/] * Task Parallel Library (.NET) [https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl https://docs.microsoft.com/en-us/dotnet/standard/parallel-pr...] * Thread Building Blocks (C++) [https://software.intel.com/content/www/us/en/develop/tools/threading-building-blocks.html https://software.intel.com/content/www/us/en/develop/tools/t...] * Cilk (C/C++) [http://cilk.mit.edu/ http://cilk.mit.edu/] * Java Fork-Join and Parallel Streams [https://docs.oracle.com/javase/tutorial/collections/streams/parallelism.html https://docs.oracle.com/javase/tutorial/collections/streams/...] * ParlayLib (C++) [https://github.com/cmuparlay/parlaylib https://github.com/cmuparlay/parlaylib]
- imtringued 6y ago
  If it's not partitioning data per processing unit then it would not be considered a "thread-per-core" architecture based on the definition the article provided. Work stealing means more than one thread can be responsible for a single piece of data. This could result in crossing NUMA nodes or servers.
  - shwestrick 6y ago
    Hmm I was separating the concept of "thread-per-core" from sharding. I would argue that typical cooperative task schedulers (e.g. work-stealing) get the performance benefits of thread-per-core without requiring any static partitioning. But if thread-per-core is fundamentally tied to the idea of sharding, then I think I see what you're saying.
lomkju 6y ago
This only makes sense if the local I/O is really fast and in many cases you contact external databases where the context switches are negligible to the I/O time spent on a query.
- gpderetta 6y ago
  Suru, but this is a good architecture for writing databases in the first place.

167 points by akshaykumar90 6y ago

Horusiath 6y ago

Can we make it work in Windows by any chance?

deleted 6y ago
[deleted]
fhdksjdhdkdl 6y ago
Surely there's an extant tool to do the same thing. > We know that thread-per-core can deliver significant efficiency gains. But what is it? If someone were to write this as parody people would call it a hack job
danielheath 6y ago
Do it!
wongarsu 6y ago
In principle you could likely do it. Either call SetProcessAffinityMask on every process on the system to keep them off your dedicated cores, and add a hook to call it on every newly started process. Or simply start your threads with the highest possible priority, with affinity for one core, and never use blocking APIs. In the Windows scheduler a thread of higher priority always gets priority over threads with lower priority, so if you set your priority sufficiently high and never tell the kernel that you are waiting on something you shouldn't get interrupted.
- jpf0 6y ago
  In practice, we've experienced 20 microsecond-100 millisecond thread interruption issues on Windows. The resources we've found are inadequate to address the questions. No such problems on Linux. Windows scheduler experts appear to be rare. We wonder if Windows is simply not amenable to certain types of high-performance applications.
WJW 6y ago
Probably, but you will have to rip out all the io_uring specific bits and replace them with their Windows equivalent. Probably something with I/O completion ports. If you are interested, look into the new Windows I/O scheduler that was implemented for GHC 9.0 (in Haskell, obv). There is also some preliminary work to integrate io_uring into the Haskell runtime if it' available, but the Haskell runtime philosophy is rather different from the Rust approach.
- Arnavion 6y ago
  Worth noting that having number of IOCP worker threads == number of cores has always been the official suggestion for building IOCP applications. Eg https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-co... >The concurrency value of a completion port is specified when it is created with CreateIoCompletionPort via the NumberOfConcurrentThreads parameter. This value limits the number of runnable threads associated with the completion port. >[...] >The best overall maximum value to pick for the concurrency value is the number of CPUs on the computer.

indy 6y ago

Would a thread-per-core architecture help mitigate security vulnerabilities like spectre and meltdown?

danielheath 6y ago
Threads are still going to make syscalls into kernel code, so... not as much, no.
pjmlp 6y ago
You are better with processes per core, with no threads.
remram 6y ago
This prevents the application's thread from moving cores, but doesn't prevent an unrelated thread from using that core (especially in a hypervised environment). That unrelated thread could still perform those attacks against the application's thread.

MrBuddyCasino 6y ago

Finally, a Seastar clone for Rust! Really impressed by some of the work coming out from Datadog. Curious if one of the popular Rust web frameworks will choose it as I/O backend, could be a great way to push down latencies even further.

glaubercosta 6y ago
Seastar folks are really smart.

rwha 6y ago

With a name like Scipio I would imagine it also uses a thread-per-core of neighboring computers as well.

Cloudef 6y ago

Why not use multiple processes instead?

gpderetta 6y ago
for a more constructive reply than a downvote, using multiple threads allows you to post closures between executors which is a very flexible and powerful message passing model. I haven't looked deep enough into scipio, but as it is inspired on seastar I assume it allows that as well. Multiple processes works better if you have little or no sharing or communication between the threads.
- glaubercosta 6y ago
  Exactly right. Scipio doesn't support that yet, but keep in mind that when Hannibal was out there putting everyone to shame Scipio was still a child. There is already a PR to support that that I intend to merge soon. Indeed without that, you might as well use multiple processes
penberg 6y ago
Excellent question! I think VoltDB, for example, uses the same application-level data partitioning approach, but with processes instead of threads. One advantage of the "thread-per-core" approach is that it allows fast communication between shards because the underlying threads share the same address space. In contrast, processes need to use a more heavy-weight approach of inter-process communication (IPC). Of course, process-based systems will have a reliability advantage, because processes cannot easily mess with each others state.
- fulafel 6y ago
  You can still use shared memory with processes.
quietbritishjim 6y ago
To turn this question on its head, why use multiple processes? That's not just rhetorical, I don't understand why you'd ask that question. The main advantages of separate processes are (1) any global variables are not shared between them, which is sometimes what you want (like Python's GIL), and (2) there is memory protection. Neither of those seem relevant to a Rust data processing program. To actually answer your question a bit: Threads allow all the benefits of processes (except the two I just mentioned), with the added benefit of being lower overhead and allow sharing information more easily (but you can still use full-blown IPC to communicate between them if you wish).

di4na 6y ago

So this sounds a lot like an application of Virding's First Rule of Programming. Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang. http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-...

unrealhoang 6y ago
How is this in any way related to erlang?
- tybit 6y ago
  State being assigned to a single thread removing the need to use locks is a major selling point of Erlang/BEAM.
  - unrealhoang 6y ago
    This post is about the architecture to pin multiple tasks (read erlang processes) that use the same data shard (read shared data) on to a single thread to avoid lock and furthermore, never block on that thread to avoid context switch.
  - imtringued 6y ago
    The only requirement the actor model demands is that actors only have access to their own data but how actors are scheduled is up to the runtime. It could be that Erlang/BEAM does pin actors to a single thread and then pins those threads to a single core but it is not 100% necessary.
    - dnautics 6y ago
      nonetheless, the BEAM does do that, and only revokes actors from a core under certain conditions (power saving). Also it's important to note that the BEAM is not "the actor model" so it is not bound by the requirements thereof, it just "looks like an actor if you squint at it a little bit".

BenoitP 6y ago

I'll go against the grain here and say that async/await, wether implemented by one thread-per-core like here or by stackless coroutines is not the solution. Async/await will make complexity explode because of the colored function problem [1]. The solution to expensive context switches is cheap context switches, plain and simple. User-mode lightweight threads like go's, or upcoming Java's with Loom [2] have proven that this is possible. Yes, it does mean that it can only happen in a language that controls its stack (so that you can slice it off and pop a continuation on it). I sincerely believe this is Rust's ballpark; hell they even started the project with that idea in mind. [1] https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... [2] https://jdk.java.net/loom/ https://jdk.java.net/loom/

unrealhoang 6y ago
Is there a mechanism to group some goroutines onto a single CPU thread to avoid locking? In the environments that have to rely on thread-per-core (sensitive to even 50us), blocking functions will become obvious in the profiling report.
- BenoitP 6y ago
  > Is there a mechanism to group some goroutines onto a single CPU thread to avoid locking? Python's GIL comes to mind :p
penberg 6y ago
So thread-per-core is not just about eliminating context switches. It's also about partitioning application-level data to reduce inter-core synchronization to let speculative, out-of-order cores run independently as much as possible. Don't get me wrong, user-level threads are great, and arguably much simpler programming model than async/await. But they're not _the solution_ either.
- BenoitP 6y ago
  They're not the solution to well-sliced data processing, but I can't see how they would hurt the performance.

faitswulff 6y ago

Scipio has seen some action on HN before in "C++ vs Rust: an async Thread-per-Core story": https://news.ycombinator.com/item?id=24444347 https://news.ycombinator.com/item?id=24444347

Luker88 6y ago

I am a fan of the thread-per-core model, but I do not share their views on the data sharding per core While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load) Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great

mjb 6y ago
> While it increases the data locality, I have seen a few software following this sharding model (notably scylla) that work really bad once the load is not evenly distributed across all shards Serial access to hot keys is a hard thing to design around, and you're right that sharding doesn't solve that problem. Worse, it exposes other keys that just happen to share the shard (or the core) to poor performance. There are a couple of well-understood solutions to this problem. The obvious one is to dynamically re-balance shards based on heat, either moving some keys or split/merge. This is the same tactic that many distributed databases use, and while it's complex to do, it's easier to do locally than distributed. Another option is stochastic re-balancing, like the Stochastic Fairness Queuing (https://ieeexplore.ieee.org/document/91316 https://ieeexplore.ieee.org/document/91316) model used in networking. Here, shards are randomly re-shuffled occasionally. Doesn't fix the noisy neighbor problem, but does mean that the noisyness moves around. That might seem silly, but it's pretty much what's going to happen under the covers of the non-sharded version of the code when the scheduler gets involved, only easier to reason about. > When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load) Lower apparent performance for neighbors of the heavy hitter, sure. Under which other circumstances does it reduce performance? > Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great I'm a bit puzzled by this. Distributed systems have exactly the same problem, and solving that problem is much harder there because the cost of contention is higher, data movement is more expensive, and you have to deal with a lot more failure cases. The statistics may be better for distributed systems because a hot tenant has to be a lot hotter to make hot box than a hot core. But that's a very specific kind of bet, and if you end up with a tenant that does cause a hot box you have an even harder problem to solve. Dynamo (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos...) solves this by moving the key space around, as do many similar kinds of systems. It's not easy, though, and filled with caveats. If you're scared of sharding, distributed sharding should be scarier than on-box sharding.

shwestrick 6y ago

Thread-per-core has been around for a while! It's how a lot of modern multicore languages work. You've probably heard of "green" or "lightweight" threads before... it's all the same idea. Typically, you'd want to use a scheduler (probably work-stealing https://en.wikipedia.org/wiki/Work_stealing https://en.wikipedia.org/wiki/Work_stealing) under the hood to dynamically assign tasks to processors, which is much more effective at load-balancing than "sharding". All of these languages/libraries use a dynamic scheduler for load-balancing: * Rayon (Rust) [https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon] * Goroutines (Go) [https://golangbyexample.com/goroutines-golang/ https://golangbyexample.com/goroutines-golang/] * OpenMP [https://www.openmp.org/ https://www.openmp.org/] * Task Parallel Library (.NET) [https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl https://docs.microsoft.com/en-us/dotnet/standard/parallel-pr...] * Thread Building Blocks (C++) [https://software.intel.com/content/www/us/en/develop/tools/threading-building-blocks.html https://software.intel.com/content/www/us/en/develop/tools/t...] * Cilk (C/C++) [http://cilk.mit.edu/ http://cilk.mit.edu/] * Java Fork-Join and Parallel Streams [https://docs.oracle.com/javase/tutorial/collections/streams/parallelism.html https://docs.oracle.com/javase/tutorial/collections/streams/...] * ParlayLib (C++) [https://github.com/cmuparlay/parlaylib https://github.com/cmuparlay/parlaylib]

imtringued 6y ago
If it's not partitioning data per processing unit then it would not be considered a "thread-per-core" architecture based on the definition the article provided. Work stealing means more than one thread can be responsible for a single piece of data. This could result in crossing NUMA nodes or servers.
- shwestrick 6y ago
  Hmm I was separating the concept of "thread-per-core" from sharding. I would argue that typical cooperative task schedulers (e.g. work-stealing) get the performance benefits of thread-per-core without requiring any static partitioning. But if thread-per-core is fundamentally tied to the idea of sharding, then I think I see what you're saying.

lomkju 6y ago

This only makes sense if the local I/O is really fast and in many cases you contact external databases where the context switches are negligible to the I/O time spent on a query.

gpderetta 6y ago
Suru, but this is a good architecture for writing databases in the first place.