4 ms·

If you have never designed a thread-per-core architecture and want some gritty inside color on what designing these systems is like, this is the article for you

If you have never designed a thread-per-core architecture and want some gritty inside color on what designing these systems is like, this is the article for you. I’ve designed thread-per-core architectures for 15 years now and this captures the zeitgeist perfectly. As for why you might want to care about “thread-per-core architectures”, the performance metrics are qualitatively better than any other software architecture you can use, it isn’t even close. These architectures were first developed in the supercomputing/HPC world but the principles apply to most software. An issue is that there are few libraries that are designed or optimized for thread-per-core architectures. Storage management is a reusable concept in these architectures but there is a lack of competent and scalable libraries that do this optimally. For someone coming into the space, you have to learn how to design e.g. high-performance storage allocators that can keep up with a JBOD of NVMe. I have mature libraries like this but they are tightly coupled to the rest of the software I design. When I first designed these architectures I sharded up resources across cores but there are many cases where this requires giving up a lot of performance. The problem of shedding load and resources across cores is really interesting. An implication is that some structures should be shared globally but you really want the performance to be as close to contention-free as possible or you defeat the purpose of thread-per-core. This is possible and there are design heuristics that approximate it but it isn’t discussed much. As mentioned in the article, no matter how you build these things your software will require a sophisticated understanding of lifetimes. Sometimes the builtin tools of the language will help with this, other times they can’t and you will have to build your own tools. I have never used Seastar. Not because I am not familiar with it but because it makes tradeoffs that are not appropriate for the workloads I tend to support. It is a good choice for many workloads. The only thing this indicates is that we are still in the early days of these types of architectures. Even if you don’t care or need to improve your architectural efficiency, these types of architectures significantly reduce the carbon/cost footprint of software systems.

kosolam 3y ago
Hello! Please tell me what kind of stuff you are doing that requires this and what are the different tradeoffs that seastar makes? I used seastar to build a software switch for SDNs some years ago…
FridgeSeal 3y ago
> want some gritty inside color on what designing these systems is like, I’ve only dabbled with them on hobby projects, but I absolutely love the idea. I’ve been on the hunt for more content about this, but I have trouble finding much, I suspect a lot of content is a bit “if you know you know” sort of insider knowledge stuff, so if you know of any other good content lurking out there I’d be so keen to know. > this is the article for you. I’ve designed thread-per-core architectures for 15 years now How’d you get into this/what do you do for work? As I said, I’ve dabbled with it on personal projects, but at least on everything I do for work it’s not always a good fit, or it is, but it’s too weird for my teammates.
- bitcharmer 3y ago
  > I suspect a lot of content is a bit “if you know you know” sort of insider knowledge stuff It is, the kind of knowledge that takes decades to accumulate you won't see on people's blog posts. This is why ultra low-latency and high-performance systems engineers can ask more for their services. > How’d you get into this/what do you do for work I'm not the OP but for HFT shops thread-per-core design is the only right way. Prop trading, big market makers etc, we all go to great lengths to isolate the workloads as much as possible.
- mgerdts 3y ago
  You may also want to look at SPDK and DPDK.
ddorian43 3y ago
What do you think of the thread-pipeline architecture that claims to be better than thread-per-core https://docs.rondb.com/design_thread_pipeline/ https://docs.rondb.com/design_thread_pipeline/ ? (creator of NDB Cluster)
samsquire 3y ago
I'm a hobbyist in this space. I have never used Seastar either, But I'm trying to design an "evented assembly" for efficiency in a thread-per-core architecture. I really want the next generation NodeJS equivalent to take advantage of all these learnings and learnings from Erlang. I think nodejs showed how awesome event loops and async can be and simple. The TFA (The Friendly Article) makes it clear there are sharp edges with coroutines and lambdas in C++: - something stored in a temporary between a yield barrier will be destroyed by the time the coroutine is resumed - you have to make sure you store things inside the coroutine promise object - lambda capturing rule complexity associated with copying and moving semantics I am just learning but I've written a multithreaded barrier in C and Java that is mutex free. In 6×2 thread pairs it can communicate between threads in 42 nanoseconds. I have the beginnings of a JIT compiler. But there's so much work to do. My barrier resembles a "phaser" pattern in a loop and bulk synchronous parallel.
infinite8s 3y ago
What sort of tradeoffs do you need to deal with that make Seastar inappropriate for your workloads?

kosolam 3y ago

Hello! Please tell me what kind of stuff you are doing that requires this and what are the different tradeoffs that seastar makes? I used seastar to build a software switch for SDNs some years ago…

FridgeSeal 3y ago

> want some gritty inside color on what designing these systems is like, I’ve only dabbled with them on hobby projects, but I absolutely love the idea. I’ve been on the hunt for more content about this, but I have trouble finding much, I suspect a lot of content is a bit “if you know you know” sort of insider knowledge stuff, so if you know of any other good content lurking out there I’d be so keen to know. > this is the article for you. I’ve designed thread-per-core architectures for 15 years now How’d you get into this/what do you do for work? As I said, I’ve dabbled with it on personal projects, but at least on everything I do for work it’s not always a good fit, or it is, but it’s too weird for my teammates.

bitcharmer 3y ago
> I suspect a lot of content is a bit “if you know you know” sort of insider knowledge stuff It is, the kind of knowledge that takes decades to accumulate you won't see on people's blog posts. This is why ultra low-latency and high-performance systems engineers can ask more for their services. > How’d you get into this/what do you do for work I'm not the OP but for HFT shops thread-per-core design is the only right way. Prop trading, big market makers etc, we all go to great lengths to isolate the workloads as much as possible.
mgerdts 3y ago
You may also want to look at SPDK and DPDK.

ddorian43 3y ago

What do you think of the thread-pipeline architecture that claims to be better than thread-per-core https://docs.rondb.com/design_thread_pipeline/ https://docs.rondb.com/design_thread_pipeline/ ? (creator of NDB Cluster)

samsquire 3y ago

I'm a hobbyist in this space. I have never used Seastar either, But I'm trying to design an "evented assembly" for efficiency in a thread-per-core architecture. I really want the next generation NodeJS equivalent to take advantage of all these learnings and learnings from Erlang. I think nodejs showed how awesome event loops and async can be and simple. The TFA (The Friendly Article) makes it clear there are sharp edges with coroutines and lambdas in C++: - something stored in a temporary between a yield barrier will be destroyed by the time the coroutine is resumed - you have to make sure you store things inside the coroutine promise object - lambda capturing rule complexity associated with copying and moving semantics I am just learning but I've written a multithreaded barrier in C and Java that is mutex free. In 6×2 thread pairs it can communicate between threads in 42 nanoseconds. I have the beginnings of a JIT compiler. But there's so much work to do. My barrier resembles a "phaser" pattern in a loop and bulk synchronous parallel.

infinite8s 3y ago

What sort of tradeoffs do you need to deal with that make Seastar inappropriate for your workloads?