4 ms·

Thanks for this article. The article mentions Nodejs libuv which makes everything async but not multithreaded. I want to combine the benefits of kernel threads

Thanks for this article. The article mentions Nodejs libuv which makes everything async but not multithreaded. I want to combine the benefits of kernel threads with coroutines or goroutines/green threads/lightweight threads. (If anybody knows anything specifically about fibers, I'd appreciate you telling me that because I'm not familiar with them.) I have a lightweight thread scheduler https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread which is a 1:M:N scheduler (1 scheduler thread, M kernel threads, N lightweight threads) with the lightweight threads being multiplexed on the kernel threads. I am working on a multithreaded architecture which I currently call 3 tier multithreaded architecture for lack of a better name. It combines request parallelism with IO and CPU parallelism and intra request parallelism. We split kernel threads into three groups: app threads, which run lightweight threads, IO threads (liburing/epoll) and traditional CPU threadpool with work stealing. * There should be low latency for handling IO because of the liburing or epoll loop is dedicated to serving that request and then dispatching it for CPU heavy task or app level task. * The IO threads have buffers that other threads can write to to queue up data for sockets. * The IO threads are basically listeners for IO events, they either spawn a lightweight thread for running on the app threads or submit a CPU task. * The app threads coordinate parallelism WITHIN a request: they spawn CPU tasks and IO tasks. * You can also spawn app threads to do work outside the context of a request, such as server tasks or schedule. * You never put a while (true) in a CPU task and only use cooperatively preemptable loops in the lightweight app threads. * The IO threads do very little work. * Nobody should block the app threads because that would prevent other requests from completing. * Lockfree Ringbuffers are used by CPU threads and lightweight threads to communicate. I invite feedback and comments: https://github.com/samsquire/three-tier-multithreaded-architecture https://github.com/samsquire/three-tier-multithreaded-archit...

opportune 3y ago
> I want to combine the benefits of kernel threads with coroutines or goroutines/green threads/lightweight threads. (If anybody knows anything specifically about fibers, I'd appreciate you telling me that because I'm not familiar with them.) Fibers mean a couple different things these days. The basics are usually things like cooperative scheduling/different context switching and pre-emption semantics (using special synchronization libraries) that make it so you can have huge numbers of active thread-like (fibrous) things with less overhead. They also usually support a tree-like structure that enables things like cancellation. Sometimes they re-use p_threads (when one fiber completes, the next fiber gets the old p_thread) between fibers under the hood to amortize thread creation and destruction, or sometimes they pack multiple fibers into a single thread for further amortization and to reduce overhead. Your ``` while (this.parent.isScheduled(this)) { ``` in the thread function itself confuses me a bit as typically you'd expect the code to just run when scheduled, not run otherwise - are you keeping all your lightweight threads running/scheduled by the OS but bit-flipping the printing from off to on based on if your scheduler determines it to be active? If so, to get the actual performance benefits, you need to keep the lightweight threads off the kernel scheduler itself
- samsquire 3y ago
  Thanks so much for your reply and explanation of fibers. I still don't get it yet but I'll do some more investigation and ask ChatGPT. That code shall only run when the thread is scheduled, the guard is to prevent the lightweight thread blocking the kernel worker loop over lightweight threads. The lightweight threads run in worker threads and the scheduler is a different thread to that. You would have to remove a lightweight thread from kernel_thread.lightweight_threads to deschedule it completely. Presumably you'd do this if you were waiting for an event. I assume all my lightweight threads are ready to run and all want to run. But they need to relinquish the CPU when preempted by the scheduler. In the C code of preemptible-thread, the user_function() of a lightweight thread is the body of the lightweight thread, it is called by one of the kernel threads assigned to the lightweight thread runtime. The kernel thread loops over each lightweight threads assigned to that kernel thread. The scheduler thread does NOT run lightweight threads. It's just used to generate signals on a time interval. It sets flags to interrupt loops. In other words, a lightweight thread will only run until it is cooperatively preempted by the scheduler thread. I've been criticised for describing it as preemption but it's a cooperative preemption. while (this.parent.isScheduled(this)) is used to cancel the execution of that lightweight thread's execution of user_function when the scheduler toggles its scheduled & preempted flag. It's the equivalent of while (true) but in a lightweight thread context. If I used while (true), the lightweight thread would never be descheduled for another lightweight thread to run. It's a way of stopping the current thread's while loop. There is one scheduler kernel thread that does the following on each time slice (ideally a few microseconds): for kernel_thread in kernel_threads: for lightweight_thread in kernel_thread.lightweight_threads: lightweight_thread.preempt() In as many kernel thread worker thread as possible, as many as you have hardware threads. They runs lightweight threads one by one until the scheduler preempts that lightweight thread and then it moves to the next one: Kernel thread lightweight thread runner/worker. It round robins the lightweight threads. while (running) { for lightweight_thread in lightweight threads: lightweight_thread.markScheduled(true) #### between here and while (lightweight_thread.isScheduled(this)) { // while i'm not preempted registerLoop(0, 0, 10000000); for (int loopVar = initialVar(0, 0); getValue(0) < getLimit(0); loopVar = increment(0)) { Math.sqrt(getValue(0)); } #### here is the lightweight thread's body. } } I've inlined the lightweight thread's body in this code above. The kernel thread scheduler loop over lightweight threads, preempt all currently running lightweight threads. While the kernel worker thread loops over that thread's lightweight threads and loops while they are not preempted. Cancellation is possible in this design. I haven't thought about removing lightweight threads from the runqueue, it's assumed every lightweight thread wants to run as much as possible. The scheduler can be enhanced to provide this.

opportune 3y ago

> I want to combine the benefits of kernel threads with coroutines or goroutines/green threads/lightweight threads. (If anybody knows anything specifically about fibers, I'd appreciate you telling me that because I'm not familiar with them.) Fibers mean a couple different things these days. The basics are usually things like cooperative scheduling/different context switching and pre-emption semantics (using special synchronization libraries) that make it so you can have huge numbers of active thread-like (fibrous) things with less overhead. They also usually support a tree-like structure that enables things like cancellation. Sometimes they re-use p_threads (when one fiber completes, the next fiber gets the old p_thread) between fibers under the hood to amortize thread creation and destruction, or sometimes they pack multiple fibers into a single thread for further amortization and to reduce overhead. Your ``` while (this.parent.isScheduled(this)) { ``` in the thread function itself confuses me a bit as typically you'd expect the code to just run when scheduled, not run otherwise - are you keeping all your lightweight threads running/scheduled by the OS but bit-flipping the printing from off to on based on if your scheduler determines it to be active? If so, to get the actual performance benefits, you need to keep the lightweight threads off the kernel scheduler itself

samsquire 3y ago
Thanks so much for your reply and explanation of fibers. I still don't get it yet but I'll do some more investigation and ask ChatGPT. That code shall only run when the thread is scheduled, the guard is to prevent the lightweight thread blocking the kernel worker loop over lightweight threads. The lightweight threads run in worker threads and the scheduler is a different thread to that. You would have to remove a lightweight thread from kernel_thread.lightweight_threads to deschedule it completely. Presumably you'd do this if you were waiting for an event. I assume all my lightweight threads are ready to run and all want to run. But they need to relinquish the CPU when preempted by the scheduler. In the C code of preemptible-thread, the user_function() of a lightweight thread is the body of the lightweight thread, it is called by one of the kernel threads assigned to the lightweight thread runtime. The kernel thread loops over each lightweight threads assigned to that kernel thread. The scheduler thread does NOT run lightweight threads. It's just used to generate signals on a time interval. It sets flags to interrupt loops. In other words, a lightweight thread will only run until it is cooperatively preempted by the scheduler thread. I've been criticised for describing it as preemption but it's a cooperative preemption. while (this.parent.isScheduled(this)) is used to cancel the execution of that lightweight thread's execution of user_function when the scheduler toggles its scheduled & preempted flag. It's the equivalent of while (true) but in a lightweight thread context. If I used while (true), the lightweight thread would never be descheduled for another lightweight thread to run. It's a way of stopping the current thread's while loop. There is one scheduler kernel thread that does the following on each time slice (ideally a few microseconds): for kernel_thread in kernel_threads: for lightweight_thread in kernel_thread.lightweight_threads: lightweight_thread.preempt() In as many kernel thread worker thread as possible, as many as you have hardware threads. They runs lightweight threads one by one until the scheduler preempts that lightweight thread and then it moves to the next one: Kernel thread lightweight thread runner/worker. It round robins the lightweight threads. while (running) { for lightweight_thread in lightweight threads: lightweight_thread.markScheduled(true) #### between here and while (lightweight_thread.isScheduled(this)) { // while i'm not preempted registerLoop(0, 0, 10000000); for (int loopVar = initialVar(0, 0); getValue(0) < getLimit(0); loopVar = increment(0)) { Math.sqrt(getValue(0)); } #### here is the lightweight thread's body. } } I've inlined the lightweight thread's body in this code above. The kernel thread scheduler loop over lightweight threads, preempt all currently running lightweight threads. While the kernel worker thread loops over that thread's lightweight threads and loops while they are not preempted. Cancellation is possible in this design. I haven't thought about removing lightweight threads from the runqueue, it's assumed every lightweight thread wants to run as much as possible. The scheduler can be enhanced to provide this.