4 ms·

I think your intuitions are incorrect. 1. Preemptive multitasking is strictly better in almost all cases than cooperative multitasking, when you have more pro

I think your intuitions are incorrect. 1. Preemptive multitasking is strictly better in almost all cases than cooperative multitasking, when you have more processes, not fewer; especially when these processes are heterogenous. Throughput decreases over the best-case cooperative scenario, but jitter significantly decreases, and most importantly, you -- the user, the operator, or the programmer -- have a strong guarantee that the schedulers will never lock up because someone wrote code wrong. That guarantee alone is why every operating system abandoned a cooperative model and pushed schedulers as close as possible to preemptive multitasking. 2. Kernel scheduling is radically worse than user-space scheduling in nearly every scenario. The kernel schedulers are necessarily very generic and workload-agnostic; kernel threads and the thread management infrastructure are incredibly heavyweight and slow relative to user space; the context switching penalty is gigantic; processes relying on kernel scheduling are forced to compete with other processes on the system; and the list goes on. This is why unikernels, exokernels, and user-space tcp implementations like onload are growing prevalent, especially in the low-latency space.

pron 11y ago
> Preemptive multitasking is strictly better in almost all cases than cooperative multitasking, when you have more processes Sorry I didn't make myself clear. Quasar most certainly employs preemptive multitasking, just not time-slice- (or reduction-) based. > Kernel scheduling is radically worse than user-space scheduling Except when descheduling on time-slices. Remember two facts: 1/ your hardware can only support very few threads that require time-slice preemption so those task-switching costs are negligible, and 2/ no other thread is dependent on low-latency access to data produced a thread that is descheduled on a time-slice, so the latency doesn't matter much either. None of that is my intuition. I've been profiling and measuring this stuff for a few years now, and I'm a huge proponent of userspace scheduling. Adding time-slice preemption didn't make things any worse -- it just didn't make them any better either, so we took it out, and we tell users to use kernel scheduling for those threads. If the user makes a mistake, we recognize it at runtime and tell her that kernel scheduling might be better for that thread.
- felixgallo 11y ago
  From the Quasar documentation: "Runaway Fibers A fiber that is stuck in a loop without blocking, or is blocking the thread its running on (by directly or indirectly performing a thread-blocking operation) is called a runaway fiber. It is perfectly OK for fibers to do that sporadically (as the work stealing scheduler will deal with that), but doing so frequently may severely impact system performance (as most of the scheduler’s threads might be tied up by runaway fibers). Quasar detects runaway fibers, and notifies you about which fibers are problematic, whether they’re blocking the thread or hogging the CPU, and gives you their stack trace, by printing this information to the console as well as reporting it to the runtime fiber monitor (exposed through a JMX MBean; see the previous section)." (http://docs.paralleluniverse.co/quasar/ http://docs.paralleluniverse.co/quasar/) that describes a classic symptom of cooperative multitasking, and that kind of broken behavior doesn't happen in a preemptively multitasked system. Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed. additionally, when you say you are doing preemption without timeslicing, that doesn't really make sense, as all preemption requires slicing time: that's literally what it means. On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot. Fundamentally I'm sure Quasar has some great use cases, and probably shows some interesting performance gains over erlang in some of them, but this discussion has dimmed my interest significantly. Quite unfortunate.
  - pron 11y ago
    Let me repeat: Quasar has the ability to do time-slice scheduling just like Erlang. However, that has proven useless in practice because, unlike Erlang, we have access to kernel scheduling, so we turned that off (I think the relevant code is still there -- it's about ten lines). Switching off that feature was completely backed by evidence, and caused zero harm. > Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed. That is easily solvable by having Quasar automatically migrate a thread from being scheduled in user-space to being scheduled by the kernel. However, even that cool feature doesn't justify itself, because things work great as they are. The programmer doesn't need to check JMX occasionally. The behavior is immediately detected in testing and reported as a warning. Simple and effective. Note that all this works even if this behavior occurs in native code, which is more than Erlang does. Threads that are less suitable for user-mode scheduling must be few, and they are very easy to recognize. Doing any of that automatically may be cool, but has no real value. We might do that at some point, but it's not a high priority. > as all preemption requires slicing time: that's literally what it means. No, it means that the scheduler has the ability to deschedule the thread when it wishes[1]. We have this ability, we just no longer use it on time-slices (we easily can, it's just not helpful in the least). > On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot. All you say is true, yet none of it matters in practice when it comes to time-slice preemption. Only a handful of threads can enjoy that form of scheduling anyway, whether they take 10MB or 10 bytes of RAM. [1]: https://en.wikipedia.org/wiki/Preemption_(computing)#Preemptive_multitasking https://en.wikipedia.org/wiki/Preemption_(computing)#Preempt...

pron 11y ago

> Preemptive multitasking is strictly better in almost all cases than cooperative multitasking, when you have more processes Sorry I didn't make myself clear. Quasar most certainly employs preemptive multitasking, just not time-slice- (or reduction-) based. > Kernel scheduling is radically worse than user-space scheduling Except when descheduling on time-slices. Remember two facts: 1/ your hardware can only support very few threads that require time-slice preemption so those task-switching costs are negligible, and 2/ no other thread is dependent on low-latency access to data produced a thread that is descheduled on a time-slice, so the latency doesn't matter much either. None of that is my intuition. I've been profiling and measuring this stuff for a few years now, and I'm a huge proponent of userspace scheduling. Adding time-slice preemption didn't make things any worse -- it just didn't make them any better either, so we took it out, and we tell users to use kernel scheduling for those threads. If the user makes a mistake, we recognize it at runtime and tell her that kernel scheduling might be better for that thread.

felixgallo 11y ago
From the Quasar documentation: "Runaway Fibers A fiber that is stuck in a loop without blocking, or is blocking the thread its running on (by directly or indirectly performing a thread-blocking operation) is called a runaway fiber. It is perfectly OK for fibers to do that sporadically (as the work stealing scheduler will deal with that), but doing so frequently may severely impact system performance (as most of the scheduler’s threads might be tied up by runaway fibers). Quasar detects runaway fibers, and notifies you about which fibers are problematic, whether they’re blocking the thread or hogging the CPU, and gives you their stack trace, by printing this information to the console as well as reporting it to the runtime fiber monitor (exposed through a JMX MBean; see the previous section)." (http://docs.paralleluniverse.co/quasar/ http://docs.paralleluniverse.co/quasar/) that describes a classic symptom of cooperative multitasking, and that kind of broken behavior doesn't happen in a preemptively multitasked system. Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed. additionally, when you say you are doing preemption without timeslicing, that doesn't really make sense, as all preemption requires slicing time: that's literally what it means. On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot. Fundamentally I'm sure Quasar has some great use cases, and probably shows some interesting performance gains over erlang in some of them, but this discussion has dimmed my interest significantly. Quite unfortunate.
- pron 11y ago
  Let me repeat: Quasar has the ability to do time-slice scheduling just like Erlang. However, that has proven useless in practice because, unlike Erlang, we have access to kernel scheduling, so we turned that off (I think the relevant code is still there -- it's about ten lines). Switching off that feature was completely backed by evidence, and caused zero harm. > Certainly it violates the programmer intuition invariant that she can just write programs, if she has to occasionally check on a JMX MBean to find out if the framework has pooped on the bed. That is easily solvable by having Quasar automatically migrate a thread from being scheduled in user-space to being scheduled by the kernel. However, even that cool feature doesn't justify itself, because things work great as they are. The programmer doesn't need to check JMX occasionally. The behavior is immediately detected in testing and reported as a warning. Simple and effective. Note that all this works even if this behavior occurs in native code, which is more than Erlang does. Threads that are less suitable for user-mode scheduling must be few, and they are very easy to recognize. Doing any of that automatically may be cool, but has no real value. We might do that at some point, but it's not a high priority. > as all preemption requires slicing time: that's literally what it means. No, it means that the scheduler has the ability to deschedule the thread when it wishes[1]. We have this ability, we just no longer use it on time-slices (we easily can, it's just not helpful in the least). > On the kernel scheduling piece, your argument doesn't make any sense to me. Hardware support is only tangentially to do with high thread context switching costs; and it is frequently the case that many actors can fit into cache, where many threads cannot. All you say is true, yet none of it matters in practice when it comes to time-slice preemption. Only a handful of threads can enjoy that form of scheduling anyway, whether they take 10MB or 10 bytes of RAM. [1]: https://en.wikipedia.org/wiki/Preemption_(computing)#Preemptive_multitasking https://en.wikipedia.org/wiki/Preemption_(computing)#Preempt...