10 ms·

Core scheduling lands in Linux 5.14

375 points by phab 5y ago

OldHand2018 5y ago
The first link in the linked article mentions Linus saying (in 2019) that performance needs to be better than simply running with SMT disabled. There is no mention of performance though. How is it? Presumably it is better!! And if so, this sounds like a concept that the OpenBSD community would be interested in since they prefer SMT disabled for security reasons.
- Arcuru 5y ago
  Yea, I'm curious what they even mean by 'performance' in this context. It would obviously impact whole-system performance, since this setting would intentionally limit what can run on each core leading to some idle cycles. That's expected though, and is merely a consequence of using this feature for security. I'm curious if it measurably slows down the scheduler and impacts performance from there.
ww520 5y ago
That's very fast in adding the feature to kernel to address the loss of performance due to Sprectre. Assigning OS threads of the same process to the hyperthreads in the same core is a good thing anyway. The threads probably share many data in the process and can benefit from the shared cache in the core.
tpxl 5y ago
Is there not a way to pin processes to cores in linux already, and/or why it cannot be used to achieve a similar thing (pin user1 to cores 0,1 and user2 to cores 2,3)?
- dharmab 5y ago
  Core pinning exists but it requires the administator to manually assign processes to specific cores. Core scheduling lets you group trusted processes together and then the OS can figure out which cores can run which processes dynamically. Also, core scheduling does not permit userspace attackers to "game the system" and target a specific core they want to attack.
  - tpxl 5y ago
    Well can't software do that automatically? I would love a piece of software that could reserve a couple of cores for games, another couple of cores for firefox, etc. dynamically based upon some settings.
    - adrian_b 5y ago
      On a computer that you control completely, in Linux you can use taskset to launch programs on whichever cores you want and to migrate already running programs to whichever cores you like and there is no obstacle that prevents you to use this in some scripts to automatically implement any policies you desire. I frequently use taskset, because otherwise Linux migrates continuously the process between cores, which can degrade the performance of the programs that do some computations with a long duration, unlike the programs that are mostly waiting for events to happen. This new feature has a different purpose, it is intended for multiuser servers, to enable the secure partitioning of the runnable threads into groups that can be scheduled on different cores, so that they will not be able to interfere with each other, even when they would have intended that.
    - flatiron 5y ago
      Security problems probably. I could only imagine the new spectre vulns if you got to pick your own core.
    - formerly_proven 5y ago
      Sure, you could use cgroups for that.
- alecco 5y ago
  A process running httpd httpd is pinned to CPU X. And it forks right away 10 worker sub-processes. Are all sub-processes supposed to run on CPU X?
  - adrian_b 5y ago
    In Linux, the CPU affinity (i.e. the list of cores on which a thread may be run) is inherited by any child process or thread, from its parent. Nevertheless, the CPU affinity of any process or thread can be changed at any time with sched_setaffinity() so the httpd process itself could run its children on different cores, if it wanted so, but it is unlikely that any httpd program does this. The new feature that is discussed in this thread will limit the cores on which the threads of a user can be scheduled to those having the same cookie, so a process will no longer be able to reschedule its children to run on the same core as the threads of another user.
  - tpxl 5y ago
    Yes? If CPU X is the only trusted CPU, or you are only trusted to run your software on that CPU, then everything should run on that core. If you rent 12 of 128 cores, then it may run on any of those 12 cores.
kevin_thibedeau 5y ago
int prctl(int option, unsigned long arg2, unsigned long arg3, unsigned long arg4, unsigned long arg5); Is there a reason why a more idiomatic void * to struct wasn't used for the args?
- caf 5y ago
  It's a leaky abstraction over the underlying system call mechanism. Because the user stack lives in user space and accessing it from the kernel requires the same dance as any other access to user memory, system call arguments are instead passed in registers where they are immediately available to the kernel with no possibility of faults or TOCTTOU holes. Here 'unsigned long' is just a convenient stand-in for "generic register-sized argument". A "void * pointing to struct" requires a copy from user space. If a particular prctl() does need a struct, it can certainly stuff a pointer into one of those 'unsigned long' parameters (in the kernel environment, a pointer can be converted to and from an unsigned long without concern).
  - dataflow 5y ago
    > Here 'unsigned long' is just a convenient stand-in for "generic register-sized argument". That wouldn't be register-sized on x86 would it?
    - oynqr 5y ago
      The minimum size for unsigned long is 4 bytes
      - dataflow 5y ago
        Oh I didn't realize its size varies on x86 and x64... I thought it's always 64-bit under compilers that target Linux! Today I learned...
        jamesfinlayson 5y ago
        I think the size can vary by compiler rather than just by architecture.
jeff_vader 5y ago
Random question: does anyone use recent mainline kernels with Ubuntu 20.04? How? How's the experience? I'm currently on Ubuntu HWE line, but that only goes to 5.11.0. Ubuntu kernel devs have debs for [mainline][1], but I'm not finding any good feedback/experience stories about these. [1]: https://kernel.ubuntu.com/~kernel-ppa/mainline/ https://kernel.ubuntu.com/~kernel-ppa/mainline/
earthscienceman 5y ago
This is such a great post, I'm surprised it doesn't have more traction but maybe people aren't into reading longform plain text. "spamming Internet users at scale requires a lot of parallel activity, after all. If those processes can be segregated so that all siblings of any given core run processes from the same customer, we can be spared the gruesome prospect of one spammer stealing another's target list — or somebody else's private keys."
- jacquesm 5y ago
  Technical writing with a sense of humor is great. Highly recommended: the user manual of the SkyRC MC3000 Li-Ion charger.
  - HanaShiratori 5y ago
    Haha yeah the good old MC3000. I love this charger, it makes a simple task a little more exciting and sometimes (a bit) unnecessarily complicated - but at the least you've got full control about charging those damn batteries!
  - wazoox 5y ago
    I still remember the fun I had 30 years ago reading the t.c. electronics 2290 digital delay user manual.
  - Fnoord 5y ago
    Link to PDFs [1] (scroll down, EN and DE it seems). Seems like an amazing device, too. Too bad it could only charge one at a time, so I guess you'd buy multiple of these on a tech dept. [1] https://www.skyrc.com/MC3000_Charger https://www.skyrc.com/MC3000_Charger
    - bayindirh 5y ago
      > Too bad it could only charge one at a time, so I guess you'd buy multiple of these on a tech dept. The phone app screenshots show three charge one discharge cycle at the same time. It's a 4 channel charger which can work even with a single battery. Currently I use IKEA's battery chargers*, but if I gonna need something more advanced, gonna look at this one. *: IKEA's battery chargers are no slouches either. They're very intelligent with battery fault detection and very good charging characteristics.
po1nt 5y ago
Everything is cool, except for the naming. At first I thought this was about implementation of Intel's new architecture.
jimmyed 5y ago
> When one sibling is executing, the other must wait. SMT is useful because CPUs often go idle while waiting for events — usually the arrival of data from memory. While one CPU waits, the other can be executing Is this accurate? I was under the impression SMT gains are not from running other threads when one is blocked (preemption is a old feature) but the processor having a multi stage pipeline so that the net number of instructions that are executed per cycle is more than 1 (closer to 2 in the above example)
- zaxcellent 5y ago
  What you're describing sounds like a superscalar processor which utilizes instruction level parallelization: https://en.wikipedia.org/wiki/Superscalar_processor https://en.wikipedia.org/wiki/Superscalar_processor
- willis936 5y ago
  As someone else said, this is superscalar, though these days companies refer to the number of concurrent out-of-order pipelines as Execution Units (EUs). SMT is a second set of registers that uses the same execution hardware. If a thread stalls waiting for a main memory read then the other thread can jump in after the pipeline is cleared out (or maybe before if you're clever and careful, but idk if this is done in practice). You can also impose some governance over time sharing so threads with low data memory footprints don't lock out other threads.
- Symmetry 5y ago
  No, it's not. SMT is specifically when you're running two threads simultaneously on a superscalar processor that can execute multiple instructions in a single cycle. During SMT one thread might be using execution reasources the other thread doesn't need further increasing performance beyond just having something to do during a cache miss. What the article describes is temporal multithreading where only one thread executes at a time. https://en.wikipedia.org/wiki/Multithreading_(computer_architecture) https://en.wikipedia.org/wiki/Multithreading_(computer_archi...
exabrial 5y ago
> While one might argue that cloud-computing providers are usually grumpy anyway, there is still value in anything that might improve their mood. I'm dead haha. This is great.
656565656565 5y ago
I’m surprised this has made into the mainline kernel, it feels like it is a niche use case. Perhaps cloud provider pressure?

375 points by phab 5y ago

OldHand2018 5y ago

The first link in the linked article mentions Linus saying (in 2019) that performance needs to be better than simply running with SMT disabled. There is no mention of performance though. How is it? Presumably it is better!! And if so, this sounds like a concept that the OpenBSD community would be interested in since they prefer SMT disabled for security reasons.

Arcuru 5y ago
Yea, I'm curious what they even mean by 'performance' in this context. It would obviously impact whole-system performance, since this setting would intentionally limit what can run on each core leading to some idle cycles. That's expected though, and is merely a consequence of using this feature for security. I'm curious if it measurably slows down the scheduler and impacts performance from there.

ww520 5y ago

That's very fast in adding the feature to kernel to address the loss of performance due to Sprectre. Assigning OS threads of the same process to the hyperthreads in the same core is a good thing anyway. The threads probably share many data in the process and can benefit from the shared cache in the core.

tpxl 5y ago

Is there not a way to pin processes to cores in linux already, and/or why it cannot be used to achieve a similar thing (pin user1 to cores 0,1 and user2 to cores 2,3)?

dharmab 5y ago
Core pinning exists but it requires the administator to manually assign processes to specific cores. Core scheduling lets you group trusted processes together and then the OS can figure out which cores can run which processes dynamically. Also, core scheduling does not permit userspace attackers to "game the system" and target a specific core they want to attack.
- tpxl 5y ago
  Well can't software do that automatically? I would love a piece of software that could reserve a couple of cores for games, another couple of cores for firefox, etc. dynamically based upon some settings.
  - adrian_b 5y ago
    On a computer that you control completely, in Linux you can use taskset to launch programs on whichever cores you want and to migrate already running programs to whichever cores you like and there is no obstacle that prevents you to use this in some scripts to automatically implement any policies you desire. I frequently use taskset, because otherwise Linux migrates continuously the process between cores, which can degrade the performance of the programs that do some computations with a long duration, unlike the programs that are mostly waiting for events to happen. This new feature has a different purpose, it is intended for multiuser servers, to enable the secure partitioning of the runnable threads into groups that can be scheduled on different cores, so that they will not be able to interfere with each other, even when they would have intended that.
  - flatiron 5y ago
    Security problems probably. I could only imagine the new spectre vulns if you got to pick your own core.
  - formerly_proven 5y ago
    Sure, you could use cgroups for that.
alecco 5y ago
A process running httpd httpd is pinned to CPU X. And it forks right away 10 worker sub-processes. Are all sub-processes supposed to run on CPU X?
- adrian_b 5y ago
  In Linux, the CPU affinity (i.e. the list of cores on which a thread may be run) is inherited by any child process or thread, from its parent. Nevertheless, the CPU affinity of any process or thread can be changed at any time with sched_setaffinity() so the httpd process itself could run its children on different cores, if it wanted so, but it is unlikely that any httpd program does this. The new feature that is discussed in this thread will limit the cores on which the threads of a user can be scheduled to those having the same cookie, so a process will no longer be able to reschedule its children to run on the same core as the threads of another user.
- tpxl 5y ago
  Yes? If CPU X is the only trusted CPU, or you are only trusted to run your software on that CPU, then everything should run on that core. If you rent 12 of 128 cores, then it may run on any of those 12 cores.

kevin_thibedeau 5y ago

int prctl(int option, unsigned long arg2, unsigned long arg3, unsigned long arg4, unsigned long arg5); Is there a reason why a more idiomatic void * to struct wasn't used for the args?

caf 5y ago
It's a leaky abstraction over the underlying system call mechanism. Because the user stack lives in user space and accessing it from the kernel requires the same dance as any other access to user memory, system call arguments are instead passed in registers where they are immediately available to the kernel with no possibility of faults or TOCTTOU holes. Here 'unsigned long' is just a convenient stand-in for "generic register-sized argument". A "void * pointing to struct" requires a copy from user space. If a particular prctl() does need a struct, it can certainly stuff a pointer into one of those 'unsigned long' parameters (in the kernel environment, a pointer can be converted to and from an unsigned long without concern).
- dataflow 5y ago
  > Here 'unsigned long' is just a convenient stand-in for "generic register-sized argument". That wouldn't be register-sized on x86 would it?
  - oynqr 5y ago
    The minimum size for unsigned long is 4 bytes
    - dataflow 5y ago
      Oh I didn't realize its size varies on x86 and x64... I thought it's always 64-bit under compilers that target Linux! Today I learned...
      - jamesfinlayson 5y ago
        I think the size can vary by compiler rather than just by architecture.

jeff_vader 5y ago

Random question: does anyone use recent mainline kernels with Ubuntu 20.04? How? How's the experience? I'm currently on Ubuntu HWE line, but that only goes to 5.11.0. Ubuntu kernel devs have debs for [mainline][1], but I'm not finding any good feedback/experience stories about these. [1]: https://kernel.ubuntu.com/~kernel-ppa/mainline/ https://kernel.ubuntu.com/~kernel-ppa/mainline/

earthscienceman 5y ago

This is such a great post, I'm surprised it doesn't have more traction but maybe people aren't into reading longform plain text. "spamming Internet users at scale requires a lot of parallel activity, after all. If those processes can be segregated so that all siblings of any given core run processes from the same customer, we can be spared the gruesome prospect of one spammer stealing another's target list — or somebody else's private keys."

jacquesm 5y ago
Technical writing with a sense of humor is great. Highly recommended: the user manual of the SkyRC MC3000 Li-Ion charger.
- HanaShiratori 5y ago
  Haha yeah the good old MC3000. I love this charger, it makes a simple task a little more exciting and sometimes (a bit) unnecessarily complicated - but at the least you've got full control about charging those damn batteries!
- wazoox 5y ago
  I still remember the fun I had 30 years ago reading the t.c. electronics 2290 digital delay user manual.
- Fnoord 5y ago
  Link to PDFs [1] (scroll down, EN and DE it seems). Seems like an amazing device, too. Too bad it could only charge one at a time, so I guess you'd buy multiple of these on a tech dept. [1] https://www.skyrc.com/MC3000_Charger https://www.skyrc.com/MC3000_Charger
  - bayindirh 5y ago
    > Too bad it could only charge one at a time, so I guess you'd buy multiple of these on a tech dept. The phone app screenshots show three charge one discharge cycle at the same time. It's a 4 channel charger which can work even with a single battery. Currently I use IKEA's battery chargers*, but if I gonna need something more advanced, gonna look at this one. *: IKEA's battery chargers are no slouches either. They're very intelligent with battery fault detection and very good charging characteristics.

po1nt 5y ago

Everything is cool, except for the naming. At first I thought this was about implementation of Intel's new architecture.

jimmyed 5y ago

> When one sibling is executing, the other must wait. SMT is useful because CPUs often go idle while waiting for events — usually the arrival of data from memory. While one CPU waits, the other can be executing Is this accurate? I was under the impression SMT gains are not from running other threads when one is blocked (preemption is a old feature) but the processor having a multi stage pipeline so that the net number of instructions that are executed per cycle is more than 1 (closer to 2 in the above example)

zaxcellent 5y ago
What you're describing sounds like a superscalar processor which utilizes instruction level parallelization: https://en.wikipedia.org/wiki/Superscalar_processor https://en.wikipedia.org/wiki/Superscalar_processor
willis936 5y ago
As someone else said, this is superscalar, though these days companies refer to the number of concurrent out-of-order pipelines as Execution Units (EUs). SMT is a second set of registers that uses the same execution hardware. If a thread stalls waiting for a main memory read then the other thread can jump in after the pipeline is cleared out (or maybe before if you're clever and careful, but idk if this is done in practice). You can also impose some governance over time sharing so threads with low data memory footprints don't lock out other threads.
Symmetry 5y ago
No, it's not. SMT is specifically when you're running two threads simultaneously on a superscalar processor that can execute multiple instructions in a single cycle. During SMT one thread might be using execution reasources the other thread doesn't need further increasing performance beyond just having something to do during a cache miss. What the article describes is temporal multithreading where only one thread executes at a time. https://en.wikipedia.org/wiki/Multithreading_(computer_architecture) https://en.wikipedia.org/wiki/Multithreading_(computer_archi...

exabrial 5y ago

> While one might argue that cloud-computing providers are usually grumpy anyway, there is still value in anything that might improve their mood. I'm dead haha. This is great.

656565656565 5y ago

I’m surprised this has made into the mainline kernel, it feels like it is a niche use case. Perhaps cloud provider pressure?