3 ms·

I've seen energy-aware scheduling, literately decades of effort that culminated on the EEVDF scheduler so that it was possible to have a good scheduler that wor

I've seen energy-aware scheduling, literately decades of effort that culminated on the EEVDF scheduler so that it was possible to have a good scheduler that worked well on desktops, servers and HPC... and, between all those efforts, a giant parallel one to prevent or influence to OOM-Killer to behave better. I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. I mean, it doesn't happen anymore, but some years ago it was relatively simple to soft-crash a system just by trying to open a file that was significantly larger than the physical RAM. By 'soft-crashing' I mean the system became so slow that it was faster to reboot than wait for it to recover by itself. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system? Is there a reason why memory-aware scheduling was never explored or am I wrong and it was explored and proved not good?

toast0 1y ago
> I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system? This approach is hard to make work, because once the system is in memory shortage, mostly all processes will be slowing the system. There's already a penalty for accessing memory that's not currently paged in --- the process will be descheduled pending the i/o, and other processes can run during that time ... until they access memory that's not paged in. You can easily get into a situation where most of your cpu time is spend in paging and no useful work gets done. This can happen even without swap; the paging will just happen on memory mapped files, even if you're not using mmap for data files, your executables and libraries are mmaped, so the system will page those out and in in an effort to manage the memory shortage. To make a system easier to operate, I like to run with a small swap partition and monitor swap usage both in % and by rate. You can often get a small window of a still responsive system to try to identify the rogue process and kill it without having to restart the whole thing. A small partition means a big problem will quickly hit the OOM killer without being in swap hell for ages. There might be research or practice from commercial unix and mainframe where multi-tenancy is more common? What I've seen on the free software side is mostly avoiding the issue or trying to addressing it with policy limits on memory usage. Probably more thorough memory accounting is a necessary step to doing a better job, but adding more ram when you run into problems is effective mitigation, so....
Asooka 1y ago
For batch jobs I would really like a scheduler that will pause and fully swap out processes until memory is available again. For example, when compiling a C++ project, some source files or some link steps will require vast amounts of memory. In that case you would want to swap out all the other currently running compiler processes so the memory hungry one can do its job, then swap them back in. I don't want to punish the memory hungry process, actually I want exactly the opposite - I want everything else to get out of its way. The build system will eventually finish running processes that take up a lot of memory and will continue the ones that require little memory.

toast0 1y ago

> I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system? This approach is hard to make work, because once the system is in memory shortage, mostly all processes will be slowing the system. There's already a penalty for accessing memory that's not currently paged in --- the process will be descheduled pending the i/o, and other processes can run during that time ... until they access memory that's not paged in. You can easily get into a situation where most of your cpu time is spend in paging and no useful work gets done. This can happen even without swap; the paging will just happen on memory mapped files, even if you're not using mmap for data files, your executables and libraries are mmaped, so the system will page those out and in in an effort to manage the memory shortage. To make a system easier to operate, I like to run with a small swap partition and monitor swap usage both in % and by rate. You can often get a small window of a still responsive system to try to identify the rogue process and kill it without having to restart the whole thing. A small partition means a big problem will quickly hit the OOM killer without being in swap hell for ages. There might be research or practice from commercial unix and mainframe where multi-tenancy is more common? What I've seen on the free software side is mostly avoiding the issue or trying to addressing it with policy limits on memory usage. Probably more thorough memory accounting is a necessary step to doing a better job, but adding more ram when you run into problems is effective mitigation, so....

Asooka 1y ago

For batch jobs I would really like a scheduler that will pause and fully swap out processes until memory is available again. For example, when compiling a C++ project, some source files or some link steps will require vast amounts of memory. In that case you would want to swap out all the other currently running compiler processes so the memory hungry one can do its job, then swap them back in. I don't want to punish the memory hungry process, actually I want exactly the opposite - I want everything else to get out of its way. The build system will eventually finish running processes that take up a lot of memory and will continue the ones that require little memory.