9 ms·

Non-uniform memory access meets the OOM killer

109 points by r4um 9y ago

saagarjha 9y ago
A lot of time seems to go into tricking the watchdogs on single purpose machines. I heard a story once of a guy who wanted to get some computation done, but the process was being deproritized by the scheduler because it seemed like it was a hung process that kept asking for CPU time. The solution he came up with was voluntarily relinquishing compute accesd right before anyone would would check up on it, making it appear as if the process was great at sharing time with others. By doing this, he could get that one process’s instructions running something like 99% of the time.
- mehrdadn 9y ago
  > The solution he came up with was voluntarily relinquishing compute accesd right before anyone would would check up on it I can't parse this; could you clarify? What exactly do you mean by "relinquish compute access right before anyone would check up on it"?
  - saagarjha 9y ago
    I can’t figure out whether it was me who wrote the words “compute access” or whether it was meant be something else and was autocorrected into that. Either way, what I meant was that the scheduler would normally come around at some fixed time, say after 1 ms, and task switch to something else if it found that the original process was still executing. Instead of letting this happen, he paused execution at 0.99 ms and told the scheduler he was done with his work. Then, he immediately resumed the process, which caused the scheduler to return control back to him.
    - mehrdadn 9y ago
      Ohh huh wow, I'm surprised the scheduler would fall for that!
    - jacquesm 9y ago
      That's a pretty bad scheduler then because normally it would have moved that task to the tail of the 'ready' list for that particular priority level rather than to the front because it yielded voluntarily.
      - adrianmonk 9y ago
        IIRC there are schedulers that dynamically adjust priority based on recent CPU usage. When a task is removed from the CPU due to its time quantum running out (as opposed to, say, making a blocking system call), that is taken as a sign that the task is relatively CPU-intensive, and its priority is reduced a little. The idea is to give IO-bound tasks a little more priority in comparison. IO-bound tasks generally run faster if they can get another IO operation started ASAP after the last one completes, and since they're not going to hog the CPU anyway, giving them relatively higher priority is one way to do this. Anyway, the point is that the trick works by gaming the system of priority levels. If you're at the tail of one priority level, you may be off than at the head of another.
- the8472 9y ago
  That's a pretty odd workaround considering that you can reserve cores to the point where not even kernel tasks run on them and then pin a single userspace thread to that core so it can run without ever being preemptively descheduled
  - chatmasta 9y ago
    For anyone like me who hadn't heard of this possibility before, I'll save you a few seconds of googling: https://stackoverflow.com/questions/13583146/whole-one-core-dedicated-to-single-process https://stackoverflow.com/questions/13583146/whole-one-core-...
  - dward 9y ago
    This wouldn't even work with the completely fair scheduler, which is the Linux default scheduler. https://en.m.wikipedia.org/wiki/Completely_Fair_Scheduler https://en.m.wikipedia.org/wiki/Completely_Fair_Scheduler
    - the8472 9y ago
      CFS obeys core isolation and task sets, so it would work
      - dward 9y ago
        I was speaking of the "odd workaround", not about using cpu isolation.
- tormeh 9y ago
  Setting niceness to minimum wasn't enough?
dis-sys 9y ago
Being able to write NUMA aware applications like the one described in the article is a luxury for ALL Go users. The current Go runtime doesn't have any NUMA awareness. As of today, you can get a two NUMA nodes processor (AMD threadripper 1900X) for as little as $449.
ben_bai 9y ago
What happened to good old returning NULL when no memory is available? No let's do overcommit (malloc always works) and OOM-kill some random process when under memory pressure!
- JeremyBanks 9y ago
  We collectively decided that this is less annoying that introducing thousands of difficult error cases to handle in every application.
  - IgorPartola 9y ago
    How is it difficult to handle? void* xmalloc(size_t s) { void *p; if (!(p = malloc(s)) { panic(“OOM”); } return p; } If your process ran out of RAM, you get to quit. Why offload it on some other random process? This is how your database process runs out of memory, and your web workers get killed (or vice versa). In either case your system isn’t usable, but one is harder to debug. In fact having the above in stdlib would have been just as effective at fixing thousands of little bugs and would have collectively saved us all thousands if not more man hours of developing the OOM killer and then raging online about it.
    - jacquesm 9y ago
      Malloc may very well succeed when there is not enough free memory in the system. You'll only know it when you try to write to memory and the system runs out of real memory to back your virtual memory. This can happen much later in time. Memory allocated by malloc is not necessarily there by the time you need it. Another issue is that a fork will simply copy the task state and set all the pages to 'copy on write' in the clone. That way the fork can execute very rapidly and only when the child process starts writing pages is there some actual memory overhead.
    - geofft 9y ago
      Because it's not necessarily your process that ran out of RAM. Imagine you have background process A which has allocated 1 GB of RAM just in case and never used it, and then process B tries to allocate maybe 100 MB, and the kernel says "No, I can't do that." Then process B quits. Is this really a better outcome than killing process A? On my current laptop, pulseaudio has 2.7 GB virtual memory and 17 MB physical memory. If I open a new Firefox tab, I would much prefer for pulseaudio to die than for Firefox to die. (You might say, "Well, pulseaudio shouldn't allocate 2.7 GB virtual memory," to which I'd say, "Sure, if you'd rather spend the thousands of hours fixing pulseaudio and every other program like it, go for it.") Also, "panic" isn't a standard C function - are you envisioning a panic that unwinds or one that doesn't? If it unwinds, what happens if you need to allocate memory during the unwind to save state? If it doesn't, why is it a better state of the world for Firefox to crash periodically and require me to recover my work, without even giving it the option to synchronize its state?
mdekkers 9y ago
someone ships a new version of some low-level support program, and it goes to all of your machines at once, and they start out-of-memory-killing your memory storage stuff. In a matter of minutes, your entire service is dead and people start running around trying to figure it out. Testing is sooooo oldfashioned. /s Did you know you can force your memory allocation request(s) to only be served up by a given NUMA node? Did you then know that if that node happens to be nearly full, the kernel's OOM killer will wake up and start looking for something to assassinate? Yes, I knew that, being an old-fashioned IT professional and all that. If you don't know about these things, and you blindly deploy patches to prod and then "run around trying to figure it out", you should step away from the terminal, and you have no business looking after a "really big system with hundreds of gigs of memory." I yearn for the good old days - when you were assessed to be ready to look after a system - by your peers. Today, everyone with google is a fucking expert, and everything breaks. Now get off my damn lawn.
ParrotyError 9y ago
The OOM killer was a misfeature when it was designed. Why is it still in the kernel? Solaris solved this problem 20 years ago.
- wilun 9y ago
  Overcommit is desirable because of the fork/exec model of creating processes; if a big one wants to spawn children, without overcommit you need to be able to recommit its VM size, regardless of how lightweight the desired child is. Sadly there is no good (& simple) solution under most Unixes (that's a problem that only affects long running and big processes, but those are often the main applications of some embedded systems). I'm not really found of vfork now that we use threads more often (well, at least on Linux it seems other threads are not suspended, but it seems that this is not a Posix guarantee)... posix_spawn could be a solution if implemented with some dedicated kernel support (or always implemented through well-behaved existing ones).
  - ams6110 9y ago
    > if a big one wants to spawn children Should this be considered poor behavior or design? It's been quite a while since I wrote explicit fork/ exec code, but wouldn't a better approach be to have a small master process that spawns off the necessary children and then either links them up or mediates communication? I mean, on a unix-like system, init is ultimately the spawner of everything else and it's not a particularly large process.
    - wilun 9y ago
      Yes on Unix it can be interesting, if needing fork/exec to spawn children, to pre spawn an helper child process that will do just that. But that would just be because the fork/exec model is insane: with a posix_spawn (or CreateProcess) centric approach, you don't have to add that extra layer, and you have very few drawbacks. Also, that will not help you when trying to use libraries that attempt to spawn on their own. As for init being small, I leave you the responsibility of your words, now that we live in a systemd world :p
- 9y ago
StreamBright 9y ago
This is the reason i am big fan of running any software with separate users and setting ulimit to a low value so that something stupid like this cannot impact the production service. I would be super keen to try to replicate this scenario on my test cluster and see if my settings catching it. Does anybody know if the software in question is an opensource tool?
- ams6110 9y ago
  This is the approach I take also. I'm also looking at totally disabling the OOM killer because it seems to be pretty useless. Anytime I see stuff killed by OOM the culprit is usually and obviously some runaway Java process, but OOM inevitably picks the SSH daemon to kill, which doesn't help anything, and the box continues to swap so badly that it just seems unrecoverable. I'd rather just have the box panic and reboot if it's truly out of memory.
  - yjftsjthsd-h 9y ago
    I have not looked into it at all, but can you not exempt sshd from the OOM killer?
    - ams6110 9y ago
      I looked into it a little bit. There are ways to tune it but I didn't see a way to exempt processes by name. It may be possible. The scenario I described above is HPC clusters in a university environment. The problem is students running programs that are poorly written. I'd rather reboot the node and tell them to fix their code than deal with trying to accommodate their careless / naive programming.
adrianmonk 9y ago
The OOM uses a heuristic to figure out what to kill. If the primary purpose of your system is to run some process that hangs on to a lot of RAM, that heuristic is exactly the opposite of what you need, so it would a good idea to disable it or exempt that process. Also, while I'm talking prophylactics, if you have (which you should) monitoring and alerting in your production environment, it seems like there should be an alert for whenever the OOM killer activates. Assuming you are allocating resources carefully enough that you expect everything to fit, if it fires, it's almost always a sign that things are not going according to plan and need to be investigated sooner rather than later.
- greenleafjacob 9y ago
  Yes, the OOM killer activation is on its face evidence of an incident; comments on other threads saying “If your process ran out of RAM, you get to quit. Why offload it on some other random process? This is how your database process runs out of memory, and your web workers get killed (or vice versa).” The scenario depicts a capacity shortage regardless of what the system decides to do in response.
n_t 9y ago
That's why one needs to be aware of memory and other load characteristics of system, particularly if it is an enterprise system. Various process should be put in different cgroups with defined resources. cgroups also provides memory pressure notification and other goodies too. If it is an embedded system, probably it is best to turn off overcommit. Finally, for critical processes, use oom.priority so that process can be excluded from being killed.
amelius 9y ago
Reminds me of: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
jschwartzi 9y ago
At my last job I wrote a build system that build maybe 30 or 40 executables from several hundred source files. Sometimes when I'd run make -j with no constraint my desktop environment would crash. It turned out that the OOM killer was triggering because I was filling up memory with compiler invocations. I was really proud of that bug.
- bmurphy1976 9y ago
  I seem to end up building a lot of stuff on memory constrained devices for some reason. The OOM killer is always a problem, but it's easily avoided by provisioning an excessive amount of swap. It's slow, but slow is faster than never. Did you have any swap at the time?
  - jschwartzi 9y ago
    Yeah, probably. It was just a desktop Ubuntu system.
- lathiat 9y ago
  This might sound really obvious but you know you can do "make -j12" or some other number, right?
  - jschwartzi 9y ago
    Yeah, I do. I just found it hilarious that my build system could take down my window manager.
BrainInAJar 9y ago
Memory overcommit is the most hostile, idiotic misfeature to ever ship in any mainstream operating system. It's such a great example of why one should pay absolutely no concern to Linus
smarks 9y ago
Time to re-up this classic from Andries Brouwer: https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
deleted 9y ago
[deleted]
cthalupa 9y ago
>This new version also had this wacky little "feature" where it tried to bind itself to a single NUMA node. This is 100% a feature. If you care at all about memory access latency, you want to remain local to the NUMA node. Foreign memory access is significantly slower. If you have NUMA enabled and your applications are not NUMA aware, and there are shared pages being access by applications running on both nodes, the NUMA rebalancing can actually cause even worse performance as it constantly moves the pages from one node to the other. Any application that cares about memory access latency should 100% be written to be NUMA aware, and if it is not, you should be using numactl to bind the application to the proper node. This also goes for PCI-E devices (including nvme drives!) as they are going to be bound to a NUMA node as well. If you have an application that is accessing an nvme volume, or using a GPU, you should 100% make sure that it is running on the same node as the pci-e bus for that device.
speedplane 9y ago
It's not commonplace for even medium size companies to run dozens of servers. Memory resources (as well as disk and CPU) are always being stretched. The OOM may have been sufficient for single server environments, where you could always provision an extra 40%, but it's far too blunt of a tool. Most environments I've worked with have to define an instance size (in memory and CPU), and determine how many parallel threads/processes will run on it. Plus you need to determine when and how to scale up to more instances. To reduce costs, the goal is to 100% utilization, but also with the capability deal with spikes in traffic an workload, and all with an acceptable error rate. Unfortunately, doing this type sizing/scaling analysis is incredibly difficult. The opaque effects of the OOM make it even more difficult. I'm sure the OOM uses a deterministic algorithm, but it's complex enough that most don't know it, or handle for it. In a server environment, if the OOM kills a service, your app and all other services are likely hosed. It would be far more preferable if the OOM had a straightforward, consistent, and deterministic method to dealing with low memory. This way programmers would know to look out for it, and could handle it more consistently.

109 points by r4um 9y ago

A lot of time seems to go into tricking the watchdogs on single purpose machines. I heard a story once of a guy who wanted to get some computation done, but the process was being deproritized by the scheduler because it seemed like it was a hung process that kept asking for CPU time. The solution he came up with was voluntarily relinquishing compute accesd right before anyone would would check up on it, making it appear as if the process was great at sharing time with others. By doing this, he could get that one process’s instructions running something like 99% of the time.

mehrdadn 9y ago
> The solution he came up with was voluntarily relinquishing compute accesd right before anyone would would check up on it I can't parse this; could you clarify? What exactly do you mean by "relinquish compute access right before anyone would check up on it"?
- saagarjha 9y ago
  I can’t figure out whether it was me who wrote the words “compute access” or whether it was meant be something else and was autocorrected into that. Either way, what I meant was that the scheduler would normally come around at some fixed time, say after 1 ms, and task switch to something else if it found that the original process was still executing. Instead of letting this happen, he paused execution at 0.99 ms and told the scheduler he was done with his work. Then, he immediately resumed the process, which caused the scheduler to return control back to him.
  - mehrdadn 9y ago
    Ohh huh wow, I'm surprised the scheduler would fall for that!
  - jacquesm 9y ago
    That's a pretty bad scheduler then because normally it would have moved that task to the tail of the 'ready' list for that particular priority level rather than to the front because it yielded voluntarily.
    - adrianmonk 9y ago
      IIRC there are schedulers that dynamically adjust priority based on recent CPU usage. When a task is removed from the CPU due to its time quantum running out (as opposed to, say, making a blocking system call), that is taken as a sign that the task is relatively CPU-intensive, and its priority is reduced a little. The idea is to give IO-bound tasks a little more priority in comparison. IO-bound tasks generally run faster if they can get another IO operation started ASAP after the last one completes, and since they're not going to hog the CPU anyway, giving them relatively higher priority is one way to do this. Anyway, the point is that the trick works by gaming the system of priority levels. If you're at the tail of one priority level, you may be off than at the head of another.
the8472 9y ago
That's a pretty odd workaround considering that you can reserve cores to the point where not even kernel tasks run on them and then pin a single userspace thread to that core so it can run without ever being preemptively descheduled
- chatmasta 9y ago
  For anyone like me who hadn't heard of this possibility before, I'll save you a few seconds of googling: https://stackoverflow.com/questions/13583146/whole-one-core-dedicated-to-single-process https://stackoverflow.com/questions/13583146/whole-one-core-...
- dward 9y ago
  This wouldn't even work with the completely fair scheduler, which is the Linux default scheduler. https://en.m.wikipedia.org/wiki/Completely_Fair_Scheduler https://en.m.wikipedia.org/wiki/Completely_Fair_Scheduler
  - the8472 9y ago
    CFS obeys core isolation and task sets, so it would work
    - dward 9y ago
      I was speaking of the "odd workaround", not about using cpu isolation.
tormeh 9y ago
Setting niceness to minimum wasn't enough?

dis-sys 9y ago

Being able to write NUMA aware applications like the one described in the article is a luxury for ALL Go users. The current Go runtime doesn't have any NUMA awareness. As of today, you can get a two NUMA nodes processor (AMD threadripper 1900X) for as little as $449.

ben_bai 9y ago

What happened to good old returning NULL when no memory is available? No let's do overcommit (malloc always works) and OOM-kill some random process when under memory pressure!

JeremyBanks 9y ago
We collectively decided that this is less annoying that introducing thousands of difficult error cases to handle in every application.
- IgorPartola 9y ago
  How is it difficult to handle? void* xmalloc(size_t s) { void *p; if (!(p = malloc(s)) { panic(“OOM”); } return p; } If your process ran out of RAM, you get to quit. Why offload it on some other random process? This is how your database process runs out of memory, and your web workers get killed (or vice versa). In either case your system isn’t usable, but one is harder to debug. In fact having the above in stdlib would have been just as effective at fixing thousands of little bugs and would have collectively saved us all thousands if not more man hours of developing the OOM killer and then raging online about it.
  - jacquesm 9y ago
    Malloc may very well succeed when there is not enough free memory in the system. You'll only know it when you try to write to memory and the system runs out of real memory to back your virtual memory. This can happen much later in time. Memory allocated by malloc is not necessarily there by the time you need it. Another issue is that a fork will simply copy the task state and set all the pages to 'copy on write' in the clone. That way the fork can execute very rapidly and only when the child process starts writing pages is there some actual memory overhead.
  - geofft 9y ago
    Because it's not necessarily your process that ran out of RAM. Imagine you have background process A which has allocated 1 GB of RAM just in case and never used it, and then process B tries to allocate maybe 100 MB, and the kernel says "No, I can't do that." Then process B quits. Is this really a better outcome than killing process A? On my current laptop, pulseaudio has 2.7 GB virtual memory and 17 MB physical memory. If I open a new Firefox tab, I would much prefer for pulseaudio to die than for Firefox to die. (You might say, "Well, pulseaudio shouldn't allocate 2.7 GB virtual memory," to which I'd say, "Sure, if you'd rather spend the thousands of hours fixing pulseaudio and every other program like it, go for it.") Also, "panic" isn't a standard C function - are you envisioning a panic that unwinds or one that doesn't? If it unwinds, what happens if you need to allocate memory during the unwind to save state? If it doesn't, why is it a better state of the world for Firefox to crash periodically and require me to recover my work, without even giving it the option to synchronize its state?

mdekkers 9y ago

someone ships a new version of some low-level support program, and it goes to all of your machines at once, and they start out-of-memory-killing your memory storage stuff. In a matter of minutes, your entire service is dead and people start running around trying to figure it out. Testing is sooooo oldfashioned. /s Did you know you can force your memory allocation request(s) to only be served up by a given NUMA node? Did you then know that if that node happens to be nearly full, the kernel's OOM killer will wake up and start looking for something to assassinate? Yes, I knew that, being an old-fashioned IT professional and all that. If you don't know about these things, and you blindly deploy patches to prod and then "run around trying to figure it out", you should step away from the terminal, and you have no business looking after a "really big system with hundreds of gigs of memory." I yearn for the good old days - when you were assessed to be ready to look after a system - by your peers. Today, everyone with google is a fucking expert, and everything breaks. Now get off my damn lawn.

ParrotyError 9y ago

The OOM killer was a misfeature when it was designed. Why is it still in the kernel? Solaris solved this problem 20 years ago.

wilun 9y ago
Overcommit is desirable because of the fork/exec model of creating processes; if a big one wants to spawn children, without overcommit you need to be able to recommit its VM size, regardless of how lightweight the desired child is. Sadly there is no good (& simple) solution under most Unixes (that's a problem that only affects long running and big processes, but those are often the main applications of some embedded systems). I'm not really found of vfork now that we use threads more often (well, at least on Linux it seems other threads are not suspended, but it seems that this is not a Posix guarantee)... posix_spawn could be a solution if implemented with some dedicated kernel support (or always implemented through well-behaved existing ones).
- ams6110 9y ago
  > if a big one wants to spawn children Should this be considered poor behavior or design? It's been quite a while since I wrote explicit fork/ exec code, but wouldn't a better approach be to have a small master process that spawns off the necessary children and then either links them up or mediates communication? I mean, on a unix-like system, init is ultimately the spawner of everything else and it's not a particularly large process.
  - wilun 9y ago
    Yes on Unix it can be interesting, if needing fork/exec to spawn children, to pre spawn an helper child process that will do just that. But that would just be because the fork/exec model is insane: with a posix_spawn (or CreateProcess) centric approach, you don't have to add that extra layer, and you have very few drawbacks. Also, that will not help you when trying to use libraries that attempt to spawn on their own. As for init being small, I leave you the responsibility of your words, now that we live in a systemd world :p
9y ago

StreamBright 9y ago

This is the reason i am big fan of running any software with separate users and setting ulimit to a low value so that something stupid like this cannot impact the production service. I would be super keen to try to replicate this scenario on my test cluster and see if my settings catching it. Does anybody know if the software in question is an opensource tool?

ams6110 9y ago
This is the approach I take also. I'm also looking at totally disabling the OOM killer because it seems to be pretty useless. Anytime I see stuff killed by OOM the culprit is usually and obviously some runaway Java process, but OOM inevitably picks the SSH daemon to kill, which doesn't help anything, and the box continues to swap so badly that it just seems unrecoverable. I'd rather just have the box panic and reboot if it's truly out of memory.
- yjftsjthsd-h 9y ago
  I have not looked into it at all, but can you not exempt sshd from the OOM killer?
  - ams6110 9y ago
    I looked into it a little bit. There are ways to tune it but I didn't see a way to exempt processes by name. It may be possible. The scenario I described above is HPC clusters in a university environment. The problem is students running programs that are poorly written. I'd rather reboot the node and tell them to fix their code than deal with trying to accommodate their careless / naive programming.

adrianmonk 9y ago

The OOM uses a heuristic to figure out what to kill. If the primary purpose of your system is to run some process that hangs on to a lot of RAM, that heuristic is exactly the opposite of what you need, so it would a good idea to disable it or exempt that process. Also, while I'm talking prophylactics, if you have (which you should) monitoring and alerting in your production environment, it seems like there should be an alert for whenever the OOM killer activates. Assuming you are allocating resources carefully enough that you expect everything to fit, if it fires, it's almost always a sign that things are not going according to plan and need to be investigated sooner rather than later.

greenleafjacob 9y ago
Yes, the OOM killer activation is on its face evidence of an incident; comments on other threads saying “If your process ran out of RAM, you get to quit. Why offload it on some other random process? This is how your database process runs out of memory, and your web workers get killed (or vice versa).” The scenario depicts a capacity shortage regardless of what the system decides to do in response.

n_t 9y ago

That's why one needs to be aware of memory and other load characteristics of system, particularly if it is an enterprise system. Various process should be put in different cgroups with defined resources. cgroups also provides memory pressure notification and other goodies too. If it is an embedded system, probably it is best to turn off overcommit. Finally, for critical processes, use oom.priority so that process can be excluded from being killed.

amelius 9y ago

Reminds me of: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

jschwartzi 9y ago

At my last job I wrote a build system that build maybe 30 or 40 executables from several hundred source files. Sometimes when I'd run make -j with no constraint my desktop environment would crash. It turned out that the OOM killer was triggering because I was filling up memory with compiler invocations. I was really proud of that bug.

bmurphy1976 9y ago
I seem to end up building a lot of stuff on memory constrained devices for some reason. The OOM killer is always a problem, but it's easily avoided by provisioning an excessive amount of swap. It's slow, but slow is faster than never. Did you have any swap at the time?
- jschwartzi 9y ago
  Yeah, probably. It was just a desktop Ubuntu system.
lathiat 9y ago
This might sound really obvious but you know you can do "make -j12" or some other number, right?
- jschwartzi 9y ago
  Yeah, I do. I just found it hilarious that my build system could take down my window manager.

BrainInAJar 9y ago

Memory overcommit is the most hostile, idiotic misfeature to ever ship in any mainstream operating system. It's such a great example of why one should pay absolutely no concern to Linus

smarks 9y ago

Time to re-up this classic from Andries Brouwer: https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/

deleted 9y ago

[deleted]

cthalupa 9y ago

>This new version also had this wacky little "feature" where it tried to bind itself to a single NUMA node. This is 100% a feature. If you care at all about memory access latency, you want to remain local to the NUMA node. Foreign memory access is significantly slower. If you have NUMA enabled and your applications are not NUMA aware, and there are shared pages being access by applications running on both nodes, the NUMA rebalancing can actually cause even worse performance as it constantly moves the pages from one node to the other. Any application that cares about memory access latency should 100% be written to be NUMA aware, and if it is not, you should be using numactl to bind the application to the proper node. This also goes for PCI-E devices (including nvme drives!) as they are going to be bound to a NUMA node as well. If you have an application that is accessing an nvme volume, or using a GPU, you should 100% make sure that it is running on the same node as the pci-e bus for that device.

speedplane 9y ago

It's not commonplace for even medium size companies to run dozens of servers. Memory resources (as well as disk and CPU) are always being stretched. The OOM may have been sufficient for single server environments, where you could always provision an extra 40%, but it's far too blunt of a tool. Most environments I've worked with have to define an instance size (in memory and CPU), and determine how many parallel threads/processes will run on it. Plus you need to determine when and how to scale up to more instances. To reduce costs, the goal is to 100% utilization, but also with the capability deal with spikes in traffic an workload, and all with an acceptable error rate. Unfortunately, doing this type sizing/scaling analysis is incredibly difficult. The opaque effects of the OOM make it even more difficult. I'm sure the OOM uses a deterministic algorithm, but it's complex enough that most don't know it, or handle for it. In a server environment, if the OOM kills a service, your app and all other services are likely hosed. It would be far more preferable if the OOM had a straightforward, consistent, and deterministic method to dealing with low memory. This way programmers would know to look out for it, and could handle it more consistently.