4 ms·

What are situations where you might want to use this? It is easy enough to set oom_score_adj from regular userspace code, the only time you would benefit from c

by remram 3y ago

What are situations where you might want to use this? It is easy enough to set oom_score_adj from regular userspace code, the only time you would benefit from computing it dynamically in the moment is if that score changes rapidly. Does that happen?

ExoticPearTree 3y ago
One situation that I think is seen more often is where a child process or thread is consuming way too much memory (large dataset, memory leak etc.) and it would be beneficial to just kill that particular process/thread and not the parent.
- remram 3y ago
  That's the default behavior of the OOM killer since the start.
krab 3y ago
One situation might be that you roughly know how much memory your process should be consuming and kill it if it's overconsuming. Something akin to but more dynamic than memory limits.
- remram 3y ago
  That's probably something you would want to do as soon as the process is outside its limits, without waiting for a OOM situation.
  - mort96 3y ago
    Right, but you may not catch it before the OOM situation arises I guess?
  - dzaima 3y ago
    That could end up with significant false-positives when updating the thing without updating the estimated memory usage (especially if slow bounded fragmentation over a long period of time applies). You might also have some expected ratio of memory usage (e.g. process B uses 3x the memory of process A), but want to allow the absolute usage to grow as more data is processed.
    - remram 3y ago
      Having the wrong limits will cause the wrong thing to die, regardless of whether they kick in during OOM or orchestration. Better to find out during your planned rollout window if you ask me.
- omoikane 3y ago
  I always used `ulimit -v` and `ulimit -H -v` for this purpose. Maybe this is more fine grained?
j16sdiz 3y ago
In Kubernetes world, you may want to kill the pods and keep all tasks from control panel (or whatever it is called).
- j16sdiz 3y ago
  Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...
  - folmar 3y ago
    I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well

by remram 3y ago

ExoticPearTree 3y ago

One situation that I think is seen more often is where a child process or thread is consuming way too much memory (large dataset, memory leak etc.) and it would be beneficial to just kill that particular process/thread and not the parent.

remram 3y ago
That's the default behavior of the OOM killer since the start.

krab 3y ago

One situation might be that you roughly know how much memory your process should be consuming and kill it if it's overconsuming. Something akin to but more dynamic than memory limits.

remram 3y ago
That's probably something you would want to do as soon as the process is outside its limits, without waiting for a OOM situation.
- mort96 3y ago
  Right, but you may not catch it before the OOM situation arises I guess?
- dzaima 3y ago
  That could end up with significant false-positives when updating the thing without updating the estimated memory usage (especially if slow bounded fragmentation over a long period of time applies). You might also have some expected ratio of memory usage (e.g. process B uses 3x the memory of process A), but want to allow the absolute usage to grow as more data is processed.
  - remram 3y ago
    Having the wrong limits will cause the wrong thing to die, regardless of whether they kick in during OOM or orchestration. Better to find out during your planned rollout window if you ask me.
omoikane 3y ago
I always used `ulimit -v` and `ulimit -H -v` for this purpose. Maybe this is more fine grained?

j16sdiz 3y ago

In Kubernetes world, you may want to kill the pods and keep all tasks from control panel (or whatever it is called).

j16sdiz 3y ago
Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...
- folmar 3y ago
  I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well