4 ms·

In Kubernetes world, you may want to kill the pods and keep all tasks from control panel (or whatever it is called).

j16sdiz 3y ago
Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...
- folmar 3y ago
  I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well
diarrhea 3y ago
Isn’t the control pane and everything else running on physically separate nodes for that reason already? (By default)
jeffbee 3y ago
That explanation doesn't really scan. It is the individual control groups that OOM. This new logic is for selecting a victim task in that control group. Processes in other pods, even in the same k8s namespace, would not be considered (or even visible, depending on the container runtime).
remram 3y ago
oom_score_adj is really good at that already.

Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...

folmar 3y ago
I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well

diarrhea 3y ago

Isn’t the control pane and everything else running on physically separate nodes for that reason already? (By default)

jeffbee 3y ago

That explanation doesn't really scan. It is the individual control groups that OOM. This new logic is for selecting a victim task in that control group. Processes in other pods, even in the same k8s namespace, would not be considered (or even visible, depending on the container runtime).

remram 3y ago

oom_score_adj is really good at that already.