6 ms·

Out-of-memory victim selection with BPF

83 points by mfrw 3y ago

remram 3y ago
What are situations where you might want to use this? It is easy enough to set oom_score_adj from regular userspace code, the only time you would benefit from computing it dynamically in the moment is if that score changes rapidly. Does that happen?
- ExoticPearTree 3y ago
  One situation that I think is seen more often is where a child process or thread is consuming way too much memory (large dataset, memory leak etc.) and it would be beneficial to just kill that particular process/thread and not the parent.
  - remram 3y ago
    That's the default behavior of the OOM killer since the start.
- krab 3y ago
  One situation might be that you roughly know how much memory your process should be consuming and kill it if it's overconsuming. Something akin to but more dynamic than memory limits.
  - remram 3y ago
    That's probably something you would want to do as soon as the process is outside its limits, without waiting for a OOM situation.
    - mort96 3y ago
      Right, but you may not catch it before the OOM situation arises I guess?
    - dzaima 3y ago
      That could end up with significant false-positives when updating the thing without updating the estimated memory usage (especially if slow bounded fragmentation over a long period of time applies). You might also have some expected ratio of memory usage (e.g. process B uses 3x the memory of process A), but want to allow the absolute usage to grow as more data is processed.
      - remram 3y ago
        Having the wrong limits will cause the wrong thing to die, regardless of whether they kick in during OOM or orchestration. Better to find out during your planned rollout window if you ask me.
  - omoikane 3y ago
    I always used `ulimit -v` and `ulimit -H -v` for this purpose. Maybe this is more fine grained?
- j16sdiz 3y ago
  In Kubernetes world, you may want to kill the pods and keep all tasks from control panel (or whatever it is called).
  - j16sdiz 3y ago
    Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...
    - folmar 3y ago
      I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well
IshKebab 3y ago
I wonder if they'll ever make it possible to ask the user what they want to do...
- zokier 3y ago
  PSI and cgroups should make that possible for typical situations.
  - zokier 3y ago
    Indeed, found something in this vein https://github.com/hakavlad/nohang/issues/58 https://github.com/hakavlad/nohang/issues/58
- Karellen 3y ago
  Doesn't that kind of assume that a) there is only one user b) they are logged in c) they will be able to respond in a timely enough manner to allow other programs to allocate memory shortly edit: d) there will be enough memory to start a program that is responsible for asking the user, and if it needs to connect to a display server then the display server will be able to allocate memory for surfaces that program can draw to to display the request to the user.
  - IshKebab 3y ago
    Yes. You can make it work on desktop. Not really an option on the server of course. I was indirectly pointing out how Linux on the Desktop is never really going to happen for normal people while the kernel devs ignore obvious desktop requirements like this. Windows solved this problem over 20 years ago.
    - rcxdude 3y ago
      How has Windows solved the problem? If anything it's more irritating on Windows as it refuses to overcommit so in a lot of setups if you don't have a humongous swapfile you will get OOM errors on random processes before you've even filled up physical RAM.
totetsu 3y ago
Chrome does a good job of killing tabs when oom, but Firefox seems to just hang the whole system. Is there a way to make FF more murderous? edit: I found about:unloads which shows tabs memory usage and unload order
- vladvasiliu 3y ago
  > edit: I found about:unloads which shows tabs memory usage and unload order I've just taken a look at that and it's weird. It says all my tabs have been last accessed "right now". If I click refresh, the last access time updates to that moment. This is Firefox 116.0.3 on Linux. I wonder if this could be related to my using a single tab per window (I have a window manager whose job is to... manage windows).
  - Filligree 3y ago
    Definitely related. Any tab that’s currently visible has to remain loaded, and windows are considered visible even if they’re covered by other windows.
    - vladvasiliu 3y ago
      > windows are considered visible even if they’re covered by other windows This is interesting. I usually use firefox with i3 in tabbed mode, so only one window visible at a time. On some sites, when I switch to another tab that hides the previous one, any sound playing would stop. But if I display the two windows side by side, switching to the other window wouldn't stop the sound in the initial window. So I'm guessing that window visibility must be somehow communicated to the browser, if websites can act on it. Edit: this also happens with chromium in the same configuration: I have Teams running in PWA mode on the same desktop as Outlook, also in PWA mode, both tiled by the WM. Teams will get suspended after a while if it's not the selected tab, but it'll stay active if it's on top.
    - mcculley 3y ago
      Is there a good reason why the app would not track window visibility? I have only used the X API for this, but I would have assumed all GUIs have such an API.
refulgentis 3y ago
BPF? Berkley Packet Filter? 14 years on here and I'm kinda alarmed by HN lately, it's gone from people reading 20% of the time to 1%. article is inscrutable, and literally the entire set of 30 comments are: - Firefox memory usage. - why decide dynamically? - why not just ask the user?
Aleklart 3y ago
It is hilarious how leaking js in some ff window kills IDE first and then random processes inside containers and finally freezes desktop. I wonder if oom developers are sponsored by memory manufacturers because 16 gb not enough to open and keep sites you want in background.

83 points by mfrw 3y ago

What are situations where you might want to use this? It is easy enough to set oom_score_adj from regular userspace code, the only time you would benefit from computing it dynamically in the moment is if that score changes rapidly. Does that happen?

ExoticPearTree 3y ago
One situation that I think is seen more often is where a child process or thread is consuming way too much memory (large dataset, memory leak etc.) and it would be beneficial to just kill that particular process/thread and not the parent.
- remram 3y ago
  That's the default behavior of the OOM killer since the start.
krab 3y ago
One situation might be that you roughly know how much memory your process should be consuming and kill it if it's overconsuming. Something akin to but more dynamic than memory limits.
- remram 3y ago
  That's probably something you would want to do as soon as the process is outside its limits, without waiting for a OOM situation.
  - mort96 3y ago
    Right, but you may not catch it before the OOM situation arises I guess?
  - dzaima 3y ago
    That could end up with significant false-positives when updating the thing without updating the estimated memory usage (especially if slow bounded fragmentation over a long period of time applies). You might also have some expected ratio of memory usage (e.g. process B uses 3x the memory of process A), but want to allow the absolute usage to grow as more data is processed.
    - remram 3y ago
      Having the wrong limits will cause the wrong thing to die, regardless of whether they kick in during OOM or orchestration. Better to find out during your planned rollout window if you ask me.
- omoikane 3y ago
  I always used `ulimit -v` and `ulimit -H -v` for this purpose. Maybe this is more fine grained?
j16sdiz 3y ago
In Kubernetes world, you may want to kill the pods and keep all tasks from control panel (or whatever it is called).
- j16sdiz 3y ago
  Context: the patch author is from bytedance -- the Chinese company behind tiktok. I can imagine this can be useful in server farms if you can read Chinese, see also: https://twitter.com/Manjusaka_Lee/status/1692084961330028802/photo/1 https://twitter.com/Manjusaka_Lee/status/1692084961330028802...
  - folmar 3y ago
    I can't, but tesseract+deepl does wonders. Text from image, cut to abberviate: > 1. the current usage scenario of bytes for such a feature. I can I can think of is to avoid some critical processes like kubelet to be OOM to ensure better stability. I don't know if my understanding is I don't know if I'm wrong or not. Yes, on the one hand, we want to make sure that important daemons like kubelet are not killed by mistake. On the one hand, we want to make sure that critical daemons like kubelet don't get killed by mistake; on the other hand, nowadays, big companies tend to run multiple types of tasks together to improve resource utilization, and in the case of OOM, we would like to make sure that we have the right resources. And when it comes to OOM, we want to make decisions based on quality of service requirements, not on memory consumption. In the past, Ali and Google used to prioritize their tasks by setting the user's priority on the memcg priority setting by users to achieve this function, but BPF may be more flexible and applicable to more scenarios. > Another question I have is if we introduce BPF in OOM eviction scenarios. I have another question of my own: if we introduce BPF in an OOM eviction scenario > program (in effect introducing a BPF > program (in effect, introducing the runtime unpredictability of the additional programmability). time unpredictability due to the additional programmability), might it not be possible that a system stability-peddling program such as OOM could lead to a system stabilization scheme such as OOM not > working as expected, leading to more systemic problems. > work as expected, leading to further destabilization of the system. Yes. I haven't done the exact math on the extra overhead, but I'm optimistic. We can see that BPF has even done some work on kernel tuning. On the other hand, a user can easily write a kernel BPF to write a "bad" or even "malicious" BPF, which is really a matter of whether we trust the user completely. At the beginning of the implementation, I also did some backstabbing on the kernel side to make sure that at least one victim could be selected, but Michal probably thought this was unnecessary (https:/llore.kernel.org must kml/ZMzhDFhvol2VQBE4@dhcp22.suse.cz/). Tweet text as seen on Nitter: A patch from ByteDance that attempts to programmable the kernel's OOM eviction behavior via BPF. I find this interesting. @flaneur2023 Aug 17 This makes sense, adding an if else anyway don't kill this logic for OOM time would work pretty well NadeshikoManju @SwingingCamping S3 2024 On! @Manjusaka_Lee Aug 17 Replying to @flaneur2023 Further discussion between me and the author Aug 17, 2023 - 8:04 AM UTC [[image]] @ayanamist Aug 17 Replying to @Manjusaka_Lee @flaneur2023 It should be mainly a mixed section claim, it's a pain in the ass to adjust the oom scores by process as well

IshKebab 3y ago

I wonder if they'll ever make it possible to ask the user what they want to do...

zokier 3y ago
PSI and cgroups should make that possible for typical situations.
- zokier 3y ago
  Indeed, found something in this vein https://github.com/hakavlad/nohang/issues/58 https://github.com/hakavlad/nohang/issues/58
Karellen 3y ago
Doesn't that kind of assume that a) there is only one user b) they are logged in c) they will be able to respond in a timely enough manner to allow other programs to allocate memory shortly edit: d) there will be enough memory to start a program that is responsible for asking the user, and if it needs to connect to a display server then the display server will be able to allocate memory for surfaces that program can draw to to display the request to the user.
- IshKebab 3y ago
  Yes. You can make it work on desktop. Not really an option on the server of course. I was indirectly pointing out how Linux on the Desktop is never really going to happen for normal people while the kernel devs ignore obvious desktop requirements like this. Windows solved this problem over 20 years ago.
  - rcxdude 3y ago
    How has Windows solved the problem? If anything it's more irritating on Windows as it refuses to overcommit so in a lot of setups if you don't have a humongous swapfile you will get OOM errors on random processes before you've even filled up physical RAM.

totetsu 3y ago

Chrome does a good job of killing tabs when oom, but Firefox seems to just hang the whole system. Is there a way to make FF more murderous? edit: I found about:unloads which shows tabs memory usage and unload order

vladvasiliu 3y ago
> edit: I found about:unloads which shows tabs memory usage and unload order I've just taken a look at that and it's weird. It says all my tabs have been last accessed "right now". If I click refresh, the last access time updates to that moment. This is Firefox 116.0.3 on Linux. I wonder if this could be related to my using a single tab per window (I have a window manager whose job is to... manage windows).
- Filligree 3y ago
  Definitely related. Any tab that’s currently visible has to remain loaded, and windows are considered visible even if they’re covered by other windows.
  - vladvasiliu 3y ago
    > windows are considered visible even if they’re covered by other windows This is interesting. I usually use firefox with i3 in tabbed mode, so only one window visible at a time. On some sites, when I switch to another tab that hides the previous one, any sound playing would stop. But if I display the two windows side by side, switching to the other window wouldn't stop the sound in the initial window. So I'm guessing that window visibility must be somehow communicated to the browser, if websites can act on it. Edit: this also happens with chromium in the same configuration: I have Teams running in PWA mode on the same desktop as Outlook, also in PWA mode, both tiled by the WM. Teams will get suspended after a while if it's not the selected tab, but it'll stay active if it's on top.
  - mcculley 3y ago
    Is there a good reason why the app would not track window visibility? I have only used the X API for this, but I would have assumed all GUIs have such an API.

refulgentis 3y ago

BPF? Berkley Packet Filter? 14 years on here and I'm kinda alarmed by HN lately, it's gone from people reading 20% of the time to 1%. article is inscrutable, and literally the entire set of 30 comments are: - Firefox memory usage. - why decide dynamically? - why not just ask the user?

Aleklart 3y ago

It is hilarious how leaking js in some ff window kills IDE first and then random processes inside containers and finally freezes desktop. I wonder if oom developers are sponsored by memory manufacturers because 16 gb not enough to open and keep sites you want in background.