7 ms·
Memstop: Use LD_PRELOAD to delay process execution when low on memory
- d00mB0t 1y agoOOM eat your heart out :D This is great, but there are security implication when using LD_PRELOAD--but I like it! More programs like this for parallel computing please.
- josephcsible 1y ago> there are security implication when using LD_PRELOAD What do you mean?
- d00mB0t 1y agoA malicious user could inject memstop.so into a critical system service and delay execution--Writing a wrapper script would work, along with keeping unprivileged users from using LD_PRELOAD.
- josephcsible 1y agoIf a malicious user can control the environment of critical system services, you're already pwned. There's no actual security issue there and no value in such a wrapper script.
- d00mB0t 1y agoYou sure about that? :)
- josephcsible 1y agoCan you give a counterexample?
- coherentpony 1y agoYou are the person that made the initial claim. The burden of proof is on you, not someone else.
- josephcsible 1y agoHuh? You claimed there was a security vulnerability and I disagreed and asked for an example of it.
- giingyui 1y ago[dead]
- pjc50 1y agoIf it's a critical service running as root, there's no way you're allowed to inject stuff into it. That's already a far bigger security vulnerability. (I don't get the wrapper script suggestion, wrap what?)
- mpyne 1y agoIn addition, the link-loader already should ignore LD_PRELOAD for setuid binaries so even if you're not root but running a setuid binary, LD_PRELOAD can't help you (and if it can it's a security flaw with the link-loader).
- deleted 1y ago[deleted]
- masfuerte 1y agoI'd be more worried about the possibility of deadlock.
- rini17 1y agoNo information about design philosophy, whether it triggers on RSS or virtual memory. And I'd think adding swap would be recommended as a place to stow away these stopped processes? Naive approach might end with deadlock when all processs that could free up memory are stopped.
- otterley 1y agoNot only that, but it’s inherently racy (TOCTOU); a process could allocate a huge block of memory between the time the determination to allow the controlled program to start and the time that program is able to finish initializing is made. A better solution is to use a proper memory-reservation scheduler, not hacks like this. Kubernetes has such a thing.
- menaerus 1y agoIt's pretty naive, it cat's from /proc/mem and extracts MemTotal and MemAvailable: https://github.com/surban/memstop/blob/master/memstop.c#L40-L43 https://github.com/surban/memstop/blob/master/memstop.c#L40-.... In other words, that's not how virtual memory works. And I actually don't know how I would go and solve this problem because of the way commit, overcommit, and paging algorithms can intervene with the goal.
- AnotherGoodName 1y agoOn Android something similar; If you have an app that absolutely needs a lot of memory (transcoding HD videos in my case), before running it; - Spawn a new process and catch the unix signals for that process, specifically SIGINT. - Have that new process allocate all the memory you need in your actual process. - The OS will kill via it's Low Memory Killer (LMK) daemon. - Since you caught the SIGINT in the parent process you can avoid the Android "Application X is using too much memory" notification that pops up to the user making it a silent failure. - Lastly sleep a few MS to let Androids LMK kill other apps. Now run your intensive memory process. This works for games too. A huge hack but needed for some of the lower end devices that don't have GB of RAM. Also needed since Java apps will just eat memory until something tells them to clean up, you need some overallocation to fail to trigger the whole memory free process on Android.
- somethingsome 1y agoOh please, say more, it seems very interesting. I didn't really get the last two bullet points
- mmastrac 1y agoI think they are describing a "balloon" - you allocate enough memory to trigger a low-memory condition. Android then tries to kill off any large apps. You can then allocate large amounts of RAM in your own app after giving Android some time to shut down background apps, etc.
- zero_shift 1y agoThis reminds me of those "memory freer" apps they used to distribute for Windows XP. Did they actually do anything useful? Or just force Windows to write everything to the pagefile?
- swinglock 1y agoThat could perhaps be a useful if you had just enough memory to run a game, if not for other processes using it too. In that case background services should be written to swap while running the game which will probably cause stutters. Doing it before starting the game instead would be better, if it has to be done. So such a utility might then be useful in some cases when someone couldn't afford enough RAM.
- ape4 1y agoThis could be a nice systemd unit option.
- arianvanp 1y agoThere is already memory.high (MemoryHigh= in systemd) you can set on cgroups which does something similar at the kernel level. But its challenging to use correctly. As its easy to end up in a live lock situation where the process never frees memory but also never gets killed. See all the details that Kubernetes had with introducing this https://github.com/kubernetes/enhancements/issues/2570 https://github.com/kubernetes/enhancements/issues/2570
- meatmanek 1y agoI assumed it paused the program while it's running, by e.g. intercepting malloc calls or something, but no it just delays the startup. I'm wondering what the value of this using LD_PRELOAD is, rather than just being a wrapper command that takes the command to execute as arguments. I guess it's easier to inject into a preexisting build system because it's all configured via environment variables?
- cryptonector 1y agoOne would think it would be better to do interpose on `execve()` or even `fork()` rather than to simply do the waiting in an ELF `.init` section. After all, if the parent is spawning lots of child processes then that is a problem in itself. But yeah, this approach will presumably work most of the time. Incidentally, as pure-Go and pure-Rust programs proliferate, `LD_PRELOAD` stops being useful.
- LegionMammal978 1y agoLD_PRELOAD still works with most Rust programs, which will link to libc as usual. (And in any case, you can't get things like full ASLR with a proper ET_EXEC binary.) What doesn't work is interjecting function calls under the assumption that the program can't work around them.
- layer8 1y agoI think the LD_PRELOAD is automatically inherited by whatever processes make executes. Otherwise you’d have to wrap each individual build step.
- zackmorris 1y agoYa that's what I wanted too. I'm actually flabbergasted that malloc() isn't a blocking call on most OSs, that would wait until the requested amount of memory was available before returning. That way programs would just suspend/resume as needed in low memory situations, rather that crashing. A process viewer could show which programs are blocked waiting for more memory, and users could optionally resume them manually. We could even serialize and unserialize entire programs and have them resume when enough memory is available. This one simple thing would have freed users from running under the assumption that programs can crash at any time, and allowed them to operate at a higher level of abstraction to get more real work done. This missed opportunity with a blocking malloc() is one of any number of obscene design decisions that I can't unsee in the tech status quo. In my experience, basically all software bugs stem from asynchronous/nonblocking behavior. Because it's difficult to prove that async code is deterministic without coming full circle and restructuring it as sync. For example, higher-order methods like map/reduce and scatter/gather arrays can replace iterators. And this can now be done automagically by LLM code assistants, and static analyzers since the middle of last century. Once we see that this is possible, especially if we use all const variables to avoid mutation (mostly future-proofing our logic), it's hard to avoid asking ourselves why we're all doing it the hard way. We should be able to click a block of code and choose "make sync" or "make functional" and vice versa. So that beginners could write batch files and macros with familiar iteration syntax and transpile it to safe and reliable functional code. And experts could write pure functional code in shorthand and export it as imperative code for others to verify. This was always another dream of mine, since I've been waiting for big companies to do it since the mid 1990s but they can't be bothered, yet I'll likely spend the rest of my life building CRUD apps to make rent.
- deleted 1y ago[deleted]
- marcodiego 1y agoI've seen energy-aware scheduling, literately decades of effort that culminated on the EEVDF scheduler so that it was possible to have a good scheduler that worked well on desktops, servers and HPC... and, between all those efforts, a giant parallel one to prevent or influence to OOM-Killer to behave better. I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. I mean, it doesn't happen anymore, but some years ago it was relatively simple to soft-crash a system just by trying to open a file that was significantly larger than the physical RAM. By 'soft-crashing' I mean the system became so slow that it was faster to reboot than wait for it to recover by itself. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system? Is there a reason why memory-aware scheduling was never explored or am I wrong and it was explored and proved not good?
- toast0 1y ago> I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system? This approach is hard to make work, because once the system is in memory shortage, mostly all processes will be slowing the system. There's already a penalty for accessing memory that's not currently paged in --- the process will be descheduled pending the i/o, and other processes can run during that time ... until they access memory that's not paged in. You can easily get into a situation where most of your cpu time is spend in paging and no useful work gets done. This can happen even without swap; the paging will just happen on memory mapped files, even if you're not using mmap for data files, your executables and libraries are mmaped, so the system will page those out and in in an effort to manage the memory shortage. To make a system easier to operate, I like to run with a small swap partition and monitor swap usage both in % and by rate. You can often get a small window of a still responsive system to try to identify the rogue process and kill it without having to restart the whole thing. A small partition means a big problem will quickly hit the OOM killer without being in swap hell for ages. There might be research or practice from commercial unix and mainframe where multi-tenancy is more common? What I've seen on the free software side is mostly avoiding the issue or trying to addressing it with policy limits on memory usage. Probably more thorough memory accounting is a necessary step to doing a better job, but adding more ram when you run into problems is effective mitigation, so....
- imp0cat 1y agoI just wanted to point out that GNU parallel has built-in options to do the same thing when running parallel processes that could possibly overwhelm the computer. --memfree size Minimum memory free when starting another job. The size can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
- nialv7 1y ago1. You don't need this. Just run programs inside cgroup and set a memory limit (systemd-run lets you do this in a single convenient command). When the program reaches its memory limit it will be throttled. 2. Also often a bad idea. If you slow down a process you are also stopping it from _releasing_ memory.
- zokier 1y ago> When the program reaches its memory limit it will be throttled. I thought it will be killed?
- jeroenhd 1y agoI believe they're talking about memory.high (https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git/tree/Documentation/admin-guide/cgroup-v2.rst#n1323 https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.gi...): memory.high A read-write single value file which exists on non-root cgroups. The default is "max". Memory usage throttle limit. If a cgroup's usage goes over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure. Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached. The high limit should be used in scenarios where an external process monitors the limited cgroup to alleviate heavy reclaim pressure. This is unlike memory.max, which does invoke the OOM killer. You can still reach memory.max after crossing memory.high, but you're less likely to do so if you have a method of dealing with the throttling.
- deleted 1y ago[deleted]