7 ms·
Quite. Can we not phase out overcommit? Windows doesn't do it. I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in
by orra 4y ago
Quite. Can we not phase out overcommit? Windows doesn't do it.
I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
- josephcsible 4y agoWhat would we do about all of the processes that fork but don't ever exec? And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
- AnIdiotOnTheNet 4y agoWhat's this whole open-source thing worth if we can't fix those to do it a better way? Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
- tinus_hn 4y agoToo bad it’s basically impossible to deal with the possibility that you request memory and don’t get it, apart from crashing.
- RunningDroid 4y agoYou could hang and hope someone else free()'s enough for you to continue, but the system will lock up if too many processes try that without giving up after a timeout
- AnIdiotOnTheNet 4y agoThere are lots of non-fatal ways to handle an out of memory condition (including doing your own paging) depending on the use-case of the allocation. Programmers suffer from learned helplessness because they're writing for systems that lie to them about memory and languages that make handling errors a pain.
- tinus_hn 4y agoThat sounds obvious until you realize that you can’t get away with just fixing the obvious parts of allocating large optional caches, you have to defend for every single allocation you do to fail. In every library you use, for every string you copy, for every structure you create. That’s a lot harder than handwaving about how someone else might be able to fix some cases.
- AnIdiotOnTheNet 4y agoYes, and? The difficulty of this is overstated. Except maybe that library thing, but my general guiding principle is to use as few dependencies as possible already anyway so that mitigates it somewhat. If there are some situations where there is no logical course of action? Then yeah, crash I guess. Assuming your software isn't responsible for people's lives. It's still better than the OS randomly killing you because some other program fucked up. I mean this is just crazy. The OS lies to you about memory, you ignore it anyway since it always lies to you, and then we randomly kill programs when we're out of memory. This is not a reasonable way to do things. The problem, as I see it, is that by now programmers are so used to not paying attention to memory (or any other resource in most cases) that there's a ton of inertia behind the "nothing can be done" sentiment.
- tinus_hn 4y agoI suggest you pick up a project and show us how easily this can be done.
- AnIdiotOnTheNet 4y agoLeave that aside for a sec. Are you honestly saying that because it isn't easy to do this, the best course of action is to make it impossible by lying to the program?
- tinus_hn 4y agoIf it bothers you, you can just turn off overcommit. It’s not hard: $ sysctl vm.overcommit_memory=2 # policy number 2 $ sysctl vm.overcommit_ratio=0 # ratio = 0% So here’s your golden opportunity to show us how easy it is to program against that! I didn’t claim that it is not easy. I claimed that it is practically impossible. You claimed it was easy. So please, prove me wrong and just show us how easy it is.
- eadmund 4y agoThat depends on what you mean by ‘crashing,’ doesn’t it? One could at least fail gracefully: at start-up, request enough memory for a buffer reading ‘request to allocate memory failed; exiting’ and then print it to standard output, then exit. One might initiate a garbage collection cycle. One might even allocate enough buffers at start-up to cleanly process outstanding work, write results and then terminate. It requires a little thought and engineering, but there are approaches which don’t just dump core.
- tinus_hn 4y agoFor very large values of ‘a little’. Just like that other poster with the ‘easy’ solutions, I challenge you to actually implement a demonstration of those easy ideas. You can actually take some ideas from the Linux kernel if you like because it has what you propose, a way to allocate memory that never fails and never blocks. And there you can tell how easy that is.
- striking 4y ago> Then only badly behaving programs will be shot in the head randomly. So, all of them then?
- orra 4y ago> What would we do about all of the processes that fork but don't ever exec? I speculate it isn't common, or at least it isn't required for ordinary desktop apps. Windows survives without fork(). > And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less? That sounds like a glibc implementation detail, not something required by the standard. I'm not an expert however. Simply, again I come back to: Windows can spawn processes without using it as an excuse to overcommit.
- josephcsible 4y agoWindows is a terrible OS, especially for servers, so pointing out that it doesn't have something isn't really a great argument for that thing not being very important.
- orra 4y agoWindows is proprietary and until recently, not very scriptable. None of that is relevant. We're talking about desktop users having their session killed.
- takluyver 4y ago> I speculate [fork] isn't common, or at least it isn't required for ordinary desktop apps. From a science perspective, the ability to fork is really convenient for parallel processing. Load some data and then fork a pool of worker processes, and they can all read almost free copies of that data. This is much easier than setting up shared memory. It's a bigger deal on an HPC machine with 128 cores than a laptop with 8 cores, but even on a laptop it's a significant point in favour of Linux. Getting rid of it in Linux would also be going against Linus' rule that the kernel never breaks userspace.
- sillystuff 4y agoLinux has never forced you to use memory overcommit. You can argue about the default, but the default seems pretty sane for most systems/use cases. https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
- orra 4y agoGood to know it's not required, but I'd certainly say it's a bad default. I can understand an app that is denied a malloc crashing. But an app which thinks it has successfully been allocated memory, being chosen at random to crash? Not ideal, just like in OP's example.
- worthless-trash 4y agoNope, and nope. Doing this wont give you the fix you think it will.
- zajio1am 4y agoDiscussions about overcommit are red herring for OOM situations. You could be fully within commit limit for allocated unreclaimable memory, and still be in effective OOM situation, if working set is larger than RAM.
- zajio1am 4y agoAnd even if we ignore issues with too large working sets and only focus on memory allocations, then if some badly written program eats all memory, then running some OOM killer (who hopefully kills the offender) is better than keeping it running and just failing all consequent allocations, as such system is unusable anyways.