10 ms·
In emergency cases a passenger was selected and thrown out of the plane (2004)
- ajdecon 13y agoIt is, in fact, possible to make a process immune to the OOM killer: echo -17 > /proc/$PID/oom_adj where $PID is the process ID you want to protect. oom_adj can be tuned with other values to make a process more or less likely to be killed. http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html http://www.oracle.com/technetwork/articles/servers-storage-d...
- axylone 13y agooom_adj is depreciated, use oom_score_adj instead. https://www.kernel.org/doc/Documentation/filesystems/proc.txt https://www.kernel.org/doc/Documentation/filesystems/proc.tx...
- MattJ100 13y agoDepending on its use, the first thing I generally do on a new server is disable the OOM killer. At runtime: sysctl vm.overcommit_memory=2 To make it persist, just add vm.overcommit_memory=2 to /etc/sysctl.conf
- HerrMonnezza 13y agoNote however, that turning off overcommit might yield unexpected results upon fork()/exec(): http://www.quora.com/What-are-the-disadvantages-of-disabling-memory-overcommit-in-Linux# http://www.quora.com/What-are-the-disadvantages-of-disabling... For instance, if you run a Redis server and turn off memory overcommit, you might not be able to background save.
- alextingle 13y agoWarning - that site requires a login.
- 0x006A 13y agoas mentioned in https://news.ycombinator.com/item?id=6300856 https://news.ycombinator.com/item?id=6300856 you can add ?share=1: http://www.quora.com/What-are-the-disadvantages-of-disabling-memory-overcommit-in-Linux?share=1 http://www.quora.com/What-are-the-disadvantages-of-disabling...
- andreasvc 13y agoThat allows me to see the first answer but not the rest.
- calinet6 13y agoWhat are the true consequences of disabling it though? What happens in the OOM situation? Or does it just not allow allocation?
- adobriyan 13y agoAllocation fails with ENOMEM.
- Piskvorrr 13y ago...which is an error that is rarely handled in the code nowadays, therefore highly unexpected behavior may be expected ;)
- dredmorbius 13y agoFile and/or tear your vendor a new one.
- dredmorbius 13y agoEdit: file a bug ...
- derefr 13y agoI guess I'm one of the few people(?) who like the OOM killer. If all your deployed software is written to be crash-only[1], and every process is supervised by some other process which will restart it on failure, then OOM is basically the trigger for a rather harsh Garbage Collection pass, where software that was leaking memory has its clock wound back by being forcefully restarted. Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :) --- [1] http://lwn.net/Articles/191059/ http://lwn.net/Articles/191059/
- primitur 13y ago>I guess I'm one of the few people(?) who like the OOM killer. Diff'rent strokes. I also like the OOM killer, its a dastardly wonderful thing to tie lots of safety-critical things to .. and in the SIL-4 OS business (my domain), it is indeed a crash imperative to understand how to use the OOM properly. Or: not. So in this light .. I know Erlang is "the thing" right now, but I feel I must just mention that: >Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :) .. one could also be designing a Lua-based distribution, or JVM, or whatever you like, essentially, and integrating with oom_killer. There's nothing Erlang'y about it. Because if you're playing with the oom_killer, you're really making a distribution choice, in the topology. If your app cares about oom, well kiddo .. you better not be doing anything less than excercising complete control over your distro, its launch policies, its use of the TextSegment as installed, and so on. Absolutely you're making Distribution decisions about the functionality of the combined system. oom_killer isn't useful by itself. My point being, worrying about oom_killer isn't just something Erlang users need think about, nor are they the only ones who really 'get' why an oom_killer can be used nicely .. if you're building a distro, either for use as an embedded machine, a tight secure server image, or indeed even as a desktop user, well .. careful memory integration is, as you say, a harsh pass. Incidentally, I use the oom_killer exactly as you mention, in a few embedded distro applications, specifically indeed, a kill of whatever 'lua' is hogging resources. Its an extraordinarily functional mechanism for recovery ..
- kudu 13y ago
- IvyMike 13y agoThis reminds me of my one and only question on Stackoverflow: "Throwing the fattest people off of an overloaded airplane." http://stackoverflow.com/q/7746648/67591 http://stackoverflow.com/q/7746648/67591
- arethuza 13y agoThat got me wondering as to how much stuff (hand luggage, trollies, food and drink, in-flight magazines...) an average flight carries!
- AmVess 13y agoRats. I read this post, and my Windows 8 install disc vanished.
- eliben 13y agoSincere congrats. Losing your Windows install disc is one of the best things that can happen to a person.
- deleted 13y ago[deleted]
- deleted 13y ago[deleted]
- deleted 13y ago[deleted]
- jameswilsterman 13y agoAt least offer parachutes?
- twistedpair 13y ago$150 each, correct change only.
- mistercow 13y agoAlso, they count as your carry on bag, and they only bring one parachute on board, so if multiple people have to be thrown off and they've both paid for a parachute, they have to draw straws to decide who gets it. Edit: And remember, OOF goes off on less than .1% of flights, so they have hundreds of times as many parachutes per flight as they have people who need them. Rumors that parachutes are oversubscribed are therefore wildly inaccurate.
- cbsmith 13y agoThe real irony here is that airlines actually do something very much like overcommit & OOM killer when it comes to reservations, and for precisely the same reasons: they know that not all the reservations will be used at the same time, but sometimes they do end up double booked, so then someone has to be kicked off the flight.
- tcoppi 13y agoPrecisely. The OOM killer has its uses, and while it's debatable if it is suited for this workload or that workload, it is easily tunable for specific use cases, and you can disable it altogether if you'd like the classic behavior.
- lucaspiller 13y ago> sometimes they do end up double booked Sometimes is pretty much all of the time. My SO works as cabin crew for one of the large airlines you will have heard of, and they typically always overcommit. It isn't just because they are greedy and want lots of money though (well I guess it kind of is). As you said, they know some people won't turn up, but also they may decide to change the class of the airline (economy and business -> economy, business and first) depending on how people book or what aircraft are available on the day.
- ubernostrum 13y agobut also they may decide to change the class of the airline (economy and business -> economy, business and first) depending on how people book or what aircraft are available on the day. United is infamous for this, prompting a sound piece of advice: if for some reason you're in first class on a flight scheduled for an A320, don't take a seat in row 3, since odds are high it'll be swapped to an A319 (which only has two rows in F) day of the flight. In theory there's a pecking order for who gets downgraded in that case (first, anyone who moved up to F on a status upgrade, then the paid F fares starting with whoever was latest to check in), but in practice the gate agent just bumps row 3 and calls it a day.
- 13y ago
- kalleboo 13y agoHere's a novel way to deal with an out of memory situation caused by slow memory leaks in a long-running server process: start swapping memory that hasn't been touched in literally days or weeks to /dev/null, and pray the process doesn't ever need it again.
- jbrechtel 13y agoIs pray a syscall?
- kalleboo 13y agoI'm pretty sure there's a linux kernel module. Obviously it'd be required for my method to work. Might not be doable in FreeBSD (incompatible with the branding).
- please_advise 13y ago> Might not be doable in FreeBSD (incompatible with the branding) Perhaps something more in line with BSD would be a sacrifice(2) syscall? It would pick a random process and...
- bonzoesc 13y agoThat's so indescribably worse than just killing that process the mind boggles. Breaking in a simple, predictable, and detectable way vs. corrupting data and hoping (excuse me, "praying") nobody notices.
- kalleboo 13y ago> vs. corrupting data and hoping (excuse me, "praying") nobody notices. Well it seems to work for MySQL Anyway, you wouldn't ever return the nulled data. If the process tries to access the data, THEN you crash it.
- networked 13y agoCompressing the memory area used by the leaky process in question would be a gentler and more robust "solution" here. There already exists a Linux kernel module called "zram" [1] with which you can accomplish just that (though you might have to tune your swappiness a bit first). [1] https://en.wikipedia.org/wiki/Zram https://en.wikipedia.org/wiki/Zram
- antocv 13y agoThe few cases when Ive seen OOM invoked, it took couple of minutes to kill chromium after flash (of course) messed up, during that time the system was unresponsive and it killed few random smaller processes until it hit the correct one, flash or chromium in some weird interdependent bug. Either way, I wasnt too happy. After a while I noticed when the bug triggered/the system started becoming unresponsive, and I had a terminal with killall -9 chromium & killall -9 flash-plugin ready to go, so I could myself preempt it and OOM wouldnt get involved. There has to be better mechanism than OOM.
- BrainInAJar 13y ago> There has to be better mechanism than OOM. Like not overcomitting memory in the first place
- Piskvorrr 13y agoToo slow for the command to pass through the usual "X - WM - TE - shell - killall" chain? Try Alt+SysRq+f ; that, in my experience, is waaaaay faster for invoking oom_killer.
- pritambaral 13y agoHis issue is that oom_killer doesn't get it right straightaway. Alt+SysRq+f would still need multiple invocations before it gets flash. Personally, I do use Alt+SysRq+f since it predictably targets GMail-on-Chrome everytime on my system. That is usually enough on my desktop OS for me to jump in manually kill the offender. I can then just F5 GMail.
- Piskvorrr 13y agoEven so, in OOM situations, sysrq invocation is still an order of magnitude faster than killall invoked from a graphical terminal emulator. As for hinting to oom_killer: I have a script which searches for chrome and flash processes every minute, and sets their oom_score_adj in the high hundreds. This makes reasonably sure that oom killer will go after these processes first.
- jballanc 13y agoI wonder why Linux hasn't adopted something like OS X's "Sudden Termination" mechanism: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSProcessInfo_Class/Reference/Reference.html#//apple_ref/doc/uid/20000316-SW3 https://developer.apple.com/library/mac/documentation/Cocoa/...
- calinet6 13y agoWell FFS that makes a lot of sense. OSX is just like, "Hey guys, if any of you happen to not need your memory sometimes, would you mind kindly letting me know and I'll go ahead and let you go at a convenient time?" Meanwhile Linux goes on a murderous rampage with unpredictable effects.
- erichurkman 13y ago... And Windows is sitting in a corner, fans kicked on high, trying valiantly to manage with swapping to disk until a sysadmin gives up trying to connect via RDC and yields to the age-old "have you tried turning it off and back on again?"
- saurik 13y agoThe part of this I hate most is that if you have physical access to the machine and can hit ctrl-alt-delete you are given an out-of-band dialog that functions flawlessly; this dialog happens to have a "task manager" button, but as the task manager functionality is part of a normal user application and not that magical dialog, you get thrown back into the swap storm, only now with yet another process (taskmgr) competing for memory :(.
- plorkyeran 13y agoThe memory alert stuff on iOS is similar; when the phone runs low on free memory it tells the app free up as much as possible, and then only starts killing processes if there still isn't enough available. It can be a bit finnicky to work with, but it does seem to be pretty much strictly better than just killing random processes without giving them a chance to be better behaved.
- Systemic33 13y agoHas anyone forwarded this to Ryanair yet?
- Argorak 13y agoIf you want to seperate your passengers in first and economy class, this is the relevant guide: http://lwn.net/Articles/317814/ http://lwn.net/Articles/317814/ Usually, I recommend that database and queue servers run the database/queue with a priority that makes it unlikely for them to be killed. I had a case where a colleague running a script on a server with high pressure killed the queue, which is unadvisable, even if is crash-safe. Before that, the queue was running for 1.5 years straight.
- zw123456 13y agoSome years back I was flying a small commuter who used small prop type airplanes (I call them pterodactyl air). Part way through the flight, I noticed one prop seemed like it was not working, so I leaned forward to alert the co-pilot (the plane was that small). He told me that they would turn off one engine and "feather the prop" to save fuel. I told him that I would be happy to take a up a collection back in the cabin from the other passengers to pay for the extra fuel to power both engines. He chuckled, but I was serious. I never flew with them again. Maybe there is a way to suspend a process (feather the prop) rather than completely kill processes.
- yannyu 13y agoJust FYI, pretty much every plane with multiple engines is able to safely fly and land with only one engine working, and all commercial pilots are trained to do so. If you were halfway through the flight already, it's entirely possible you were already in descent and didn't need the other prop at all to complete the flight.
- yardie 13y agoWow, that was ridiculous. So, because you don't understand how planes work this is the pilots' fault. Even if you were serious about the fuel exactly where do you propose they put it? In-flight refuelling of commercial planes doesn't exist.
- cmbaus 13y agoMy memory is a bit hazy in this area, but I think by default memory is over committed in Linux. What that means is malloc() can return an address that doesn't have physical memory assigned in the page table. Memory isn't committed until it is written to. This isn't the case with the default MSVC implementation of malloc() in Windows. In Windows address space is reserved and committed with VirtualAlloc(), and typically that is done in one step. I think memory is over committed because Linus wanted to keep the memory footprint lower than NT early on in the development of the kernel. The drawback is applications may segfault when writing to memory that was successfully returned by malloc().
- kyberias 13y agoYou seem to have some terminology problems here. Windows VirtualAlloc may "commit" memory but that does not mean it actually reserves physical pages [1]. That always happens only when the memory is accessed. On the other hand, MSVC's malloc() probably uses HeapAlloc(), which in turn uses VirtualAlloc(). I don't think there are any fundamental differences between Linux and Windows here. [1] "Actual physical pages are not allocated unless/until the virtual addresses are actually accessed." http://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...
- MichaelGG 13y agoThat link says: "Allocates memory charges (from the overall size of memory and the paging files on disk) for the specified reserved memory pages." It does count against the total memory allowed. My laptop has 8GB of RAM and 1GB page, giving me a 9GB overall commit limit. If I spawn a process that eats up 1GB at a time, even Task Manager can clearly show me going up and hitting 8/9GB, then I'll get OOM in my process. Windows won't commit memory that a process can't use. You can't overcommit, although you might end up in the pagefile. Without the odd concept of fork, you don't end up with processes having huge "committed" address spaces that aren't ever going to be used. The note about physical pages is just saying it's not mapping it, not that it's not guaranteeing it.
- 13y ago
- fusiongyro 13y agoOr, here's a crazy idea: how about we actually allocate the memory when you call malloc(), and if there isn't any, give you an error instead? Programs could check the return code and decide what to do when they run out of memory themselves. Crazy, I know.
- ars 13y agoYou can do that if you like. Use vm.overcommit_memory - setting it to 2 will still allow overcommit if there is room in swap for it. (So a process doesn't have to use the memory, it's just assured of a place in swap should it need it.)
- wmf 13y agoIf you disable overcommit then large processes can't fork()/exec(). In theory we should all switch to spawn() but that's not going to happen.
- mikegagnon 13y agoThat post is a great, poetic allegory. But ultimately, I think the analogy presents a bad idea. The allegory makes the point that we could entirely avoid OOM errors by engineering a system such that resources are never overcommitted. This is true; we could do that. However it would be bad. Under-committing resources (thus removing the need for an OOM killer) will NOT lead to a net gain compared to over-committing resources (and thus requiring an OOM killer of some sort). If we are unwilling to overcommit resources then it would be woefully uneconomical to run algorithms that have bad worst-case performance (because to avoid over committing you would necessarily need to assume the worst case is encountered every time). It's just not feasible to avoid algorithms that have bad worst-case performance. Rather, we need to develop better abstractions for dealing with components (e.g. computations, programs, processes, threads, actors, functions etc.) that go over budget. Here's my attempt at developing a better abstraction for web servers: mikegagnon.com/beergarden Ultimately, we need to treat every system like a soft real-time system, because at the end of the day every program has timeliness requirements and has resource constraints. The current POSIX model does not provide such abstractions and I think that's why we have these debates about OOM killers.
- AsymetricCom 13y agoI love this blog's UI. Did you make it yourself? I like the idea of the doorman but what if you could somehow pass back useful math to the client? Of course, then you'd have to disregard that useful work yourself or double check it negating the energy savings. Or, perhaps return a map of traveling sales man type problem (maybe a map of metadata and their traversal cost), and they could navigate that map depending on exactly what kind of data they really want, thus reducing your load for valid, heavy requests and if they return a path with lots of nodes, you know to de-prioritize it or drop the request.
- mikegagnon 13y ago> I love this blog's UI. Did you make it yourself? Thank you! Yes, I made the UI. It's open source: sidenote.io > I like the idea of the doorman but what if you could somehow pass back useful math to the client? I think it would be great to have clients perform useful computations instead of just burning cycles. But that's not MVP, even for a research project.
- joelthelion 13y agoI wonder if someone is suddenly going to come up with a magical solution for the OOM problem and put and end to all these pointless discussions.