14 ms·
Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59
- mrmondo 10y agoIt's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you want a Debian based system use Debian (and accept that there's now a lot of missing packages are barely any proper SELinux or clustering support) or use RHEL/CentOS for a more complete district and add in repos like epel and elrepo as required.
- aikorevs 10y agoCan you please share your experience a bit more ? How much SELinux is really required ? What about grsecurity. My use case is custom build Django/Postgres app.
- snuxoll 10y agogrsecurity is enhancing the security of the kernel itself, SELinux is for user space applications.
- viraptor 10y agoThat's an oversimplification. Grsec protects both kernel and applications, although mostly via separate systems. Selinux mostly protects applications, but can also stop malicious interactions with kernel code (syscall and virtual file access restrictions)
- mrmondo 10y agoSELinux is an absolute must in todays world, simply put it prevents applications and namespace from doing thing outide of their area of interet, for example a web server should never be allowed to execute files from /tmp so it prevents that, another great example is when you have any form of multi-tennancy - for example you might have several docker or LXC containerised apps running on a host SELinux will / can ensure that even if of those apps is compromised and someone popped a shell in it, they couldn't go beyond anything running in that namespace, i.e. to the host itself or other apps running in containers on the same host. It prevents a lot of sitations where an app might behave badly, for example perhps you're running a PHP app and a 0-day PHP flaw is discovered, there's a good chance that SELinux restrictions could prevent that exploit from being useful to the attacker by not allowing the app to do unusual things. We have a rule - if it's not running SELinux in enforicng then it's not going into production. We use to run GRSec as well, however with modern kernels a lot of the GRSec code base has been merged in so it's often not required.
- lima 10y agoI agree with you except for the GRSec part - most GRSec features are still missing except for a few low hanging fruit.
- tayo42 10y agoCentos/rhel are on a pretty old kernel at this point though?
- samhamilton 10y agoFor Centos 7 its 3.10 according to good old http://distrowatch.com/table.php?distribution=centos http://distrowatch.com/table.php?distribution=centos
- mrmondo 10y agoand it's whatever is the latest, stable, tested version of the kernel when using Elrepo's Kernel-ml repo: http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml
- mrmondo 10y agoWe run the latest mainline kernel on all our 200-300ish CentOS hosts after it's passed standard testing and is available in Elrepo: http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml Back when we ran Debian everywhere I wrote a CI project that would build the latest mainline kernel, applying GRSec and adding some SELinux policies in as best I could (backported from CentOS/Fedora funnily enough) and build them into a Debian package for deployment across our hosts, see: https://github.com/sammcj/kernel-ci https://github.com/sammcj/kernel-ci
- quotemstr 10y agoWhat the hell? What happened to "don't break userspace"?
- mrmondo 10y agoYeah I know right, Ubuntu breaks userspace all the time, Debian isn't quite as bad but it's much worse than it used to be thanks to so many missing packages, try building a cluster / HA system on it - you actually can't!
- Pxtl 10y agoYeah, Ubuntu still keeps running out of space in /boot by filling it w old kernels in the default partition profile. A bug that I've been hitting since about 2011. In spite of their best efforts, it is not the "grandma friendly" quality software it hoped to be.
- peller 10y agoYou can run `apt-get autoremove` and it will purge old kernels/headers for you (amongst any other no-longer-used dependencies).
- Pxtl 10y agoOh, I know it's easy to fix, I just single that bug out as "I have done nothing with this box but install updates and surf the web. I am Grandma. It breaks".
- nnq 10y agoIf you're Grandma you don't have /boot on a separate partition, you have all / on a big 120GB at least, so if you run out of space you simply know you've run out of space. Not perfect but not a problem for 90% of people I guess...
- cakeface 10y agoUbuntu installer has separate /boot partition by default. If you don't do anything special during installation you end up with it.
- spikengineer 10y agoNo you don't. Ubuntu's default partition map is one big / and one efi(if necessary).
- lima 10y ago
- ivank 10y agoI didn't mess with swappiness, but I also didn't have a swap.
- Jedd 10y agoCrikey - they did what?! I recall a fascinating article / thread years ago about swappiness=0 not actually meaning what people thought it meant, and explaining why it was (generally) a bad idea. A low number, yes, but not 0. Doing some research now, and found a blog written by someone bitten by this change back in 2014-04 -- referring to the commit that introduced this in 3.5-rc1. Not clear why the change was made, though, as a) it breaks lots of people's understanding / expectation, and b) if you want to never swap out then just forego the creation of a swap file / partition. [1] https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/ https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...
- StreamBright 10y agoExactly, people in this thread missing the point of the use of RAM. A perfect operating system always going to use as much RAM is available for caching and there are long running processes that can be swapped out to free RAM for more important use cases. Memory management and paging/swapping is an interviewing topic for SREs/systems engineers, and this is not an accident. Oversimplifying the problem does not help.
- ahoka 10y agoIt's surprising that many professionals have broken mental models about how an operating system supposed to work. I think they have something like DOS in mind. I was just told last week about how caching static (!) files from web server in redis increase performance in their infrastructure. I was baffled.
- mrmondo 10y agoYeah it wasn't well announced when the change was made, esentially if you find any linud machines running any kernel dating way back to 3.5 make sure you change vm.swappiness=0 to vm.swappiness=1
- userbinator 10y agoIMHO if you can't not use swap without running out of memory in normal workloads, you either need more RAM or to work on reducing your memory consumption; swapping is for temporary overloads or gracefully recovering from one, not a substitute for RAM. It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. That's a surprising behaviour change (from the perspective of "don't break userspace"), but looks more like a "fix of previously broken behaviour" as in 0 not being able to disable swap entirely; from that perspective, using 1 makes sense.
- feld 10y agoFunny, but Linux is the only unix-like OS I've ever run where using swap means you have a problem. Other OSes have much more intelligent memory management.
- brianwawok 10y agoFor me swap is an error. I would rather a server crash then go into swap and run slow. If your servers are at 75% CPU and start swapping you are screwed. Time to start adding servers. In this day and age you are wasting a lot of money to be sitting around waiting for swap.
- fpoling 10y agozswap makes swap useful on the server as it results in gradual slowdown when the system has used all the ram. And if the system runs at 75% of CPU, then there is room for such slowdown.
- twic 10y agoI hadn't heard of zswap [0]: "Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based memory pool. [...] Zswap is a new feature as of v3.11 [...] zswap is a work in progress and should be considered experimental." The return of RAM Doubler! The naming is initially confusing if you're already familiar with ZFS, especially: "Zswap makes use of zpool for the managing the compressed memory pool" [0] https://www.kernel.org/doc/Documentation/vm/zswap.txt https://www.kernel.org/doc/Documentation/vm/zswap.txt
- dima55 10y agoWhat are the many missing packages on Debian? From my experience Centos looks like a bad joke.
- solarengineer 10y agoI happen to run most of the latest of things on CentOS (server-side only, though). My approach has been to package and deliver modern versions of various runtimes (python, ruby, jre) into specific directories for my apps. This way, I am not dependent on the system-provided runtimes which are an older version. For my work with Belenix (revamp not announced yet), I am working on examples where users no longer depend upon system-supplied package versions, and instead choose their own local instances.
- bonzini 10y agoWhy don't you use Software Collections to get the newer versions of the runtimes?
- solarengineer 10y agoThough I had a commercial contract in place for Software Collection, these were not as up to date with what the communities released (e.g. Ruby). Hence my approach of packaging and testing my own builds of these.
- jlgaddis 10y agoAre you aware of the "software collections"? That may or may not be of use to you.
- bonzini 10y agoCentOS only includes packages that Red Hat builds and supports on RHEL. Extra packages are maintained by the community as part of Fedora EPEL.
- mrmondo 10y ago
- StreamBright 10y agoExactly. I have stopped using it long time ago, after they introduced the following bug: https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bug/544935 https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bu...
- eikenberry 10y agoDon't use EPEL in production, it is an unstable mess. If you need more packages than are in the core of CentOS, Debian is the best choice.
- lima 10y agoHow is EPEL an unstable mess?
- polpo 10y agoIt's not -- I'm getting OOM on some java processes and my swappiness is set to 60.
- mrmondo 10y agoJava has its own memory limits that it will trigger OOMs at.
- kasabali 10y agoRHEL has had a similar behavioral change during lifetime of a release so this particular case is not actually a good example for recommending RHEL over Ubuntu for stability. https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/ https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...
- Cieplak 10y agoI stopped using Ubuntu after Amazon affiliate marketing showed up on the desktop. I stopped using Arch after I read the source code for SystemD. FreeBSD had quite a learning curve, but now it's on my laptops and is my preferred server OS. I love DTrace. I love ZFS. I love pf. I love Jails. I love ports, even though I generally use pkg. I love how easy it is to compile a kernel. The FreeBSD handbook is awesome. All the JetBrains IDEs work well with it. Hardware support is spotty and battery life is terrible. The BSD license means you can incorporate FreeBSD into commercial products without infecting yourself with the GPL.
- chrisper 10y agoCan you provide any good entry points to the BSD world?
- Cieplak 10y agoAre you coming from Linux, Windows or OSX?
- chrisper 10y agoI am very good with Linux and Windows. So either of the two are fine.
- Cieplak 10y agoI would perhaps spend some time running it in a VM, and then if you decide you like it and think it's worth the time investment, maybe try installing it on your laptop. I run it on a Thinkpad X1. The handbook is a great resource: https://www.freebsd.org/doc/handbook/ https://www.freebsd.org/doc/handbook/ The book, Design and Implementation of the FreeBSD Operating System is quite good if you like technical reading.
- JdeBP 10y agoThere are several, depending from what one wants. TrueOS, formerly known as PC-BSD, is a good entry point if one wants a desktop-style system. Or one can try vanilla FreeBSD. Or OpenBSD. Have some user guides and Handbooks to read: * NetBSD Guide: https://netbsd.org/docs/guide/en/ https://netbsd.org/docs/guide/en/ * FreeBSD Handbook: https://freebsd.org/doc/handbook/book.html https://freebsd.org/doc/handbook/book.html * DragonFlyBSD Handbook: https://www.dragonflybsd.org/docs/handbook/ https://www.dragonflybsd.org/docs/handbook/ * TrueOS User Guide: https://www.trueos.org/handbook/trueos.html https://www.trueos.org/handbook/trueos.html * PC-BSD User Guide: http://web.pcbsd.org/doc-archive/10.1.2/html/pcbsd.html http://web.pcbsd.org/doc-archive/10.1.2/html/pcbsd.html (viewable off-line directly in both PDF and HTML forms in /usr/local/share/pcbsd/doc/)
- vivin 10y agoIf you want something similar to Ubuntu, but something that is much more stable, I highly recommend Linux Mint. I've been using it for about 4 years now and it's a delight. Very stable and doesn't give me a lot of issues after upgrades. Their upgrade-schedule and the upgrades itself are very conservative. They've based off a Ubuntu LTS, but I think they are starting to move away from that.
- deleted 10y ago[deleted]
- SSTitan 10y agoLinux Mint has issues - http://www.techrepublic.com/article/why-the-linux-mint-hack-is-an-indicator-of-a-larger-problem/ http://www.techrepublic.com/article/why-the-linux-mint-hack-...
- exDM69 10y agoYes, a failure at safely distributing software is a pretty big failure when it comes to distributions. Mint, elementary and the other "user friendly" distros should focus on creating the desktop and leave the distribution part to projects that have the resources and the staff to maintain the infrastructure.
- lscotte 10y agoWell, I'd argue just use base debian. All the same benefits, none of Mint's security issues.
- SkyRocknRoll 10y agoWe are affected by this. Our elasticsearch process were getting killed even though we set xmx to half of the ram. Now reverting the kernel.
- slitaz 10y agoIs the problem with just those users that changed the snappiness value to 0?
- elementalcells 10y agoI finally switched my desktop to Xubuntu this week (from Win 10) after using it happily on the laptop for months. Stability is, unfortunately, not great. Games exit unexpectedly, Firefox tabs frequently crash. uname -r prints this exact kernel version so I assume I'm affected.