5 ms·

(I'm the author of the original post) You bring up good points and my article would need some clarification on some parts, I agree on that. I'll just write qui

by jyrki 15y ago

(I'm the author of the original post) You bring up good points and my article would need some clarification on some parts, I agree on that. I'll just write quick replies back to you, and try to format something on the article itself later. Purge really did free memory and quite a lot. I'm not too expert (as you probably can tell) how the OS X memory management works, but I mostly settled with solutions that seemed to help my problem. Maybe there was some third party software that messed things up. The problem with the inactive memory is that it is not freed, it is swapped. So when hitting memory limits of my system, the computer started swapping. Just freeing the memory, in my case, would have been much quicker. Practically my machine was constantly swapping when the memory limit came up. As you said, repair disk permissions caused all this to happen due to filling memory with disk cache. So that was a nice solution to my problem; a way to force swapping on inactive memory. Python point is bit wrong, I was indeed trying to argue that installing python packages is impossible through homebrew. However, I did use a lot pip+virtualenv, so that's at best a bit vague argument on the OS X side. However, in production I always rely on the packages provided by the OS, not pip + venv, unless really necessary. This is mostly because it makes it easier to keep system up-to-date. I'm sorry if this showed up as uninformed rant, but I just wanted to share how I felt using OS X and Macbook for a year as my primary computer.

jrmg 15y ago
The problem with the inactive memory is that it is not freed, it is swapped. So when hitting memory limits of my system, the computer started swapping. Just freeing the memory, in my case, would have been much quicker. Practically my machine was constantly swapping when the memory limit came up. That's just not true. If inactive memory is something that's already backed by disk (like a memory mapped file), it'll be discarded. Only if it's read-write memory that's /currently held allocated by a running process/ will it be swapped out to disk. Unless you're doing something pathological (like suddenly allocating lots of RAM and forcing paging - what purge utilities do...), the architecture /speeds things up/. If it's really true that, without doing anything special, things were always being swapped out to disk for you, it necessarily means that there was a process that had allocated (lots, it sounds like) of RAM and written to it, so that stuff had to be paged out to disk to free up RAM without losing data. It sounds like you were running purge commands or utilities to 'free up RAM'. That is counterproductive. It causes the system to release cached 'clean' (i.e. as already on disk - a mapped file, essentially) mapped RAM and swap read-write memory out to disk, only then have to re-read it all when you actually need it. In other words, using purge 'utilities' actually puts the system into the worst possible state.
- mellifluousmind 15y ago
  The problem with the way how OS X keeps data cached in the inactive memory is based on the assumption that you are going to re-use the same app within reasonable amount of time. With the current behavior/performance response (without knowing exactly how Apple engineers implemented it), it really feels like a giant garbage collection system that takes age to free up its own memory with no real sense of concept that if you don't use certain apps for day, chances are, it is going to take a long while before I use them again. I am one of the devs out there running a Macbook Pro with 8Gig of memory (I wish I could have more but I have a 2010 old model). For web development, I have at least Firefox/Chrome/PhpStorm/SmartGit/Mamp/Thunderbird/Terminal/Notational Velocity/Dropbox/Alfred/Sophos AntiVirus open at all times. Now, a long the way, I may open a few other apps that I use rarely, like Photoshop/CyberDuck/VMWare Fusion/iTune/iCal/iOS Simulator/Preview/LibreOffice/Skype. Now, pretty quickly 8Gig gets used up, and the system runs to the ground shortly after. If I then have VMWare Fusion shutdown for awhile, and relaunch later, the system really just can't take it anymore. The last resort? purge& At least, that's my day-to-day experience with OSX. Personally, I find the memory management really lousy, worse than other other OSes I used in the past (both Windows & Ubuntu/Fedora/Gentoo). So why the heck I use Mac still? Because of the driver support is still far better than Linux. With Mac, you are less likely in need of blacklist of some drivers because of freeze up issues. Either way, I am definitely not a happy camper with the current memory management system in OS X
  - jrmg 15y ago
    The inactive count is a misleading figure in many ways. It includes both 'dirty' and 'clean' (i.e. identical to what's on disk, and hasn't been altered) memory that's not been used for a long time. Purging 'clean' memory is instantaneous. The only cost you'll see is writing 'dirty' memory out to disk. Your explicit purging is changing the cost of writing dirty data out from an ongoing cost to a single, longer, upfront cost. Instead of writing only when more RAM is required, you're forcing it all to happen at once. Incidentally, the OS does try to keep an area of free RAM so that some memory can be allocated instantly, it's not only swapping things out when RAM is absolutely full. It's possible though to outrun this process if an app tries to allocate huge amounts of RAM at once though (i.e. more than is kept free for this purpose). For your specific case, presuming apps are behaving well (see below) you would be better off quitting apps and relaunching them when you need them later. This will free up app the dirty RAM they've allocated (just like when you purge), but the 'clean' inactive RAM will not be purged (because that's not necessary - as I said, it's free to purge that kind of memory when it's needed for something else). You also want to run Activity Monitor when your system is in it's bad state and see if it's one of the apps you're using in particular that's allocating lots of memory (check out the "Real Mem" column). The OS can't do anything if it's an app that's really allocating and writing to memory, it's obviously not able to just discard this written-to memory. Really though, if you want to do all those things at once, more RAM might be required. Remember, with VMWare and the iOS Simulator running, you've got two whole other OSes running at the same time, it's reasonable they'd require lots of memory to work well!). By the way, the purge command was written to simulate /worst case/ conditions when performance testing. It's designed to flush out caches so that the system has to e.g. load all an app's code from disk when launching. [Source: I worked analysing this kind of thing at Apple until a couple of years ago].
RBerenguel 15y ago
I think the problem is that you used OS X for a year. I've been using my Mac for 4 years, and Linux for another 4. And I have Python (in fact I have at the same time Python 2.5, 2.6, 2.7, 3 and 3.1 for testing purposes), I have installed so many packages I already lost the count. You only need to be sure the path is correct when doing it. And I have 2 Gigs of RAM. Mac OS can be a bit hungry, but there's an "easy" solution for this. Closing unused applications. That bright dot in the dock means something is using an incredible amount of RAM just to sit there doing nothing. If I only leave open my mail app, Chrome, emacs and twitter client, I can go for days without ever hitting the disk cache. If I start adding more things (or tabs like I'm crazy), I'll hit the cache, of course. But this also happens in my Linux systems, it's not a Mac OS problem. As for the DVI port... I usually leave the DVI adapter linked to the monitor I use. And use bluetooth devices when possible. So far only once I have used a hub to connect 3 things to my Macbook (an SD reader to copy images, my Ben Nanonote to install some packages and my iTouch to sync with iTunes). Happened once, in 4 years (and I also have a drawing tablet, btw).
- slowpoke 15y ago
  >Closing unused applications. That's not a solution, that's a workaround. I'm sitting here with a 4GB RAM machine running Arch Linux with XMonad. I usually run the following applications in day to day usage: - Firefox Nightly with 50-100 tabs (largest memory hog, the rest doesn't even come close) - Thunderbird - Pidgin - XChat - smplayer, which I usually keep open - deluge - mpd + ncmpcpp - half to one dozen of terminator instances with zsh, most running some text mode applications like vim, htop or the aforementioned ncmpcpp I reboot my PC once in a blue moon, usually after kernel updates. otherwise it's running 24/7. Right now, I sit at ~35% memory (and that should include memory used for disk caching), a bit less than half of which is Firefox with ~16% (65 tabs). I usually never go over 50% unless compiling heavy stuff (like QT level heavy). I don't really know what the fuck OS X does to eat all that RAM, but it apparently does something wrong.
  - lloeki 15y ago
    Emulating a setup similar to yours, the OSX machine infront of me sits a 2.8GB active, of which a little less than 800MB are taken by Terminal.app due to me having activated infinite scrollback and having two rails processes being hit 8 hours a day since like forever as I develop. So it's really closer to 2GB and I didn't even try hard. This includes mds+mdworker (the indexer) which clocks in at 200MB. Normally I also have LibreOffice, Pixelmator, iCal, iTunes, Reeder, VMWare Fusion, which makes it balloon to 3.1GB and up as I open more documents/VMs. OSX is not doing anything "wrong".
Terretta 15y ago
I do happen to have an 8 GB machine, and almost never reboot (only reboot for OS updates). I run half a menu-bar full of resident helper apps, like Dropbox (a big one), Fantastical, ScanSnap, Xmarks (another big one), Transmit (another big one), Evernote, and more. I also keep Apple Mail running, mapped to a half dozen Gmail IMAP accounts. I have "geek tool" updating my desktop with iCal appointments and various ps outputs. I'm running a local MAMP stack and local Django stack. I run a Parallels Windows 7 VM for testing things in IE and testing from Windows in general. Other running software is usually Safari, Terminal, Sublime Text 2, Codebox, Source Tree, Sequel Pro, Adium. I run and quit Office 2011 every time I need to edit a document. I run and quit Aperture and Photoshop. Using this command line to see memory used by processes: ps -x -o rss,command | awk '{sum+=$1} END {printf "%9.3f %s\n", sum/1024, "MB" }' Gives 3000.535 MB for all of the above, 3235.000 MB after adding Word and Excel, or 3794.277 MB after adding Photoshop CS5 and iTunes streaming radio. Makes me wonder what you're doing to run out of memory.
- pooriaazimi 15y ago
  I'm skeptical about your little script (though I too have a lot of heavy apps open all the time and almost never reboot, and certainly have never, ever had any memory-related problems whatsoever). My MBP has 8 GBs of RAM, and this is what Activity Monitor tells me: Free: 2.19 GB Wired: 1.01 GB Active: 3.70 GB Inactive: 1.08 GB Used: 5.80 GB but, you scripts gives me: 2598.480 MB Why is that? Isn't Active kinda analogous to -o rss?
  - prakashk 15y ago
    I think that's because 'ps -x' command only gives your own processes and not all processes. Add '-a' option to show stats for all processes.
    - Terretta 15y ago
      You're right, typo. It's supposed to be -ax.
  - 4ad 15y ago
zeekay 15y ago
I have experienced OSX's "swappiness", having gone as far as disabling dynamic paging in an attempt to avoid it. Upgrading memory was the only real solution. A little bit of research would reveal a lot of other people have run into the same problem, you aren't alone at all in that. I split my time between OSX/Linux and it's pretty obvious to me that Linux is vastly superior in terms of performance, in a wide-range of scenarios. I prefer to use Linux on older and/or memory-constrained systems.

by jyrki 15y ago

jrmg 15y ago

The problem with the inactive memory is that it is not freed, it is swapped. So when hitting memory limits of my system, the computer started swapping. Just freeing the memory, in my case, would have been much quicker. Practically my machine was constantly swapping when the memory limit came up. That's just not true. If inactive memory is something that's already backed by disk (like a memory mapped file), it'll be discarded. Only if it's read-write memory that's /currently held allocated by a running process/ will it be swapped out to disk. Unless you're doing something pathological (like suddenly allocating lots of RAM and forcing paging - what purge utilities do...), the architecture /speeds things up/. If it's really true that, without doing anything special, things were always being swapped out to disk for you, it necessarily means that there was a process that had allocated (lots, it sounds like) of RAM and written to it, so that stuff had to be paged out to disk to free up RAM without losing data. It sounds like you were running purge commands or utilities to 'free up RAM'. That is counterproductive. It causes the system to release cached 'clean' (i.e. as already on disk - a mapped file, essentially) mapped RAM and swap read-write memory out to disk, only then have to re-read it all when you actually need it. In other words, using purge 'utilities' actually puts the system into the worst possible state.

mellifluousmind 15y ago
The problem with the way how OS X keeps data cached in the inactive memory is based on the assumption that you are going to re-use the same app within reasonable amount of time. With the current behavior/performance response (without knowing exactly how Apple engineers implemented it), it really feels like a giant garbage collection system that takes age to free up its own memory with no real sense of concept that if you don't use certain apps for day, chances are, it is going to take a long while before I use them again. I am one of the devs out there running a Macbook Pro with 8Gig of memory (I wish I could have more but I have a 2010 old model). For web development, I have at least Firefox/Chrome/PhpStorm/SmartGit/Mamp/Thunderbird/Terminal/Notational Velocity/Dropbox/Alfred/Sophos AntiVirus open at all times. Now, a long the way, I may open a few other apps that I use rarely, like Photoshop/CyberDuck/VMWare Fusion/iTune/iCal/iOS Simulator/Preview/LibreOffice/Skype. Now, pretty quickly 8Gig gets used up, and the system runs to the ground shortly after. If I then have VMWare Fusion shutdown for awhile, and relaunch later, the system really just can't take it anymore. The last resort? purge& At least, that's my day-to-day experience with OSX. Personally, I find the memory management really lousy, worse than other other OSes I used in the past (both Windows & Ubuntu/Fedora/Gentoo). So why the heck I use Mac still? Because of the driver support is still far better than Linux. With Mac, you are less likely in need of blacklist of some drivers because of freeze up issues. Either way, I am definitely not a happy camper with the current memory management system in OS X
- jrmg 15y ago
  The inactive count is a misleading figure in many ways. It includes both 'dirty' and 'clean' (i.e. identical to what's on disk, and hasn't been altered) memory that's not been used for a long time. Purging 'clean' memory is instantaneous. The only cost you'll see is writing 'dirty' memory out to disk. Your explicit purging is changing the cost of writing dirty data out from an ongoing cost to a single, longer, upfront cost. Instead of writing only when more RAM is required, you're forcing it all to happen at once. Incidentally, the OS does try to keep an area of free RAM so that some memory can be allocated instantly, it's not only swapping things out when RAM is absolutely full. It's possible though to outrun this process if an app tries to allocate huge amounts of RAM at once though (i.e. more than is kept free for this purpose). For your specific case, presuming apps are behaving well (see below) you would be better off quitting apps and relaunching them when you need them later. This will free up app the dirty RAM they've allocated (just like when you purge), but the 'clean' inactive RAM will not be purged (because that's not necessary - as I said, it's free to purge that kind of memory when it's needed for something else). You also want to run Activity Monitor when your system is in it's bad state and see if it's one of the apps you're using in particular that's allocating lots of memory (check out the "Real Mem" column). The OS can't do anything if it's an app that's really allocating and writing to memory, it's obviously not able to just discard this written-to memory. Really though, if you want to do all those things at once, more RAM might be required. Remember, with VMWare and the iOS Simulator running, you've got two whole other OSes running at the same time, it's reasonable they'd require lots of memory to work well!). By the way, the purge command was written to simulate /worst case/ conditions when performance testing. It's designed to flush out caches so that the system has to e.g. load all an app's code from disk when launching. [Source: I worked analysing this kind of thing at Apple until a couple of years ago].

RBerenguel 15y ago

I think the problem is that you used OS X for a year. I've been using my Mac for 4 years, and Linux for another 4. And I have Python (in fact I have at the same time Python 2.5, 2.6, 2.7, 3 and 3.1 for testing purposes), I have installed so many packages I already lost the count. You only need to be sure the path is correct when doing it. And I have 2 Gigs of RAM. Mac OS can be a bit hungry, but there's an "easy" solution for this. Closing unused applications. That bright dot in the dock means something is using an incredible amount of RAM just to sit there doing nothing. If I only leave open my mail app, Chrome, emacs and twitter client, I can go for days without ever hitting the disk cache. If I start adding more things (or tabs like I'm crazy), I'll hit the cache, of course. But this also happens in my Linux systems, it's not a Mac OS problem. As for the DVI port... I usually leave the DVI adapter linked to the monitor I use. And use bluetooth devices when possible. So far only once I have used a hub to connect 3 things to my Macbook (an SD reader to copy images, my Ben Nanonote to install some packages and my iTouch to sync with iTunes). Happened once, in 4 years (and I also have a drawing tablet, btw).

slowpoke 15y ago
>Closing unused applications. That's not a solution, that's a workaround. I'm sitting here with a 4GB RAM machine running Arch Linux with XMonad. I usually run the following applications in day to day usage: - Firefox Nightly with 50-100 tabs (largest memory hog, the rest doesn't even come close) - Thunderbird - Pidgin - XChat - smplayer, which I usually keep open - deluge - mpd + ncmpcpp - half to one dozen of terminator instances with zsh, most running some text mode applications like vim, htop or the aforementioned ncmpcpp I reboot my PC once in a blue moon, usually after kernel updates. otherwise it's running 24/7. Right now, I sit at ~35% memory (and that should include memory used for disk caching), a bit less than half of which is Firefox with ~16% (65 tabs). I usually never go over 50% unless compiling heavy stuff (like QT level heavy). I don't really know what the fuck OS X does to eat all that RAM, but it apparently does something wrong.
- lloeki 15y ago
  Emulating a setup similar to yours, the OSX machine infront of me sits a 2.8GB active, of which a little less than 800MB are taken by Terminal.app due to me having activated infinite scrollback and having two rails processes being hit 8 hours a day since like forever as I develop. So it's really closer to 2GB and I didn't even try hard. This includes mds+mdworker (the indexer) which clocks in at 200MB. Normally I also have LibreOffice, Pixelmator, iCal, iTunes, Reeder, VMWare Fusion, which makes it balloon to 3.1GB and up as I open more documents/VMs. OSX is not doing anything "wrong".

Terretta 15y ago

I do happen to have an 8 GB machine, and almost never reboot (only reboot for OS updates). I run half a menu-bar full of resident helper apps, like Dropbox (a big one), Fantastical, ScanSnap, Xmarks (another big one), Transmit (another big one), Evernote, and more. I also keep Apple Mail running, mapped to a half dozen Gmail IMAP accounts. I have "geek tool" updating my desktop with iCal appointments and various ps outputs. I'm running a local MAMP stack and local Django stack. I run a Parallels Windows 7 VM for testing things in IE and testing from Windows in general. Other running software is usually Safari, Terminal, Sublime Text 2, Codebox, Source Tree, Sequel Pro, Adium. I run and quit Office 2011 every time I need to edit a document. I run and quit Aperture and Photoshop. Using this command line to see memory used by processes: ps -x -o rss,command | awk '{sum+=$1} END {printf "%9.3f %s\n", sum/1024, "MB" }' Gives 3000.535 MB for all of the above, 3235.000 MB after adding Word and Excel, or 3794.277 MB after adding Photoshop CS5 and iTunes streaming radio. Makes me wonder what you're doing to run out of memory.

pooriaazimi 15y ago
I'm skeptical about your little script (though I too have a lot of heavy apps open all the time and almost never reboot, and certainly have never, ever had any memory-related problems whatsoever). My MBP has 8 GBs of RAM, and this is what Activity Monitor tells me: Free: 2.19 GB Wired: 1.01 GB Active: 3.70 GB Inactive: 1.08 GB Used: 5.80 GB but, you scripts gives me: 2598.480 MB Why is that? Isn't Active kinda analogous to -o rss?
- prakashk 15y ago
  I think that's because 'ps -x' command only gives your own processes and not all processes. Add '-a' option to show stats for all processes.
  - Terretta 15y ago
    You're right, typo. It's supposed to be -ax.
- 4ad 15y ago

zeekay 15y ago

I have experienced OSX's "swappiness", having gone as far as disabling dynamic paging in an attempt to avoid it. Upgrading memory was the only real solution. A little bit of research would reveal a lot of other people have run into the same problem, you aren't alone at all in that. I split my time between OSX/Linux and it's pretty obvious to me that Linux is vastly superior in terms of performance, in a wide-range of scenarios. I prefer to use Linux on older and/or memory-constrained systems.