3 ms·
This is a fantastic read, I highly recommend it if you like kernel performance stuff. I did much the same sort of work in SunOS ~25 years ago. Very similar, I
by luckydude 9y ago
This is a fantastic read, I highly recommend it if you like kernel performance stuff.
I did much the same sort of work in SunOS ~25 years ago. Very similar, I fixed the file system so we could run at platter speed and then the VM system couldn't keep up. I wrote over a dozen experimental pageout daemons before I came to the conclusion the stock one was as good as anything I could come up with.
I ended up "solving" the problem in the read ahead logic, when memory was starting to get tight I "freed behind" so this file wouldn't swamp memory. It was a crappy answer but it worked well enough. I suspect there is still an LMXXX in ufs_getpage() about it.
One thing that I did, that never shipped, was to keep track of how many clean/dirty pages were associated with a particular vnode. I wrote a "topvn" that worked like top but looked at vnodes instead of processes. Shannon nixxed, actually took it out of the kernel, because I added a vp->basename that was "a" basename
of the vnode. He didn't like that hardlinks created confusion so he shit canned the whole thing. If anyone has the SCCS history I'm pretty sure it's in there, thought might be 4.x (SunOS) instead of Solaris. I think the full set of things I added was
vp->basename;
vp->pages_clean;
vp->pages_dirty;
vp->last_fault; // timestamp of the last time we mapped/faulted/whatever this page
The advantage of that info is you can very very quickly find pages to be released. If a vnode is all dirty pages you skip that one, if it was all clean pages and last_fault is old, dump 'em all. It's way the heck faster than scanning each page.
- drewg123 9y agoThanks. That is a heck of a complement coming from you! I never used bitkeeper, but I've used lmbench since the 90s. Did you get so far as to implement a pageout daemon that scanned vnodes like that? We send only via sendfile, and we give the kernel hints about what should be freed using the SF_NOCACHE flag. This helps a lot. When we think we're serving something cold based on various weighed popularity rankings, we pass the SF_NOCACHE flag to sendfile(). This causes the page to be released immediately when the last mbuf referencing it is freed.
- luckydude 9y agoNo, I lost interest when Shannon pulled it out. I'm dancing with a company that might hire me to do some performance work in this area so maybe I'll redo that code. I don't think it is hard and I'd do it in FreeBSD. And bring back topvn. We've got so much memory these days it might help. But..... back in the day the top vnode was always swap because everyone used the same swap vnode. Howard Chartok (Mr Swapfs) and I discussed at length an idea of a swap vnode per process group. You want the set of processes that work together to use the same vnode so you have some sort of idea if it has gone idle. Just imagine the stats that the pageout daemon looks being summarized in the vnode, you want atime, mtime, dirty, clean, etc. I suspect for your workload the swap vnode isn't an issue. If the pageout daemon is still like it was then it's crazy. 4K pages, 128GB of ram, that's ~33 million pages to scan. If you summarize that you can find stuff to free really fast. And probably drop per file hints in there for the pageout daemon (like you are doing). Be happy to talk it over over a beer or something. I'm at the usual email addresses.
- wscott 9y agoSome of the LMXXX's yet remain: https://github.com/joyent/illumos-joyent/blob/master/usr/src/uts/common/fs/ufs/ufs_vnops.c https://github.com/joyent/illumos-joyent/blob/master/usr/src...
- luckydude 9y agoThanks. This is the one I was thinking of, unchanged decades later: https://github.com/joyent/illumos-joyent/blob/master/usr/src/uts/common/vm/vm_pvn.c https://github.com/joyent/illumos-joyent/blob/master/usr/src... 200 pages before the pager starts running maybe made sense back then, needs to be rethought now. I'm reading the code to see if anyone other that UFS uses it. Something called UDFS does but that looks like DVDs, not that interesting.