4 ms·
> Leaving memory management to the operating is nice idea - in reality it does not scale and does not play very well. This is why I think that Linus's tirade a
by haberman 15y ago
> Leaving memory management to the operating is nice idea - in reality it does not scale and does not play very well.
This is why I think that Linus's tirade against O_DIRECT is misguided: https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233
Here's the thing: the kernel is a library. It took me a long time to fully understand this deep idea. The kernel is just a library that has a different and more expensive calling convention (syscalls) and runs at a higher privilege level.
It's also much less flexible than user-space libraries. Its interface is an unholy mix of syscalls, ioctl(), /proc, vdso, etc. There is a high bar to adding new interfaces. Removing or changing existing interfaces is basically not allowed.
The resources that the kernel uses are much harder to account for or predict. How can you ensure that a process always gets at least X MB of page cache, and that some enormous "cp" that some sysadmin is running won't evict all your MongoDB pages that are caching your database? Sure you could mlock() your pages, but now you're basically side-stepping all of this smart kernel cache management that was supposed to be helping you so much in the first place.
User-space management of buffers and caches is more flexible, easier to account to its owner, and more predictable. It can't handle page faults with Linux's current interfaces, but the L4 guys have figured out how to let pagers run in user-space and handle page faults. I hope that someday this work becomes mainstream. Our 20-year-old OS design is showing its age.
- MattRogish 15y agoThis is why most DBMS products do their own memory management. They just suck up as much memory as you will allow and use it for their own devices. Specialized, tunable, application-level memory management will probably always beat a general-purpose, application-ignorant OS-based scheme. But, there's something to be said for the simplicity - for most folks you don't need to manage the memory yourself. When you need it though, there's really no easy substitute.
- haberman 15y ago> But, there's something to be said for the simplicity - for most folks you don't need to manage the memory yourself. Yes, and most people don't need to implement printf() themselves, which is why there is libc. Just because you're doing it in-process, in user space, doesn't mean you're rolling your own!
- cynoclast 14y agoSo does Java, for instance.