3 ms·
Database engineer folks: I recently got to talk to someone who had written a query engine on the JVM One thing I thing they said is that: > "The buffer man
by gavinray 4y ago
Database engineer folks:
I recently got to talk to someone who had written a query engine on the JVM
One thing I thing they said is that:
> "The buffer manager should probably be rewritten to use the native OS page cache. When we wrote it originally this functionality wasn't easily available and so we used Java DMA (Direct Memory Access) instead."
I'm not familiar with what this means, would anyone be willing to explain more about OS page cache and how you'd implement something like that?
Would really appreciate it
- dharmab 4y agoThis article is a decent introduction to the page cache: https://lwn.net/Articles/457667/ https://lwn.net/Articles/457667/ You typically have to go out of your way to _not_ use the page cache.
- gavinray 4y agoThis looks super useful -- thank you!
- sakras 4y agoI’m not familiar with JVM things and what DMA is there, but usually when people talk about the OS page cache, they mean the in-memory file cache stored by the kernel. This means that if you read or write to a page in the cache, you’d be accessing memory instead of disk. The alternative is to open your file with O_DIRECT, which makes your reads/writes always interact with the storage system and bypasses the page cache.
- deleted 4y ago[deleted]
- cpleppert 4y agoI don't understand what that means unfortunately. I would think the second part of the comment refers to direct buffers in NIO that were added in java 1.4. One possibility is that he means directly interfacing with the operating system I/O system calls and using sun.misc.Unsafe to read or write to buffers.
- gavinray 4y agoIf it helps, here's the docs and source for what I'm referring to: https://docs.jboss.org/author/display/TEIID/Memory%20Management.html https://docs.jboss.org/author/display/TEIID/Memory%20Managem... https://github.com/teiid/teiid/blob/master/engine/src/main/java/org/teiid/common/buffer/impl/BufferManagerImpl.java https://github.com/teiid/teiid/blob/master/engine/src/main/j... https://github.com/teiid/teiid/blob/master/engine/src/main/java/org/teiid/common/buffer/BufferManager.java https://github.com/teiid/teiid/blob/master/engine/src/main/j...