3 ms·

Has this pointer swizzling concept been adapted by any big data-focused databases? Would be interesting to see how this works with bigger pools of volatile and

Has this pointer swizzling concept been adapted by any big data-focused databases? Would be interesting to see how this works with bigger pools of volatile and persistent memory (disclaimer: I work at Intel on Optane DC SSD/PM-related things).

lichtenberger 7y ago
I'm not entirely sure, but I'll apply the technique for my open source storage system. I have reference-instances with a log-key, persistent-key (file offset for instance) and will (re-add) the Java object references. In order to set them to null when the page is going to be evicted from the buffer cache I'll add a simple in-memory map with the proposed child => parent mapping, such that I can set the in-memory page reference to null again.
lichtenberger 7y ago
BTW: What's your take on random read speed? I think with SSDs now and maybe NVRAM in the future we can address physical storage more fine granular than with HDDs and for instance with http://sirix.io http://sirix.io data is only clustered during writes (batched and synced to the flash drive) and it versions variable length in-memory and physical pages such that not every record has to be copied and written again. For instance to reconstruct a page we have to read from random places (maybe in parallel) to reconstruct a page in-memory. Thus Sirix relies on fast random read speed and on find granular access to the physical stored records.
- Rafuino 7y ago
  My take on random read speed? Check out this paper from Lenovo testing the performance of Optane DC Persistent Memory on their servers: https://lenovopress.com/lp1083.pdf https://lenovopress.com/lp1083.pdf. There's another good paper from a team at UCSD testing all sorts of usages and file systems with NVRAM here too: https://arxiv.org/pdf/1903.05714.pdf https://arxiv.org/pdf/1903.05714.pdf. Sirix sounds like a good fit to take advantage of App Direct access on Optane DCPM, but you're way more of an expert than me. Perhaps make a request for alpha access at GCP to test Sirix with the new tech? https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVioClFX5N-061jqO46vzpHzAPHkjwzVw/viewform https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVi...

lichtenberger 7y ago

I'm not entirely sure, but I'll apply the technique for my open source storage system. I have reference-instances with a log-key, persistent-key (file offset for instance) and will (re-add) the Java object references. In order to set them to null when the page is going to be evicted from the buffer cache I'll add a simple in-memory map with the proposed child => parent mapping, such that I can set the in-memory page reference to null again.

lichtenberger 7y ago

BTW: What's your take on random read speed? I think with SSDs now and maybe NVRAM in the future we can address physical storage more fine granular than with HDDs and for instance with http://sirix.io http://sirix.io data is only clustered during writes (batched and synced to the flash drive) and it versions variable length in-memory and physical pages such that not every record has to be copied and written again. For instance to reconstruct a page we have to read from random places (maybe in parallel) to reconstruct a page in-memory. Thus Sirix relies on fast random read speed and on find granular access to the physical stored records.

Rafuino 7y ago
My take on random read speed? Check out this paper from Lenovo testing the performance of Optane DC Persistent Memory on their servers: https://lenovopress.com/lp1083.pdf https://lenovopress.com/lp1083.pdf. There's another good paper from a team at UCSD testing all sorts of usages and file systems with NVRAM here too: https://arxiv.org/pdf/1903.05714.pdf https://arxiv.org/pdf/1903.05714.pdf. Sirix sounds like a good fit to take advantage of App Direct access on Optane DCPM, but you're way more of an expert than me. Perhaps make a request for alpha access at GCP to test Sirix with the new tech? https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVioClFX5N-061jqO46vzpHzAPHkjwzVw/viewform https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVi...