4 ms·

In-Memory Performance for Big Data (2014) [pdf]

29 points by lichtenberger 8y ago

cl0ckt0wer 7y ago
Had to look this up: https://en.wikipedia.org/wiki/Pointer_swizzling https://en.wikipedia.org/wiki/Pointer_swizzling
- petergeoghegan 7y ago
  The term pointer swizzling isn't familiar to me, but it seems to be the same technique by which variable-width tuples on a page are accessed via a fixed width item pointer array (sometimes called an indirection vector) within most database systems.
mpweiher 7y ago
Interesting, sounds a bit like the way write-barriers made the combination of copying garbage collectors and direct object pointers feasible.
Rafuino 7y ago
Has this pointer swizzling concept been adapted by any big data-focused databases? Would be interesting to see how this works with bigger pools of volatile and persistent memory (disclaimer: I work at Intel on Optane DC SSD/PM-related things).
- lichtenberger 7y ago
  I'm not entirely sure, but I'll apply the technique for my open source storage system. I have reference-instances with a log-key, persistent-key (file offset for instance) and will (re-add) the Java object references. In order to set them to null when the page is going to be evicted from the buffer cache I'll add a simple in-memory map with the proposed child => parent mapping, such that I can set the in-memory page reference to null again.
- lichtenberger 7y ago
  BTW: What's your take on random read speed? I think with SSDs now and maybe NVRAM in the future we can address physical storage more fine granular than with HDDs and for instance with http://sirix.io http://sirix.io data is only clustered during writes (batched and synced to the flash drive) and it versions variable length in-memory and physical pages such that not every record has to be copied and written again. For instance to reconstruct a page we have to read from random places (maybe in parallel) to reconstruct a page in-memory. Thus Sirix relies on fast random read speed and on find granular access to the physical stored records.
  - Rafuino 7y ago
    My take on random read speed? Check out this paper from Lenovo testing the performance of Optane DC Persistent Memory on their servers: https://lenovopress.com/lp1083.pdf https://lenovopress.com/lp1083.pdf. There's another good paper from a team at UCSD testing all sorts of usages and file systems with NVRAM here too: https://arxiv.org/pdf/1903.05714.pdf https://arxiv.org/pdf/1903.05714.pdf. Sirix sounds like a good fit to take advantage of App Direct access on Optane DCPM, but you're way more of an expert than me. Perhaps make a request for alpha access at GCP to test Sirix with the new tech? https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVioClFX5N-061jqO46vzpHzAPHkjwzVw/viewform https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVi...

29 points by lichtenberger 8y ago

cl0ckt0wer 7y ago

Had to look this up: https://en.wikipedia.org/wiki/Pointer_swizzling https://en.wikipedia.org/wiki/Pointer_swizzling

petergeoghegan 7y ago
The term pointer swizzling isn't familiar to me, but it seems to be the same technique by which variable-width tuples on a page are accessed via a fixed width item pointer array (sometimes called an indirection vector) within most database systems.

mpweiher 7y ago

Interesting, sounds a bit like the way write-barriers made the combination of copying garbage collectors and direct object pointers feasible.

Rafuino 7y ago

Has this pointer swizzling concept been adapted by any big data-focused databases? Would be interesting to see how this works with bigger pools of volatile and persistent memory (disclaimer: I work at Intel on Optane DC SSD/PM-related things).

lichtenberger 7y ago
I'm not entirely sure, but I'll apply the technique for my open source storage system. I have reference-instances with a log-key, persistent-key (file offset for instance) and will (re-add) the Java object references. In order to set them to null when the page is going to be evicted from the buffer cache I'll add a simple in-memory map with the proposed child => parent mapping, such that I can set the in-memory page reference to null again.
lichtenberger 7y ago
BTW: What's your take on random read speed? I think with SSDs now and maybe NVRAM in the future we can address physical storage more fine granular than with HDDs and for instance with http://sirix.io http://sirix.io data is only clustered during writes (batched and synced to the flash drive) and it versions variable length in-memory and physical pages such that not every record has to be copied and written again. For instance to reconstruct a page we have to read from random places (maybe in parallel) to reconstruct a page in-memory. Thus Sirix relies on fast random read speed and on find granular access to the physical stored records.
- Rafuino 7y ago
  My take on random read speed? Check out this paper from Lenovo testing the performance of Optane DC Persistent Memory on their servers: https://lenovopress.com/lp1083.pdf https://lenovopress.com/lp1083.pdf. There's another good paper from a team at UCSD testing all sorts of usages and file systems with NVRAM here too: https://arxiv.org/pdf/1903.05714.pdf https://arxiv.org/pdf/1903.05714.pdf. Sirix sounds like a good fit to take advantage of App Direct access on Optane DCPM, but you're way more of an expert than me. Perhaps make a request for alpha access at GCP to test Sirix with the new tech? https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVioClFX5N-061jqO46vzpHzAPHkjwzVw/viewform https://docs.google.com/forms/d/e/1FAIpQLSeX1tN6Qt-aQUK2iVVi...