5 ms·

Eh. None of this is new, and we already anticipated this with LMDB back in 2009. The fact that NVRAM is directly addressable (and thus can bypass the page cach

Eh. None of this is new, and we already anticipated this with LMDB back in 2009. The fact that NVRAM is directly addressable (and thus can bypass the page cache) will eventually play out as irrelevant. It will always be a fact that slower mass-storage will exist, more cheaply than fast in-core storage, persistent or not. The page cache will still be needed even for NVRAM, and the "page" will still be the necessary atomic unit of memory interchange. (Direct access is of course a great thing, but it will be direct access to virtual addresses. Virtual memory, and paging in/out between primary and secondary storage, is never going away. Every commodity system that has tried to do without PMMUs has failed, for numerous good reasons.) The continual allusions to "frequent writes can destroy memory cells" seems to mainly relate to the extremely short lifetime of Intel's 3DXpoint memory, which is from every measure a total failure. http://semiaccurate.com/2017/03/10/intel-mislead-press-xpoint-next-week/ http://semiaccurate.com/2017/03/10/intel-mislead-press-xpoin... It would be best to ignore 3DXpoint and just focus on STT-MRAM, which is already at parity with DRAM for endurance. (But still lacking in density.) Much of the other stuff in those slides is still off the mark. E.g., LMDB today has perfect crash reliability with zero recovery time. The "write behind logging" they propose still has logging overhead and non-zero crash recovery time - which equals wasted work. Anything that requires logging or any form of compaction or garbage collection is wasted work. It's completely unnecessary, and that has nothing to do with NVM. Treating NVM DB design as if it's an entirely new and different animal is frankly ignorant. The right design works in all scenarios - as LMDB does.

cryptonector 9y ago
I'm not sure that using mmap(2) is enough. You're still writing pages, so you're writing more than you have to, so writes will not go as fast on NVM as they could. LMDB does logging since it does copy-on-write -- it just reuses free pages as soon as possible, which means that over time the log disappears. (LMDB is not an append-only DB.) I agree as to GCs. Having to GC is not just wasted work, it's a performance disaster, and any study that hand-waves about GC without actually measuring its impact on performance is fatally flawed. Assume petabytes of data, and assume performance between SSDs and DRAM: GC will still take enormous amounts of time and I/O that could have been used for something else. Write-behind logging is terrible if it means you need a GC. Why even bother with write-behind logging if a failure means that you must GC anyways. Only write-ahead logging helps you avoid a GC. A write-ahead log could be optimized to log only (address, length) tuples for each transaction so as to minimize WAL writes. However, the insights about smaller-than-page I/Os, particularly as to writes, seem likely to be correct. Though having anything like a b-tree on NVM with smaller-than-page writes seems to me to imply in-place writes, but I'm not ready to give up on COW.
- mtanski 9y ago
  > I'm not sure that using mmap(2) is enough. You're still writing pages, so you're writing more than you have to, so writes will not go as fast on NVM as they could. If you're using the new fangled memory there's a few components in play. First in Linux there's DAX. DAX lets you mmap in the devices pages directly without going through the page cache. And the processor can handle this like a memory mmaped device (without interactions from the OS) including subpage read/writes. Second, Intel chose to reuse previously existing operations that you would normally use for flushing cache lines (to main memory). Previously there were going to use additional pcommit operation. https://software.intel.com/en-us/blogs/2016/09/12/deprecate-pcommit-instruction https://software.intel.com/en-us/blogs/2016/09/12/deprecate-...
  - cryptonector 9y ago
    I got that. My point is that merely using this is insufficient to make your DB go fast on NVM. As the presentation says, you need to consider changing the on-disk^W^W^Wpersistent storage format to be one that writes less (not so much because of write cycle limits, but because writes are slower, and to reduce the number of additional writes needed in, say, a COW format.
10165 9y ago
"Every commodity system that has tried to do without PMMUs has failed..." I would like to read about these. Anyone have any pointers?
- DjvuLee 9y ago
  Yes, I am curious about this argument too. Is there any links related with this?
jarulraj 9y ago
Meta-observation: LMDB is a highly-optimized storage engine. However, it is not a full-featured DBMS (e.g., no support for query optimization). This talk focuses on the potential impact of NVM on different layers of a full-featured DBMS -- not only the storage engine. > The fact that NVRAM is directly addressable (and thus can bypass the page cache) will eventually play out as irrelevant. It will always be a fact that slower mass-storage will exist, more cheaply than fast in-core storage, persistent or not. Well, it depends on the latency of the NVM device. If the NVM device is directly addressable and has DRAM-like access latencies, I suspect that we must re-examine the assumptions underlying storage virtualization. Although I agree that the OS/DBMS would still need to maintain a page cache for data residing on slow block-addressable SSD/HDDs, I suspect that this will not be the case for data residing on fast byte-addressable NVM. > Anything that requires logging or any form of compaction or garbage collection is wasted work. It's completely unnecessary, and that has nothing to do with NVM. Treating NVM DB design as if it's an entirely new and different animal is frankly ignorant. The right design works in all scenarios - as LMDB does. If you think about it, logging is equivalent to versioning. To ensure atomicity and durability, a DBMS must maintain BEFORE and AFTER images of tuples. Unless it is a pure append-only system, the DBMS cannot refrain from reclaiming the storage space associated with OLDER tuple images. Note that these images can be stored either in the log or in multi-versioned heap storage. Maintaining a second B+tree to keep track of free pages, as is done is LMDB, is simply an implementation detail.

cryptonector 9y ago

I'm not sure that using mmap(2) is enough. You're still writing pages, so you're writing more than you have to, so writes will not go as fast on NVM as they could. LMDB does logging since it does copy-on-write -- it just reuses free pages as soon as possible, which means that over time the log disappears. (LMDB is not an append-only DB.) I agree as to GCs. Having to GC is not just wasted work, it's a performance disaster, and any study that hand-waves about GC without actually measuring its impact on performance is fatally flawed. Assume petabytes of data, and assume performance between SSDs and DRAM: GC will still take enormous amounts of time and I/O that could have been used for something else. Write-behind logging is terrible if it means you need a GC. Why even bother with write-behind logging if a failure means that you must GC anyways. Only write-ahead logging helps you avoid a GC. A write-ahead log could be optimized to log only (address, length) tuples for each transaction so as to minimize WAL writes. However, the insights about smaller-than-page I/Os, particularly as to writes, seem likely to be correct. Though having anything like a b-tree on NVM with smaller-than-page writes seems to me to imply in-place writes, but I'm not ready to give up on COW.

mtanski 9y ago
> I'm not sure that using mmap(2) is enough. You're still writing pages, so you're writing more than you have to, so writes will not go as fast on NVM as they could. If you're using the new fangled memory there's a few components in play. First in Linux there's DAX. DAX lets you mmap in the devices pages directly without going through the page cache. And the processor can handle this like a memory mmaped device (without interactions from the OS) including subpage read/writes. Second, Intel chose to reuse previously existing operations that you would normally use for flushing cache lines (to main memory). Previously there were going to use additional pcommit operation. https://software.intel.com/en-us/blogs/2016/09/12/deprecate-pcommit-instruction https://software.intel.com/en-us/blogs/2016/09/12/deprecate-...
- cryptonector 9y ago
  I got that. My point is that merely using this is insufficient to make your DB go fast on NVM. As the presentation says, you need to consider changing the on-disk^W^W^Wpersistent storage format to be one that writes less (not so much because of write cycle limits, but because writes are slower, and to reduce the number of additional writes needed in, say, a COW format.

10165 9y ago

"Every commodity system that has tried to do without PMMUs has failed..." I would like to read about these. Anyone have any pointers?

DjvuLee 9y ago
Yes, I am curious about this argument too. Is there any links related with this?

jarulraj 9y ago

Meta-observation: LMDB is a highly-optimized storage engine. However, it is not a full-featured DBMS (e.g., no support for query optimization). This talk focuses on the potential impact of NVM on different layers of a full-featured DBMS -- not only the storage engine. > The fact that NVRAM is directly addressable (and thus can bypass the page cache) will eventually play out as irrelevant. It will always be a fact that slower mass-storage will exist, more cheaply than fast in-core storage, persistent or not. Well, it depends on the latency of the NVM device. If the NVM device is directly addressable and has DRAM-like access latencies, I suspect that we must re-examine the assumptions underlying storage virtualization. Although I agree that the OS/DBMS would still need to maintain a page cache for data residing on slow block-addressable SSD/HDDs, I suspect that this will not be the case for data residing on fast byte-addressable NVM. > Anything that requires logging or any form of compaction or garbage collection is wasted work. It's completely unnecessary, and that has nothing to do with NVM. Treating NVM DB design as if it's an entirely new and different animal is frankly ignorant. The right design works in all scenarios - as LMDB does. If you think about it, logging is equivalent to versioning. To ensure atomicity and durability, a DBMS must maintain BEFORE and AFTER images of tuples. Unless it is a pure append-only system, the DBMS cannot refrain from reclaiming the storage space associated with OLDER tuple images. Note that these images can be stored either in the log or in multi-versioned heap storage. Maintaining a second B+tree to keep track of free pages, as is done is LMDB, is simply an implementation detail.