9 ms·

We've had a variant of this in the form of intel's persistent memory (named Octane DC memory or something stupid like this). It's addressable like RAM, but pers

We've had a variant of this in the form of intel's persistent memory (named Octane DC memory or something stupid like this). It's addressable like RAM, but persistent. Performance is not quite that of DRAM (within an order of magnitude), but reads and writes go through the normal memory cache so latency and throughput effects are often hidden by the cache. It's been a research topic to decide how to best use persistent byte addressable memory. Performance disparities are one problem, but the larger issue is actually one of correctness. Data structures are written to protect against race conditions in critical sections, but generally they are not written to maintain correctness in the event of a power failure and subsequent bring-up with some indeterminate persistent state (anything that went through cache is persistent, nothing else is. what was happening when the machine went down)? It's an interesting problem, lots of solutions. Most use a transactional interface--now doing this efficiently is the harder problem separate from but informed by the actual performance of the hardware.

stcredzero 5y ago
I know of "persistent" data structures from dabbling in Clojure. Just thinking off the top of my head just from that little bit of knowledge. If the very last thing that happens in an update, is a pointer update that "commits" the new data, it's correct if we assume RAM is dependable and the pointer update is atomic. What if there was a command which updated the value in the persistent store, then coerced a cache miss upwards through all of the cache levels? If the bottom level update at the persistent store is atomic, then it doesn't matter if the propagating cache misses are interrupted by a power failure. EDIT: But oh, the huge latency!
- aidenn0 5y ago
  Ordering is what kills you there; lets say you have update X and Y both in cache and for correctness they must happen in that order. If you just do a global cache flush, you lose if Y happens before X. So you need to flush after X and after Y. This means that every update to a persistent store involves hitting memory twice. This also means that every implementation of a data structure needs to do all of this work correctly. Filesystems have many fewer data structures than general purpose code, and they still occasionally have correctness bugs here. You might be able to salvage some performance by having a large write queue and running all cache as write-through, but that will make writes, on average, far more expensive than reads. Perhaps a more reasonable option would be to use only static memory for the caches and have a battery-backed coprocessor flush the caches on power-loss.
  - arghnoname 5y ago
    Well said. I will just add that there are cache write-through instructions and fence instructions. You can use these for just the pointer swap components the parent was talking about, but absolutely you’re correct, you can just take a naive data structure and have this work out of the box. There is work to make this transparent to existing data structures, but it (of course) imposes its own overhead.
- lichtenberger 5y ago
  Why not using DAX FS mode for Optane DC memory for instance and serialize the data structure by hand? I'm not sure how much performance you lose. That said the data store I'm working on serializes the huge persistent in-memory index to durable storage in a postorder traversal (mainly changed records in a data page Plus path copies) and atomically sets an offset to the main root, the UberPage to the new revision in a file. So, I'd say it's precisely what you describe :-)
mzs 5y ago
At least on PPC you can map write-through, that solves 99% of this on an embedded system with battery backed SRAM I use.
tinus_hn 5y ago
Does it have unlimited write cycles?
- arghnoname 5y ago
  No, it would last a decent bit if used ‘appropriately,’ but sadly in a heavy use environment it would be a consumable.
abraxas 5y ago
I think technology like this should force us to rethink how we do stuff at a fundamental level. Does the concept of treating "storage hardware" and "compute hardware" even make sense in this world? We should probably start rethinking everything we do from the OS layer to the way we write applications and exchange data. Truly performant non-volatile RAM would be (will be?) a revolution that rivals some of the earliest groundbreaking work in computer science.
- arghnoname 5y ago
  Definitely. You also have stuff like putting compute into the hardware. I read somewhere about SSDs with on-board compute doing serialization/deserialization of some data structures on the device itself, for instance. You can, in principle, take compute on board a NIC and if you could address the memory controller directly, you could serve NVRAM backed key-value store directly, without the CPU at all. Things are getting mushy and our way of addressing hardware resources and the OS abstractions behind them aren’t up to the task anymore. In the research world, due to various things about the kind of work that is feasible to do and get published in a reasonable timeframe, they just aren’t looking at these bigger questions as deeply as I think they ought to. People do more minor iterations on the existing approaches, which is a pity.
  - pishpash 5y ago
    That's not putting compute into hardware (i.e. for the first time), that's just building custom application-specific hardware, which happens all the time. You used to have the floating point unit outside the CPU.
- pmontra 5y ago
  Some fundamentals don't change. One one side there are processors (CPU, GPU, whatever) and on the other side we want to persistent some data for decades in some format that lets us exchange it with other people or multiple applications inside the same box. I don't expect ext4 and NTFS to go away soon. In between could totally change.
- pishpash 5y ago
torginus 5y ago
Imo, what kills Optane, is that it has limited write cycles, like an SSD, and because of that you can't treat it like RAM.
- jw14 5y ago
  If I'm reading this right, I think UltraRAM doesn't solve this problem. From the article: > its fast switching speed and program-erase cycling endurance is "one hundred to one thousand times better than flash." I know flash can get some impressive endurance now, but if you start treating it like RAM, is 1000x that endurance even enough?
  - chmod775 5y ago
    > I know flash can get some impressive endurance now, but if you start treating it like RAM, is 1000x that endurance even enough? Anything with limited endurance will need some kind of controller in front of it that makes sure that it wears out evenly. You'll end up with "copy-on-write" RAM that will be inherently slow/space inefficient, either due to block size being way too large for typical RAM access pattern or the data structure managing it becoming too large. And of course you incur extra latency from having to do look-ups in the first place.
- arghnoname 5y ago
  This is a problem. When I worked on it I didn’t treat it as RAM, but as something RAM would spill to. Essentially treating it as where memory mapped files are stored and then being clever about how those files are read/written. This reduces writes to once per modified byte address within a transaction (e.g., between msyncs).
twoodfin 5y ago
Several of the folks I've talked to who were most excited about Optane (and disappointed about its delayed introduction & market rejection) were interested primarily in the greater per-chip density, with persistence as at best a "nice-to-have". Take your typical in-memory database: The primary scalability constraint is how much "fast enough" memory you can park close to your CPU cores. Persistence lets you make some faster / broader failure & recovery guarantees, but if those are major application concerns, you probably wouldn't be using an in-memory database in the first place.
- arghnoname 5y ago
  That’s a good point. If I remember correctly, optane has a operating mode where the memory is only meant to be used as RAM. It’s stored on the non-volatile RAM in an encrypted form and the key is intentionally discarded/lost when rebooted. Another obvious operating mode is to make the NVRAM be the first line of swap and have things spill from DRAM into NVRAM when there is memory pressure. I think Intel also offered a mode like this, where it made everything look like DRAM and it would spill onto NVRAM as it needed to. Of course, this would be better rolled into the OS or application level so smarter decisions can be made, but given very high NVRAM densities it’s a great application for the technology.
- ksec 5y ago
  >and disappointed about its delayed introduction & market rejection Even at peak NAND and DRAM price ( at ~3x of normal ), where Optane had a slight chance of competing, Intel was still selling it at cost, with a volume that could not even fulfil their minimum purchase agreement with Micron. With a technical roadmap that doesn't fulfil their initial promise, and questionable cost reduction plan even at optimal volume. Compared that to DRAM ( DDR5 / DDR6 and HMB ) and NAND ( SLC / Z-NAND ), as much as I love the XPoint technology the cost benefits just didn't make any sense once DRAM and NAND falls back to normal pricing. Micron already know this all the way back in 2018, but the agreement with Intel kept them from writing it out clearly.
  - petra 5y ago
    I'm curious: The promise of Optane has pushed Intel's stock upwards. Did it's rejection by the market pushed the stock downward in a similar amount ?
lichtenberger 5y ago
Another comment already mentions this, but I guess a functional data structure would be easiest to maintain, even though there's stuff like this for PMEM such that any data structure would work essentially: https://pmem.io/ https://pmem.io/ This[1] Open Source data store I'm maintaining is storing a huge tree of tries eventually on durable storage, but it's essentially a huge persistent index stored in an append-only file. The last layer of "indirect index pages" stores list of pointers to page fragments, which store the actual records (the list however is bound to a predefinded size). A single read-write trx per resource syncs the new page fragments, which mainly only include the updated or new records plus the paths to the root page to a durable device during a postorder traversal of the in-memory log. The last thing is an atomic update, which sets a new offset to point to the new UberPage, the root of the storage. All pages are only word aligned except for RevisionRootPages, which are aligned to a multiple of 256 bytes. All pages are compressed and might in the future optionally be encrypted. I guess these byte addressable persistent memory devices are an ideal fit for small sized parallel random reads of potentially tiny page fragments to reconstruct a full page in-memory. However, I hope they'll eventually achieve a better price in comparison to DRAM and SSDs than Intel Optane DC PMEM current price range. [1] https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix
Dylan16807 5y ago
All of that stuff is really interesting from a technical perspective, but if the persistent memory isn't cheaper than RAM then 95% of machines won't need it, or won't need more than a small buffer of it. It'll be amazing for database servers and almost nobody else. And Optane failed that test.

stcredzero 5y ago

I know of "persistent" data structures from dabbling in Clojure. Just thinking off the top of my head just from that little bit of knowledge. If the very last thing that happens in an update, is a pointer update that "commits" the new data, it's correct if we assume RAM is dependable and the pointer update is atomic. What if there was a command which updated the value in the persistent store, then coerced a cache miss upwards through all of the cache levels? If the bottom level update at the persistent store is atomic, then it doesn't matter if the propagating cache misses are interrupted by a power failure. EDIT: But oh, the huge latency!

aidenn0 5y ago
Ordering is what kills you there; lets say you have update X and Y both in cache and for correctness they must happen in that order. If you just do a global cache flush, you lose if Y happens before X. So you need to flush after X and after Y. This means that every update to a persistent store involves hitting memory twice. This also means that every implementation of a data structure needs to do all of this work correctly. Filesystems have many fewer data structures than general purpose code, and they still occasionally have correctness bugs here. You might be able to salvage some performance by having a large write queue and running all cache as write-through, but that will make writes, on average, far more expensive than reads. Perhaps a more reasonable option would be to use only static memory for the caches and have a battery-backed coprocessor flush the caches on power-loss.
- arghnoname 5y ago
  Well said. I will just add that there are cache write-through instructions and fence instructions. You can use these for just the pointer swap components the parent was talking about, but absolutely you’re correct, you can just take a naive data structure and have this work out of the box. There is work to make this transparent to existing data structures, but it (of course) imposes its own overhead.
lichtenberger 5y ago
Why not using DAX FS mode for Optane DC memory for instance and serialize the data structure by hand? I'm not sure how much performance you lose. That said the data store I'm working on serializes the huge persistent in-memory index to durable storage in a postorder traversal (mainly changed records in a data page Plus path copies) and atomically sets an offset to the main root, the UberPage to the new revision in a file. So, I'd say it's precisely what you describe :-)

mzs 5y ago

At least on PPC you can map write-through, that solves 99% of this on an embedded system with battery backed SRAM I use.

tinus_hn 5y ago

Does it have unlimited write cycles?

arghnoname 5y ago
No, it would last a decent bit if used ‘appropriately,’ but sadly in a heavy use environment it would be a consumable.

abraxas 5y ago

I think technology like this should force us to rethink how we do stuff at a fundamental level. Does the concept of treating "storage hardware" and "compute hardware" even make sense in this world? We should probably start rethinking everything we do from the OS layer to the way we write applications and exchange data. Truly performant non-volatile RAM would be (will be?) a revolution that rivals some of the earliest groundbreaking work in computer science.

arghnoname 5y ago
Definitely. You also have stuff like putting compute into the hardware. I read somewhere about SSDs with on-board compute doing serialization/deserialization of some data structures on the device itself, for instance. You can, in principle, take compute on board a NIC and if you could address the memory controller directly, you could serve NVRAM backed key-value store directly, without the CPU at all. Things are getting mushy and our way of addressing hardware resources and the OS abstractions behind them aren’t up to the task anymore. In the research world, due to various things about the kind of work that is feasible to do and get published in a reasonable timeframe, they just aren’t looking at these bigger questions as deeply as I think they ought to. People do more minor iterations on the existing approaches, which is a pity.
- pishpash 5y ago
  That's not putting compute into hardware (i.e. for the first time), that's just building custom application-specific hardware, which happens all the time. You used to have the floating point unit outside the CPU.
pmontra 5y ago
Some fundamentals don't change. One one side there are processors (CPU, GPU, whatever) and on the other side we want to persistent some data for decades in some format that lets us exchange it with other people or multiple applications inside the same box. I don't expect ext4 and NTFS to go away soon. In between could totally change.
pishpash 5y ago

torginus 5y ago

Imo, what kills Optane, is that it has limited write cycles, like an SSD, and because of that you can't treat it like RAM.

jw14 5y ago
If I'm reading this right, I think UltraRAM doesn't solve this problem. From the article: > its fast switching speed and program-erase cycling endurance is "one hundred to one thousand times better than flash." I know flash can get some impressive endurance now, but if you start treating it like RAM, is 1000x that endurance even enough?
- chmod775 5y ago
  > I know flash can get some impressive endurance now, but if you start treating it like RAM, is 1000x that endurance even enough? Anything with limited endurance will need some kind of controller in front of it that makes sure that it wears out evenly. You'll end up with "copy-on-write" RAM that will be inherently slow/space inefficient, either due to block size being way too large for typical RAM access pattern or the data structure managing it becoming too large. And of course you incur extra latency from having to do look-ups in the first place.
arghnoname 5y ago
This is a problem. When I worked on it I didn’t treat it as RAM, but as something RAM would spill to. Essentially treating it as where memory mapped files are stored and then being clever about how those files are read/written. This reduces writes to once per modified byte address within a transaction (e.g., between msyncs).

twoodfin 5y ago

Several of the folks I've talked to who were most excited about Optane (and disappointed about its delayed introduction & market rejection) were interested primarily in the greater per-chip density, with persistence as at best a "nice-to-have". Take your typical in-memory database: The primary scalability constraint is how much "fast enough" memory you can park close to your CPU cores. Persistence lets you make some faster / broader failure & recovery guarantees, but if those are major application concerns, you probably wouldn't be using an in-memory database in the first place.

arghnoname 5y ago
That’s a good point. If I remember correctly, optane has a operating mode where the memory is only meant to be used as RAM. It’s stored on the non-volatile RAM in an encrypted form and the key is intentionally discarded/lost when rebooted. Another obvious operating mode is to make the NVRAM be the first line of swap and have things spill from DRAM into NVRAM when there is memory pressure. I think Intel also offered a mode like this, where it made everything look like DRAM and it would spill onto NVRAM as it needed to. Of course, this would be better rolled into the OS or application level so smarter decisions can be made, but given very high NVRAM densities it’s a great application for the technology.
ksec 5y ago
>and disappointed about its delayed introduction & market rejection Even at peak NAND and DRAM price ( at ~3x of normal ), where Optane had a slight chance of competing, Intel was still selling it at cost, with a volume that could not even fulfil their minimum purchase agreement with Micron. With a technical roadmap that doesn't fulfil their initial promise, and questionable cost reduction plan even at optimal volume. Compared that to DRAM ( DDR5 / DDR6 and HMB ) and NAND ( SLC / Z-NAND ), as much as I love the XPoint technology the cost benefits just didn't make any sense once DRAM and NAND falls back to normal pricing. Micron already know this all the way back in 2018, but the agreement with Intel kept them from writing it out clearly.
- petra 5y ago
  I'm curious: The promise of Optane has pushed Intel's stock upwards. Did it's rejection by the market pushed the stock downward in a similar amount ?

lichtenberger 5y ago

Another comment already mentions this, but I guess a functional data structure would be easiest to maintain, even though there's stuff like this for PMEM such that any data structure would work essentially: https://pmem.io/ https://pmem.io/ This[1] Open Source data store I'm maintaining is storing a huge tree of tries eventually on durable storage, but it's essentially a huge persistent index stored in an append-only file. The last layer of "indirect index pages" stores list of pointers to page fragments, which store the actual records (the list however is bound to a predefinded size). A single read-write trx per resource syncs the new page fragments, which mainly only include the updated or new records plus the paths to the root page to a durable device during a postorder traversal of the in-memory log. The last thing is an atomic update, which sets a new offset to point to the new UberPage, the root of the storage. All pages are only word aligned except for RevisionRootPages, which are aligned to a multiple of 256 bytes. All pages are compressed and might in the future optionally be encrypted. I guess these byte addressable persistent memory devices are an ideal fit for small sized parallel random reads of potentially tiny page fragments to reconstruct a full page in-memory. However, I hope they'll eventually achieve a better price in comparison to DRAM and SSDs than Intel Optane DC PMEM current price range. [1] https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix

Dylan16807 5y ago

All of that stuff is really interesting from a technical perspective, but if the persistent memory isn't cheaper than RAM then 95% of machines won't need it, or won't need more than a small buffer of it. It'll be amazing for database servers and almost nobody else. And Optane failed that test.