11 ms·

Preparing for nonvolatile RAM

76 points by willvarfar 14y ago

guard-of-terra 14y ago
What you really need is stick a virtual memory manager on top of NVM instead of RAM. In this case NVM works like disk and RAM as the same time. In your file system you just have a /ram file which is used as system memory. Then, fun things start to happen: You no longer need disk buffers, since your RAM is your disk. And mmap()ing no longer consumes "RAM" because you just map a part of your file to whatever virtual address you want. You need no swap. You never have to swap in or out. You never have to sync your disks or observe buffer dirtiness (if only on CPU cache level) because your buffer is a part of your file - once you write to it the file is already updated. Of course, software will have to adapt: prefer mmap()ing file to read()ing one; use persistent memory structures. The goal is to minimize copy-on-write by either read-only access or safe in place data transformation.
- carterschonwald 14y ago
  While I look forward to that future, realistically nvm ram is going to be at least as pricey as ssds or volatile ram, which means that a more likely in between approach that tries To minimize the amortized io between fast nvm (our ram replacement) and cheap and relatively slow nvm (hd or ssd).
  - guard-of-terra 14y ago
    If it would cost as SSDs do, it's pretty simple then: you mount NVM as /, and /var & /srv are still on disk. All your programs lose start-up time and most of swapping. If it would cost more, you can use it as a persistent block cache (that survives reboot), but it doesn't make much economic sense since nobody is willing to pay money for just faster start-up and warm-up after reboot. If you have 8G of NVM, you could just read 8G off disk in something like 30 sec and be settled with regular RAM.
gouranga 14y ago
Crazy futurist rant... Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force. Paradigm shift time. There is pretty much no reason to use any filesystem APIs or filesystem any more. You just keep your data in the process address space - its just not going to go anywhere. Just make a data segment persistent across processes and you can survive restarts. If you backup, you can just dump the address space. Screw hard disks as well. i imagine some form of rpc will be in place between processes so they can talk to each other and that is it. Lots of small redis instances would be a similar concept. Imagine an mp3 server process which can provide persistence in the heap, metadata and decoding services and you're there. it'll be like a small internet inside your machine. Lisp would fit nicely in this world. Imagine A persistent root environment. load a defun once and it's forever. Teracotta do something similar with java. Then again I could just be insane.
- guard-of-terra 14y ago
  Being able to reboot is nice in case you screw up really bad. Having a transparent storage (file system) instead of opaque cross-linked in-process structures is a good thing too.
  - gouranga 14y ago
    I disagree. If you use a safe programming language which doesn't piss over memory (I.e. Haskell, Python, Ruby, Lisp etc) you won't need to reboot. Just reload the broken function into your environment and carry on. As for data, the same thing can happen to your disk... The filesystem only exists because we couldn't put data in process due to the cost of memory. We cram everything through the filesystem API if it fits or not due to this. It's why our machines are slow and primitive. Consider the case of video - it is better represented as streams of audio, picture and metadata. To get this from a filesystem, we have to mux them all into a single stream and then demux them afterwards and hand them over to other APIs to process. This just wouldn't need to exist.
    - crusso 14y ago
raverbashing 14y ago
I will only believe this once it's sitting in my desk And, for now, it won't change filesystems much. Unless you can get a similar ammount of it as a disk (or maybe a compromise, let's say today around 8GB ram is common, and 1TB of HD, then if you can get around 128GB NVM this can be your new 'SSD') It is of course, a very important development, and may make things faster
- sp332 14y ago
  How about 240 GB almost as fast as DDR 200 (PC-1600) RAM? http://www.newegg.com/Product/Product.aspx?Item=N82E16820227744 http://www.newegg.com/Product/Product.aspx?Item=N82E16820227... Edit: according to reviews it uses compression to boost its bandwidth (but not capacity). Still seems like a decent tradeoff.
  - raverbashing 14y ago
    Wow, very tempting Too bad it doesn't fit my MBP =)
  - egillie 14y ago
    Crazy! I was looking into how much it would cost to get 500GB of RAM, looked like it was going to be around $3500. I hadn't even considered this...
- notatoad 14y ago
  You can easily run a desktop pc wih a full featured distro like Ubuntu desktop off a 16GB disk. You certainly don't need 128GB.
leoc 14y ago
Obigatory mention that the DG Nova http://en.wikipedia.org/wiki/Data_General_Nova http://en.wikipedia.org/wiki/Data_General_Nova was the star of http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine The Soul of a New Machine.
vicaya 14y ago
Commodity NVM is huge for big data transactions. Bye bye WAL Bye bye fsync Hello NVM replication I think I'm gonna cry
sweis 14y ago
Widespread adoption of NVRAM may require a significant change in security models, since data once assumed ephemeral may be persistent. For example, it may be trivial to recover cryptographic keys from a running system. (Disclaimer: I'm working on a solution to this.)
- kijin 14y ago
  This. I'm surprised nobody else mentioned security. Instead of chilling DRAM sticks immediately after shutdown and trying to read its contents with specialized tools, people could just take the NVRAM out and analyze it at leisure. I don't think it will be too difficult for security software to wipe their keys from memory before shutting down, and many programs already do this. But so much more would remain vulnerable unless the decrypted data structures were also wiped from memory. Implementing effective security with NVRAM-equipped computers might therefore negate much of the benefit of using NVRAM in the first place.
- aidenn0 14y ago
  There are already solutions out there for the paranoid who are afraid of cryogenic attacks on DRAM
  - sweis 14y ago
    Yes and no. There are some solutions out there, but they are typically not ready for production, require custom hardware, or make assumptions about physical security controls. (Disclaimer: I am biased since I am working on this.)
- gouranga 14y ago
  That's fairly easy to deal with - TPM module in PCs does this already.
  - sweis 14y ago
    The TPM does not provide full memory protection. TPMs are primarily used to attest the state of running software. It does not prevent someone from reading memory contents.
mmphosis 14y ago
What about processors that rather than accessing external RAM and levels of cache, instead a large amount of (register) memory (nonvolatile or not) is directly included within each CPU?
oh_sigh 14y ago
Can anyone tell me why we don't just hook normal RAM up to a small rechargeable battery such that it can maintain its state during a power loss. Alongside that, there is an equivalent amount of flash memory. The flash is never used, except when you get within, let's say, 5% battery life, at which point the entire contents of the RAM are dumped to the flash. Then, on system start up, if the RAM is still loaded, swell. If there is a ram image on the flash drive, load that and continue as normal. Isn't this essentially NVRAM? What are the downsides to it?
- shabble 14y ago
  We do, something like http://techreport.com/articles.x/16255 http://techreport.com/articles.x/16255 or http://www.anandtech.com/show/1697/5 http://www.anandtech.com/show/1697/5 although I don't think they actually have a full NV backing store. They definitely exist though, and have done even before SSDs, it just had a normal disk hanging off it. Battery-backed cache has probably been around even longer in write caches for large RAID systems. The 5% power and shutdown is also how the Macbook laptops handle sleep. Sleep mode is a low power nothing-but-ram mode, and when the battery gets too low, it goes into 'safe sleep', basically a dump-to-disk all-powered-off hibernate. The main reason it's not all that common is that for the sort of workloads you're prepared to pay for a shitload of RAM, you're probably just using it as a cache for a DB or some monster app, and actually keeping it around isn't that much of a priority. You've got failover somewhere else in the stack, and it's one less thing to buy and maintain. The other critical flaw is that there is a (potentially huge) performance hit in presenting as a disk vs hanging off the northbridge MMU. Even the latest in new fancy SATA is hilariously slow compared to the actual memory bus (6Gbps for SATA3 vs maybe 100Gbps for DDR3[1]), and having all the filesystem abstraction on top, as in the titular article of this thread mentions, is a whole lot more overhead. So yeah. We can. Sometimes people do. But it's probably easier and better to just stick it in the actual RAM slots, and use it differently for everything except 5-second boot times. [1] https://en.wikipedia.org/wiki/List_of_device_bandwidths#Storage https://en.wikipedia.org/wiki/List_of_device_bandwidths#Stor...
- toddh 14y ago
  If you want battery backed ram you might as well use a UPS. There are also corruption issues. A failure can leave data structures in an inconsistent state that applications typically don't worry about in RAM.

76 points by willvarfar 14y ago

What you really need is stick a virtual memory manager on top of NVM instead of RAM. In this case NVM works like disk and RAM as the same time. In your file system you just have a /ram file which is used as system memory. Then, fun things start to happen: You no longer need disk buffers, since your RAM is your disk. And mmap()ing no longer consumes "RAM" because you just map a part of your file to whatever virtual address you want. You need no swap. You never have to swap in or out. You never have to sync your disks or observe buffer dirtiness (if only on CPU cache level) because your buffer is a part of your file - once you write to it the file is already updated. Of course, software will have to adapt: prefer mmap()ing file to read()ing one; use persistent memory structures. The goal is to minimize copy-on-write by either read-only access or safe in place data transformation.

carterschonwald 14y ago
While I look forward to that future, realistically nvm ram is going to be at least as pricey as ssds or volatile ram, which means that a more likely in between approach that tries To minimize the amortized io between fast nvm (our ram replacement) and cheap and relatively slow nvm (hd or ssd).
- guard-of-terra 14y ago
  If it would cost as SSDs do, it's pretty simple then: you mount NVM as /, and /var & /srv are still on disk. All your programs lose start-up time and most of swapping. If it would cost more, you can use it as a persistent block cache (that survives reboot), but it doesn't make much economic sense since nobody is willing to pay money for just faster start-up and warm-up after reboot. If you have 8G of NVM, you could just read 8G off disk in something like 30 sec and be settled with regular RAM.

gouranga 14y ago

Crazy futurist rant... Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force. Paradigm shift time. There is pretty much no reason to use any filesystem APIs or filesystem any more. You just keep your data in the process address space - its just not going to go anywhere. Just make a data segment persistent across processes and you can survive restarts. If you backup, you can just dump the address space. Screw hard disks as well. i imagine some form of rpc will be in place between processes so they can talk to each other and that is it. Lots of small redis instances would be a similar concept. Imagine an mp3 server process which can provide persistence in the heap, metadata and decoding services and you're there. it'll be like a small internet inside your machine. Lisp would fit nicely in this world. Imagine A persistent root environment. load a defun once and it's forever. Teracotta do something similar with java. Then again I could just be insane.

guard-of-terra 14y ago
Being able to reboot is nice in case you screw up really bad. Having a transparent storage (file system) instead of opaque cross-linked in-process structures is a good thing too.
- gouranga 14y ago
  I disagree. If you use a safe programming language which doesn't piss over memory (I.e. Haskell, Python, Ruby, Lisp etc) you won't need to reboot. Just reload the broken function into your environment and carry on. As for data, the same thing can happen to your disk... The filesystem only exists because we couldn't put data in process due to the cost of memory. We cram everything through the filesystem API if it fits or not due to this. It's why our machines are slow and primitive. Consider the case of video - it is better represented as streams of audio, picture and metadata. To get this from a filesystem, we have to mux them all into a single stream and then demux them afterwards and hand them over to other APIs to process. This just wouldn't need to exist.
  - crusso 14y ago

raverbashing 14y ago

I will only believe this once it's sitting in my desk And, for now, it won't change filesystems much. Unless you can get a similar ammount of it as a disk (or maybe a compromise, let's say today around 8GB ram is common, and 1TB of HD, then if you can get around 128GB NVM this can be your new 'SSD') It is of course, a very important development, and may make things faster

sp332 14y ago
How about 240 GB almost as fast as DDR 200 (PC-1600) RAM? http://www.newegg.com/Product/Product.aspx?Item=N82E16820227744 http://www.newegg.com/Product/Product.aspx?Item=N82E16820227... Edit: according to reviews it uses compression to boost its bandwidth (but not capacity). Still seems like a decent tradeoff.
- raverbashing 14y ago
  Wow, very tempting Too bad it doesn't fit my MBP =)
- egillie 14y ago
  Crazy! I was looking into how much it would cost to get 500GB of RAM, looked like it was going to be around $3500. I hadn't even considered this...
notatoad 14y ago
You can easily run a desktop pc wih a full featured distro like Ubuntu desktop off a 16GB disk. You certainly don't need 128GB.

leoc 14y ago

Obigatory mention that the DG Nova http://en.wikipedia.org/wiki/Data_General_Nova http://en.wikipedia.org/wiki/Data_General_Nova was the star of http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine The Soul of a New Machine.

vicaya 14y ago

Commodity NVM is huge for big data transactions. Bye bye WAL Bye bye fsync Hello NVM replication I think I'm gonna cry

sweis 14y ago

Widespread adoption of NVRAM may require a significant change in security models, since data once assumed ephemeral may be persistent. For example, it may be trivial to recover cryptographic keys from a running system. (Disclaimer: I'm working on a solution to this.)

kijin 14y ago
This. I'm surprised nobody else mentioned security. Instead of chilling DRAM sticks immediately after shutdown and trying to read its contents with specialized tools, people could just take the NVRAM out and analyze it at leisure. I don't think it will be too difficult for security software to wipe their keys from memory before shutting down, and many programs already do this. But so much more would remain vulnerable unless the decrypted data structures were also wiped from memory. Implementing effective security with NVRAM-equipped computers might therefore negate much of the benefit of using NVRAM in the first place.
aidenn0 14y ago
There are already solutions out there for the paranoid who are afraid of cryogenic attacks on DRAM
- sweis 14y ago
  Yes and no. There are some solutions out there, but they are typically not ready for production, require custom hardware, or make assumptions about physical security controls. (Disclaimer: I am biased since I am working on this.)
gouranga 14y ago
That's fairly easy to deal with - TPM module in PCs does this already.
- sweis 14y ago
  The TPM does not provide full memory protection. TPMs are primarily used to attest the state of running software. It does not prevent someone from reading memory contents.

mmphosis 14y ago

What about processors that rather than accessing external RAM and levels of cache, instead a large amount of (register) memory (nonvolatile or not) is directly included within each CPU?

oh_sigh 14y ago

Can anyone tell me why we don't just hook normal RAM up to a small rechargeable battery such that it can maintain its state during a power loss. Alongside that, there is an equivalent amount of flash memory. The flash is never used, except when you get within, let's say, 5% battery life, at which point the entire contents of the RAM are dumped to the flash. Then, on system start up, if the RAM is still loaded, swell. If there is a ram image on the flash drive, load that and continue as normal. Isn't this essentially NVRAM? What are the downsides to it?

shabble 14y ago
We do, something like http://techreport.com/articles.x/16255 http://techreport.com/articles.x/16255 or http://www.anandtech.com/show/1697/5 http://www.anandtech.com/show/1697/5 although I don't think they actually have a full NV backing store. They definitely exist though, and have done even before SSDs, it just had a normal disk hanging off it. Battery-backed cache has probably been around even longer in write caches for large RAID systems. The 5% power and shutdown is also how the Macbook laptops handle sleep. Sleep mode is a low power nothing-but-ram mode, and when the battery gets too low, it goes into 'safe sleep', basically a dump-to-disk all-powered-off hibernate. The main reason it's not all that common is that for the sort of workloads you're prepared to pay for a shitload of RAM, you're probably just using it as a cache for a DB or some monster app, and actually keeping it around isn't that much of a priority. You've got failover somewhere else in the stack, and it's one less thing to buy and maintain. The other critical flaw is that there is a (potentially huge) performance hit in presenting as a disk vs hanging off the northbridge MMU. Even the latest in new fancy SATA is hilariously slow compared to the actual memory bus (6Gbps for SATA3 vs maybe 100Gbps for DDR3[1]), and having all the filesystem abstraction on top, as in the titular article of this thread mentions, is a whole lot more overhead. So yeah. We can. Sometimes people do. But it's probably easier and better to just stick it in the actual RAM slots, and use it differently for everything except 5-second boot times. [1] https://en.wikipedia.org/wiki/List_of_device_bandwidths#Storage https://en.wikipedia.org/wiki/List_of_device_bandwidths#Stor...
toddh 14y ago
If you want battery backed ram you might as well use a UPS. There are also corruption issues. A failure can leave data structures in an inconsistent state that applications typically don't worry about in RAM.