36 ms·
Intel Announces Optane DIMMs
- Aardwolf 8y agoSo it's like putting an SSD in your RAM slots? But what about speed, and especially amount of write cycles?
- ukblewis 8y agoNope, because the whole thing about Optane is that it is super low latency, like RAM, so yes, it’s slower to actually perform read-writes, but way faster than even an SSD disk for random reads making it reasonable suitable as a RAM replacement.
- Moral_ 8y agoIt should also be noted for those saying "well the Optane ssd's are fast but not THAT fast" A large majority of the latency from those drives is the PCIe layer.
- dmix 8y agoI believe the whole idea, beyond pure speed of RAM vs SSDs, is they are promoting a programming 'model' where you bypass the operating system I/O (transiting via pages/blocks) and instead directly read/write data (via bytes) to the memory from the application. Which could be useful for particular usecases / subsets of data. They mention write cycles in the article... > The existing enterprise Optane SSD DC P4800X initially launched with a write endurance rating of 30 drive writes per day (DWPD) for three years, and when it hit widespread availability Intel extended that to 30 DWPD for five years. Intel is now preparing to introduce new Optane SSDs with a 60 DWPD rating
- JohannFlobuster 8y agoThe idea is to drive CPU utilization as well. For example, in the x86 world, pairing NVMe drives with putting portions of the application writing to NVDIMM drives core performance, say, in SQL, from 40% util to 100% util. Even a single 8Gb DIMM can dramatically increase utilization and performance.
- monocasa 8y agoAnd literally half the bandwidth. NVMe means that you have to write the data, flush the caches over those ranges to ensure that it's actually in DRAM, then instruct the NVMe controller to read it back out of DRAM. With these drives you just write (and maybe flush), and it's done. And they probably have their own dedicated DRAM protocol controller. And it might even be slightly better than halving the bandwidth needs, since swapping DRAM banks isn't free, so you might be saving on mildly thrashing your DRAM controller when you're using DRAM and the NVMe drive is trying to read at the same time.
- xorblurb 8y agoModern DMA is cache coherent (maybe except if you opt-out of it? I'm not even sure you can). It is still costly though.
- deleted 8y ago[deleted]
- Symmetry 8y agoThat level of endurance is great if you've got wear leveling routines like a SSD has but not if writes to the same logical cache line hit the same physical block each time.
- monocasa 8y agoDoes 3D xpoint have a need for wear leveling? It's a fundamentally different technology than NAND flash. The numbers stated are very close to what you get out of spinning rust in the enterprise space.
- wmf 8y agoIt does use wear leveling, especially since the quoted endurance is far lower than promised.
- jsight 8y agoYeah, I am really curious as to the actual read/write performance as well. The write cycles are covered in the article, though without much detail as to how wear leveling works with this kind of setup.
- AsianScum 8y agoPlease measure the latency too.
- ukblewis 8y agoLol, looks like Linus actually inspired them. I mean with the whole DDR industry acting corruptly to keep prices sky high, this seems like a reasonable way to add some competition.
- white-flame 8y agoThere's no reason to believe this won't be sky-higher than current DDR4 RAM prices.
- wmf 8y agoIntel is saying it will be cheaper but they've raised prices before.
- lambda 8y ago> Lol, looks like Linus actually inspired them How so? Point to the quote?
- pitaj 8y agoLinusTechTips made a video recently taking about this. https://youtu.be/cwy4ujt0qHM https://youtu.be/cwy4ujt0qHM
- deleted 8y ago[deleted]
- linux2647 8y agoTotally thought GGP was referring to Linus Torvalds
- lambda 8y agoIt is quite unlikely that Intel was actually inspired by Linus of LinusTechTips. He's exaggerating for dramatic effect, just like the fake drama of not being able to find his RAM. Persistent memory is something that Intel has been working on for at least 5 years (https://github.com/pmem https://github.com/pmem), and given that that's the public face of the software side of it, they were likely starting to develop the hardware even earlier.
- whitepoplar 8y agoHow would Optane-on-DIMM affect the performance of relational databases as compared to the NVMe variant of Optane?
- en4bz 8y agoAs far as I know nvme is exclusively block based. So you get to write in 512 or 4K blocks. I believe the DIMM versions are byte addressable or at least they will be at some point in the future.
- 21 8y agoI'm pretty sure the smallest addressable unit of RAM in modern computers is a whole cache line (64 bytes).
- rrix2 8y agoWhile being technically correct, that is still much smaller than 512 or 4K blocks.
- deleted 8y ago[deleted]
- CyberDildonics 8y agoActually 64 bytes is the false sharing boundary - on modern CPUs addressing a single byte not in cache will pull down 128 bytes.
- fsaintjacques 8y agoAFAIK, the Optane disks are _byte_ addressable.
- dragontamer 8y agoNot even DDR4 RAM is byte-addressable. DDR4 is typically burst-length 8 for 64-bytes per burst (although BL2 exists, I'm fairly certain all modern processors have settled on BL8)
- ksec 8y agoSo we could fit, 512GB DIMM, at up to 8 TB of Optane, and using them like in~memory DB with persistence? I wonder what performance improvement would I get on a Postgre DB compare to RAM + NVMe SSD.
- derefr 8y agoA lot of the overhead there might come from Postgres (e.g. building temporary in-memory indices for hash-joins, because it's assuming random access to the tablespace is slow.) Ideally you have a DBMS that already understands that it's running on nonvolatile memory, and so doesn't have separate "in memory" and "on disk" formats for its data. Maybe something like Aerospike? (don't know much about it but I've heard it's "for" that)
- ovao 8y agoOr some LMDB-backed database, sure. With LMDB you'd essentially map the entire address space of the disk, then just persist pointers. Writes would still not be as performant as you would hope, but random read performance would be in the small handfuls of nanoseconds per.
- jdiaz5513 8y agoRunning Redis would be pretty fun on this.
- pbalcer 8y agoJust running Postgres on an NVDIMM, or any other database for that matter, will give you a large speedup compared to normal storage, especially when you have strong durability requirements for a transaction (fsync() will be faster). But to fully benefit from persistent memory, the DBMS will need to modified. To see how that might look, read this [0] post by Microsoft about their efforts in SQL Server. There's also an interesting research paper from CMU [1] that talks about challenges associated with pmem in the context of databases. [0] - https://blogs.msdn.microsoft.com/bobsql/2016/11/08/how-it-works-it-just-runs-faster-non-volatile-memory-sql-server-tail-of-log-caching-on-nvdimm/ https://blogs.msdn.microsoft.com/bobsql/2016/11/08/how-it-wo... [1] - https://www.cs.cmu.edu/~jarulraj/papers/2015.storage.sigmod.pdf https://www.cs.cmu.edu/~jarulraj/papers/2015.storage.sigmod....
- patrickg_zill 8y agoSo will regular RAM become "level 4 cache" and Optane exist on the other side of that? Basically you run everything you can in memory and then just mmap() in the files you want to use?
- CyberDildonics 8y agoI would say that it basically already is.
- marshray 8y agoBut that's how we do it now. It's just that Optane persistent storage is attached to a different bus with much better latency and throughput. Probably categorically and disruptively better. How will user-mode applications refer to their persistent data if not by a filesystem path? You gotta put access permissions on some kind of object that humans can copy-and-paste into their backup scripts.
- theandrewbailey 8y agoWhat's the API for these Optane DIMMs? Does the program decide what gets placed in them? Does the OS? (if so, how?)
- wmf 8y agohttp://pmem.io/ http://pmem.io/
- justincormack 8y agoTo the OS, its just memory and you can just mmap it. There are some filesystem abstractions built on top though eg https://lwn.net/Articles/729770/ https://lwn.net/Articles/729770/
- jnwatson 8y agoWhat about multithreading? An OS can read from an SSD then context switch to do useful other stuff until the data comes back. Once the OS issues a request to (relatively) slow memory, the only thing that can use that core are other instructions in the pipeline and other hyperthreads; i.e no other OS threads allowed.
- namibj 8y agoYes. But the Optane chips are fast enough for that to be only a minor nuisance.
- wtallis 8y agoThe fastest NVMe SSDs are already right around the threshold where there isn't time to complete a pair of context switches before you get the data back, and the difference in latency between polling or waiting for an interrupt is significant. These Optane DIMMs should be fast enough that only hardware-managed context switches like hyperthreading/SMT are usable without performance loss.
- JudasGoat 8y agoWe never really worried about write endurance with dynamic RAM, but it would seem to be a lot busier than disk I/O at least in some server applications.
- marshray 8y agoWe need reviewers report a new metric: time to failure at continuous max write throughput. For some SSDs, it's under a week.
- Moral_ 8y agoThe write to failure is usually reported in the drive specs. https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/optane-dc-p4800x-series/p4800x-750gb-aic.html https://www.intel.com/content/www/us/en/products/memory-stor... See Endurance Rating (Lifetime Writes)
- marshray 8y ago"Endurance Rating (Lifetime Writes) 41.0 PBW" "Sequential Write (up to) 2200 MB/s" 41e15 / 2.2e9 /60/60/24 = 215 days sequential write to failure "Mean Time Between Failures (MTBF) 2 million hours" 2e6 / 24/365.25 = 228 years MTBF So it seems the MTBF is being stated at 0.26% average write utilization. [corrected math]
- mjevans 8y agoFor some applications that might be worth it.
- deleted 8y ago[deleted]
- Retric 8y ago0.26% * 2200 MB/s = 5.72 MB/s for 24/7/365 which seems about right for most users. A 750 GB drive assuming you want to store the data for 48 hours can only write (750 GB/24/60/60*1000) = 4.34 MB/s on average. Dropping that to even 1 hour still gives reasonable lifetimes.
- bitL 8y agoHow does it compare to SanDisk ones? Is this going to enable HP's The Machine without memristors?
- onli 8y agoI don't really understand how this is supposed to work. These are DIMMs, meaning they are ram alternatives? Will they be cheaper and slower than DDR4, but persistent and getting benefits from that? Which advantages exactly? Wouldn't the OS need to be aware of that, defeating the point of them being DIMMs? 128GB, 256GB and 512GB per module is sadly too much for consumer motherboards. Why not a 16GB version, didn't Intel even launch Optane with those small sizes?
- dragontamer 8y agoThere's a new slot called "NVDIMM", where "non-volatile" memory can be plugged into the motherboard so that the CPU can interact with it. Its cheaper and slower than DDR4, but persistent and likely to be way denser. > 128GB, 256GB and 512GB per module is sadly too much for consumer motherboards. NVDIMMs are for database applications. If you were running 1TB in-memory databases, but are willing to lose a bit of performance to severely reduce costs, you're in the market for an NVDIMM.
- BeefySwain 8y agoMy understanding is that all these have in common with RAM is sharing the DIMM connector. There are a number of technologies that use DIMM connectors that are not RAM, for example, the Raspberry Pi Compute Module[0] uses a SO-DIMM connector to save space and money by not having to break out IO nor provide any sort of connectors. [0] https://www.raspberrypi.org/products/compute-module-3/ https://www.raspberrypi.org/products/compute-module-3/
- wtallis 8y agoIt's not just the connector. The electrical interface is DDR4, and it's accessed through the CPU's DRAM controller as part of the existing memory hierarchy. It's just that the memory addresses corresponding to the Optane DIMMs will be slower than the addresses corresponding to DRAM. New CPUs will be required, and if there's a technical justification for that it will probably be that accessing an Optane DIMM requires timings that are far outside the normal range for DRAM modules that the existing memory controllers were designed to accommodate.
- xchip 8y ago"640K ought to be enough for anybody." (apparently 4 people in HN hated this comment)
- kibwen 8y agoAnybody want to give a brief comparison of how these compare in practice to DDR4 and SSD storage? I assume it's "slower than the former, faster than the latter", but having an idea of the magnitudes would be useful.
- konceptz 8y agoLinus did a video about Optane I found helpful. https://youtu.be/cwy4ujt0qHM https://youtu.be/cwy4ujt0qHM
- djsumdog 8y agoThis video is about the Optane devices you place in M2/nvme slots to be used as cache. It will be interesting to see the real benchmarks on this DIMMS. We all want to know if they're comparable to real DDR4 memory. The current market is terribly overpriced (there's some debate on if there's price fixing with the big three or if it's a genuine shortage/supply problem with the Note recalls and new phone releases). DDR4 is nearly double the price it was the last time I did a build over a year ago. :-/ EDIT: Looks like these chips will be specialized for certain server boards/CPUs and only share the DIMM interface and not protocol.
- kibwen 8y agoThanks for the clarification, I'm watching the video now and it's interesting (Optane is way cheaper than I expected) and still seems useful as a lower bound for what we can expect from the DIMM version.
- fpoling 8y agoThanks for the link! The Blender benchmark is really telling. Using Optane M2 modules as a swap disk allowed to run a task that required 12GB on a system with 4 GB of RAM and it took just 1.8 more in time then the native RAM case. So I guess I can put those Optane modules into a laptop with 16GB of RAM and run a calculation that requires 64GB of memory and it will be much cheaper than a laptop with 64GB of RAM.
- ChuckMcM 8y agoThese are going to change things for a lot of reasons, not the least of which is significantly higher transaction rates in systems. In particular being past a memory barrier means that your cache is consistent and persistent across reboots in nanoseconds rather than milliseconds or even microseconds means large transnational systems can do more operations at up to 6 orders of magnitude faster. In file systems, and especially so called 'IRON' or error corrected systems, this will significantly boost the speed of operations as more writes can be delayed safely to insure that dense spinning media is writing sequentially rather than randomly. Using the old NetApp cluster architecture I expect you could get a couple of gigabytes per second of random (from the clients) R/W performance on a very reasonable number of drives. Of course it also makes it possible to rapidly reboot into an OS if all you need in the boot loader is to copy a gigabyte of 'ram' from one place to another and then jump to it. What that enables is powering down servers that aren't being used with the ability to power them up in milliseconds rather than seconds. That has been of Google and other cloud providers to have 'power proportionate computing' clusters. Very cool stuff indeed.
- baq 8y agofor rebooting, you'd have to have super-fast POST and device initialization after power cycle, so i wouldn't say milliseconds.
- linsomniac 8y agoDefinitely. Most of my Dell machines take 2-5 minutes to get to the "booting OS" part of the process, but a couple of them have at times taken 3 hours. The latter is probably a firmware bug, but the best Dell ProSupport has been able to do to solve it is "flea drain" (unplug, hold the power button for 60 seconds, plug back in).
- arghwhat 8y agoDell servers are notoriously slow to boot, but this is due to the remote management subsystem being extremely slow. It's quite a pain in the arse, and one of the reasons we prefer to use Supermicro machines for development at work, using only Dell for testing.
- reacharavindh 8y agoI can already imagine storage systems where the blocks are just written to memory before acknowledging as a synchronous commit, and reads occurring completely from this cache or for one direct hop from this cache. It's going to be amazing in the future as this tech matures. A persistent RAM will change a lot of our architectures for the better.
- wmf 8y agoThat's pretty much how SANs used to work.
- reacharavindh 8y agoIf you're referring to NetApp's use of NVRAM, to take in writes, and acknowledging the IO before actually writing to disk, it is not that simple. They do NVRAM mirroring to save those writes to another partner's NVRAM for HA. This is a network tax on core IO path. If there is a mature Optane DIMM, it may not be necessary to do the network dance in IO path. I sincerely hope no storage company acknowledges a write after just writing to volatile main memory. That's a recipe for disaster when a node goes down.
- voodootrucker 8y agoI don't know if it's SAN specific, but while waiting for XPoint, several vendors went to battery-backed DRAM. They confirm writes when in RAM, then whenever power is lost all RAM is dumped to flash, then loaded back up on boot: https://en.wikipedia.org/wiki/NVDIMM https://en.wikipedia.org/wiki/NVDIMM Fortunately XPoint should solve this.
- atomicUpdate 8y ago> I sincerely hope no storage company acknowledges a write after just writing to volatile main memory. That's a recipe for disaster when a node goes down. Sorry to break this to you, but this is exactly what every major storage product does. Data will be mirrored to separate DDR behind and then good status is given to the host. The data will be destaged at some point later when cache space is required for some other operation. The data itself is safe as long as the battery backup (or capacitor for smaller systems) is charged enough to handle a power outage. The storage system knows the battery levels and may not allow a write cache if there isn't enough supplemental power to destage the full write cache in the event of a power loss.
- AsianScum 8y agoPersistent Memory age is coming. Can we install NVDIMM into a desktop?
- pbalcer 8y agoThe product being announced is for data centers.
- libeclipse 8y ago> Optane DC Persistent Memory DIMMs have twice the error correction overhead of ECC DRAM modules. That is pretty impressive.
- rurban 8y agoSee also the timeline of those PCM, based on chalcogenide glass, which were invented in the 60ies by Stanford_R._Ovshinsky. He always insisted that it's better than DRAM. Finally we are getting there. https://en.wikipedia.org/wiki/Phase-change_memory#Timeline https://en.wikipedia.org/wiki/Phase-change_memory#Timeline Theoretically it should be 1000x more durable than flash, and also 1000x faster, but they are not there yet. But it looks like they solved the packing problem. And Micron insists that it is chalcogenide based, but not "phase-change memory", the one they started in 2012 and took back in 2014.
- mattthebaker 8y agoPCM to Micron is an existing product/design that is NOR-Flash like but utilizes PC material as the storage medium. 3D XPoint is said to be able to stack the storage medium on the die and requires no access transistor -- which "PCM" has. So it is based on PC material but has some new kind of selector. The 3D/stacking part is important for scaling/density -- even NAND-Flash has hit 2d limits and gone vertical.
- deleted 8y ago[deleted]
- blt 8y agoReading this thread, it seems like there are doubts about whether this particular product will really work like non-volatile RAM. Either way, it's an indicator that the real thing will be here soon. I can't wait to see how fast persistent memory will change the architecture of our computing environments. I think it will have huge effects that we can't foresee yet.
- magoon 8y agoOh, the security implications of this... e.g. malloc owner PID x is now PID y
- bbulkow 8y agoI was on the outside vendor was panel at the event (along with Oracle and redislabs) if anyone would like an engineering view of the technology.
- wmf 8y agoEverything we want to know is exactly what you're not allowed to talk about.
- georgeaf99 8y agoSimon Peter, at UT Austin, has been working on a file system that works better for systems with NVM. It looks like existing file systems (e.g. EXT4) are going to suck in this new paradigm (DRAM -> NVM -> SSD) . It’s a pretty interesting read and the benchmarks are damn impressive. https://www.cs.utexas.edu/~simon/sosp17-final207.pdf https://www.cs.utexas.edu/~simon/sosp17-final207.pdf
- rybosome 8y agoInteresting read, thanks for sharing. It didn’t even occur to me that the file systems will need to change to fully take advantage of NVRAM. I wonder at what point the abstraction will stop leaking and require another higher layer to account for differences in performance. I’m sure the OS will need tuning, but applications might not unless they’re pretty bare metal.
- pbalcer 8y agoDisclaimer: I work at Intel on PMDK (pmem.io) There's been a lot of interesting research around file systems for persistent memory. One that shows a lot of promise is NOVA [0]. Its focus is on making full us on this new type of memory. And it's not just pure research, they are attempting to get NOVA included into Linux kernel [1, 2]. And while talking about file systems, we shouldn't forget about the effort that was put into modifying the existing ones to support DAX (Direct Access) [3, 4]. [0] - http://nvsl.ucsd.edu/index.php?path=projects/nova http://nvsl.ucsd.edu/index.php?path=projects/nova [1] - https://lwn.net/Articles/729812/ https://lwn.net/Articles/729812/ [2] - https://lkml.org/lkml/2017/8/5/188 https://lkml.org/lkml/2017/8/5/188 [3] - https://lwn.net/Articles/717953/ https://lwn.net/Articles/717953/ [4] - https://lwn.net/Articles/731706/ https://lwn.net/Articles/731706/
- JohnBooty 8y agoI'm so confused. I can't find a good explanation of how applications and/or the OS "see" these DIMMs. How does my OS/app see this? Is it accessed like regular DRAM memory... except slower and persistent? Or would my OS see it as a "normal" drive... except one that's really fast and happens to be connected via a DIMM slot instead of PCIe/SATA/whatever?
- sethhochberg 8y agoYour operating system will see them as storage, not regular DRAM. Intel has released SDKs for developers to use if their applications need to be able to interact with persistent memory in any kind of detailed way (say, if you develop a database product, or a filesystem). http://pmem.io/pmdk/ http://pmem.io/pmdk/ It is very likely that generic kernel support will come for use in Linux and Windows directly, building on top of the existing DAX systems in those operating systems (DAX - direct access - APIs being used for IO to memory-like devices, bypassing cache layers which are useful for more traditional storage types). This would allow a user to create a regular old storage volume in their NVDIMM for general use. https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt Do note that NVDIMMs aren't a drop-in replacement for regular DRAM DIMMS, despite using the same bus and electrical subsystem. You'll need proper hardware support on your motherboard and CPU, since memory controllers are on CPU these days.
- JohnBooty 8y agoThat's a fantastic summary. Thank you!
- twic 8y ago> Your operating system will see them as storage Do you mean it will present them to userland as storage, rather than see them as storage? Seeing them as storage implies to me that the DIMM emulates an AHCI, which i don't think is the case.
- sethhochberg 8y ago
- pulse7 8y agoNow hibernation will be fast and usable. Stand-by will not be needed anymore...
- chrisper 8y agoI am guessing that this now means that we also have to encrypt our devices in DIMM, especially if it is persistent.
- dajonker 8y agoThere are lots of papers out there discussing the possibilites of non-volatile memory, e.g. Let’s Talk About Storage & Recovery Methods for Non-Volatile Memory Database Systems (Arulraj, Pavlo, Dulloor, 2015)[1]. However, they all seem to work with "simulated" NVM. It will be interesting to see how these real units compare to the simulations. [1] https://www.cs.cmu.edu/~jarulraj/papers/2015.storage.sigmod.pdf https://www.cs.cmu.edu/~jarulraj/papers/2015.storage.sigmod....
- al_james 8y agoWhat I can't work out from both the article and the comments here: from an application point of view, do I use this like I use memory, or do I use it like I use a disk? No matter how fast a disk is, using it means either some expensive serialization/deserialization step (and also the associated memory access to create the 'working' object that my logic actually works on) or writing my algorithms to forego in memory objects (and the associated features offered by my programming language, e.g. classes / objects or whatever) and working from the raw byte values. What I really want, and would be a game changer as to how we use things, would be that my programming languages heap can be made persistent (or at least a part of it). In this case instead of: var mything = new Thing(); load_thing_from_disk(mything); I might have: persistent var mything = new Thing(); Done. However this also introduces more questions, like transactional commits to memory etc (as few apps are coded to ensure consistency of memory across reboots). However I cant help thinking that some way to harness persistent fast memory without needed some complex disk->logic mapping would be a game changer. Edited: spelling and wording
- arghwhat 8y agoIt would appear to be treated as a type of object store that you access with a special library, or through the OS as a filesystem: https://software.intel.com/en-us/articles/introduction-to-programming-with-persistent-memory-from-intel https://software.intel.com/en-us/articles/introduction-to-pr...
- al_james 8y agoThats very very interesting. Many thanks for sharing. So its still a serialize/deserialize cycle, but the access libs built on top of the persistent memory look interesting.
- nickflood 8y agoI see it as a disk drive where memory mapping is a NOP and its performance close to RAM. So an .exe starting from this disk already has all of its contents mapped to RAM, and you can create and memory-map a 512 GB file on it and use it essentially as a RAM blovk in your processe's memory space. But you then can close and reopen it meaning it's guaranteed to be persistent.
- redshirt 8y agoNo need for paging for the most part with these. Current virtual memory systems won’t be able to deal...can’t wait to see what’s next.
- joshbaptiste 8y agoI found this PDF to be very helpful, explaining the general use case of persistent memory programming. https://www.usenix.org/system/files/login/articles/login_summer17_07_rudoff.pdf https://www.usenix.org/system/files/login/articles/login_sum...
- fooyc 8y agoWhat databases are built with this kind of storage in mind already?
- augustl 8y agoVoltDB comes to mind. This interview from 2013 is still worth a listen imo! http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/ http://www.se-radio.net/2013/12/episode-199-michael-stonebra...
- erichocean 8y agoThis is ideal for something like LMDB. Can't wait.
- xrobot7 8y agoThrowaway account for obvious reasons. Was closely involved in the development of it. To summarize, this product is shot, and is just hype. I'll check the technical questions/gaps and answer or fill them in tomorrow.