18 ms·
Available on Google Cloud: Intel Optane DC Persistent Memory
- zilchers 8y agoDoes anyone know much about the tech on this? I assume when they say persistent they mean across VM restarts, but are they actually doing some sort of disk persistence too?
- locacorten 8y agoThis is byte-addressable persistent memory. They look like DRAM DIMMs and they plug into DIMM slots. You access them using your memory controller and not your storage controller. People sometimes refer to them as non-volatile memory (NVM). Intel used to call it Apache Pass. They're a nightmare to program because OSes do not have a good abstraction for them (at least not yet). Accessing them through the file-system seems sub-optimal (this is byte-addressable memory and not a block device). Accessing them through virtual memory is also pretty bad because they're much slower than DRAM.
- devit 8y agoIs there are a reason to not just use DRAM along with a battery to achieve the same persistence but as fast as DRAM?
- manigandham 8y agoThe reason is the battery. These devices can be powered off and save state, like an SSD.
- piotrkaminski 8y agoBut in the context of the OP, presumably devices in a data center would never be powered down on purpose to save energy? In which case it seems that battery-backup DRAM would work just as well for this use case, and be both cheaper and faster.
- jordanthoms 8y agoOptane is in between DRAM and flash in terms of cost (and performance) - it's also denser, so you can fit much more storage on each DIMM slot.
- juancampa 8y agoProbably because systems that passively do their job tend to be preferred. Also a battery would only last so long. IIRC DRAM needs to be constantly refreshed, so, it would be a trade-off between capacity and duration. Optane seems[1] to be 20~30X slower than DRAM but 4~10X faster than server SSDs [1]: https://superuser.com/a/1195674/187732 https://superuser.com/a/1195674/187732
- zokier 8y agoDRAM can not be simply naively battery-backed; it needs active refreshing. And as memory controllers reside in CPUs these days, that would mean keeping the CPU powered up.
- wtallis 8y agoThere are NVDIMMs that have DRAM and a matching quantity of NAND flash memory to save the contents to in the event of a power failure. They require an external capacitor module and are limited in data capacity by how much DRAM you can fit on the module. You can fit far more 3D XPoint memory on a module than DRAM, and it doesn't require the external capacitors to achieve persistence, and it should be significantly cheaper on a per-GB basis.
- cdoxsey 8y agoJust to reiterate the point... this is an instance with terabytes of near-memory-speed storage. If persistent memory pans out as a technology it will completely upend the way we think about building software and the cost tradeoffs of hardware. (as much as or more so than the transition from spinning disks to ssds)
- Dylan16807 8y agoHow do you define "pans out"? What performance and price differences between it, flash, and DRAM do you have in mind? Because I'll keep reminding people that putting a DRAM cache in front of some flash can very closely approximate a large persistent memory. If people wanted to build software for that kind of system, they could do it today. The hardware is not the blocker.
- wmf 8y agoDiablo Memory1 used flash DIMMs with DRAM DIMMs as cache. When we tried it, it worked OK for some workloads and poorly for others, but it was also buggy and then the company went out of business. So a large part of "pans out" is simply a production-quality implementation that you can buy. Note that Optane DIMMs have been delayed by around two years at this point and we still don't know what they will cost.
- SergeAx 8y agoCache will always stay just cache, unless it is the same size that an underlying persistent storage. You cannot read or write larger-than-cache chunks without performance degradation. Also you have a start-up cache-warming problem.
- the8472 8y ago> They're a nightmare to program because OSes do not have a good abstraction for them (at least not yet). With DAX[0] linux already has the ability to put a filesystem (currently ext4 and xfs) on NVDIMMS and then let userspace address them through mmap while skipping the page cache indirection. I.e. you're directly byte-addressing them through the memory controller via standard memory-mapped file abstractions. Direct block device mapping of nvdimms without filesystem is also possible. [0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt
- markdoubleyou 8y agoThe Persistent Memory Development Kit (PMDK) offers high-level abstractions over DAX. Looks like most mortals would use libpmemobj (or its C++ bindings). http://pmem.io/pmdk/ http://pmem.io/pmdk/
- pbalcer 8y agoDisclaimer: I work at Intel on PMDK (pmem.io) Both Windows and Linux implement DAX, which, as @the8472 explained, allows bypassing page cache in memory mapped I/O. Additionally, DAX optionally allows you to flush your data directly from user-space instead of calling msync. And that's the gist of NVM programming model [0], its entire point is to allow applications to avoid the now hugely excessive abstraction layer of traditional storage. And I will freely admit that programming to raw memory mapped files can be difficult, but there is ongoing work on making it easier. An example of that is, excuse the shameless plug, Persistent Memory Development Kit [1], which makes writing new software for this new type of memory much simpler. Performance of an NVDIMM is obviously hardware dependent, but the now widely accepted programming model works with the assumption that persistent memory is fast enough so that it is reasonable to stall a CPU while an instruction is accessing it. I'm not sure on what hardware evaluations you are basing your claims on, but let me assure you that the HW solution being described in the blog post does not violate that assumption. [0] - https://www.snia.org/tech_activities/standards/curr_standards/npm https://www.snia.org/tech_activities/standards/curr_standard... [1] - http://pmem.io/ http://pmem.io/
- simcop2387 8y agoDo you know how well tools like Cap'n Proto and Protocol Buffers help for dealing with this kind of scenario? I'd imagine that some kind of low latency/cost serialization system would help significantly in using the device. Cap'n Proto I'd imagine would work nicely for reading data off since it should be able to read and use the structure with no extra copying or decoding, but I have no idea how the situation with writing would win out.
- wtallis 8y agoThe simplest way of using this is to not do any serialization at all, just store any information you want persisted in memory allocated from the region you mmaped to the Optane DIMMs instead of the DRAM DIMMs.
- simcop2387 8y ago
- Symmetry 8y agoIt seems like Single-level store[1] would be a really good fit for this. I was going to make a crack about bringing back Multics but apparently IBM has an OS using this. [1]https://en.wikipedia.org/wiki/Single-level_store https://en.wikipedia.org/wiki/Single-level_store
- garkin 8y agoIt uses Intel Optane. It's basically super fast SSD used as RAM.
- nodesocket 8y agoHow does this compare to NVMe based AWS EC2 instances like m5d, c5d, r5d?
- mrep 8y agoA better comparison would be EC2 x1 instances which are high memory instances designed for SAP HANA like these ones: https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/
- manigandham 8y agoIt's between RAM and SSD. Much closer to RAM speeds but data persists after turning off power. Biggest advantage over just being a faster disk is that it's byte-addressable random access like memory, so you don't have to deal with writing pages/blocks to drives. The throughput and latency is fast enough that many apps can skip using memory and work straight from this. We already see this in some mobile devices that only have solid state and instant boot times because the data is always ready, with no need to shift from disk to ram. It'll be awhile before OS and applications take advantage but it has potential to be a major shift in how storage works.
- boulos 8y agoDisclosure: I work on Google Cloud. NVMe is (now) a more general purpose "speak to flash like things". This Optane memory stuff is made of "flash", but unlike our Local SSD offering (or AWS's i3) it's at a latency and throughput closer to DRAM. Despite it being marketing material, I find the pyramid diagram [1] helpful. This blog post is about "Optane Persistent Memory". [1] https://newsroom.intel.com/wp-content/uploads/sites/11/2018/05/filling-the-gaps-between-memory-and-storage-after.png https://newsroom.intel.com/wp-content/uploads/sites/11/2018/...
- ssvss 8y agoWhat is its latency compared to RAM, 10X ? Most latency comparisons mentioned in other comments compare RAM to pcie based optane memory, not the DRAM optane memory. Edit: Article[1] here says the latency was 40µs with 13M IOPS, if you consider RAM latency to be 100ns[2], then it looks like 40X [1] - https://blogs.technet.microsoft.com/filecab/2018/10/30/windows-server-2019-and-intel-optane-dc-persistent-memory/ https://blogs.technet.microsoft.com/filecab/2018/10/30/windo... [2] - https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832
- espeed 8y agoRedis on Optane [1]. The GraphBLAS stars continue to align [2]. GPUs/TPUs next. Distributed to come. [1] Redis on Optane https://redislabs.com/blog/redis-enterprise-flash-intel-optane/ https://redislabs.com/blog/redis-enterprise-flash-intel-opta... [2] GraphBLAS & RedisGraph https://news.ycombinator.com/item?id=18099520 https://news.ycombinator.com/item?id=18099520
- geggam 8y agoWhat is the benefit ? Why are we constantly creating pets out of what should be cattle ?
- ztorkelson 8y agoThe benefits of persistent memory, as I understand it, are principally improved performance (higher throughput, lower latency) and secondarily ease of programming. (Edit: another obvious benefit is cost per gigabyte; this stuff is cheaper than RAM.) The benefits of having access to that technology on GCP (or another cloud) are the usual: reduced operational burden, increased availability, flexible pricing structure, elastic scalability, etc. Your question about pets vs cattle is a non sequitur. Nothing about this announcement or the underlying technologies are suggestive of how they should be used (or misused). It's just a new tool in the toolbox—and as a distributed systems engineer specializing in database technology, having easy access to this hardware at scale is extremely compelling.
- segmondy 8y agotoo bad you got downvoted, i would imagine in-memory DB and large cache.
- Dylan16807 8y ago> too bad you got downvoted It's pretty weird to call out people for making 'pet' servers because they're daring to attach storage to them.
- rebelde 8y agoHis native language isn't English, and people probably misunderstood his literally-translated idiom. Pet servers?
- sciurus 8y agoPet's vs cattle is an analogy that originated with English-speakers. http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-...
- thibautg 8y agoI still do not completely understand what SAP HANA really is: - an in-memory database technology? - the name for SAP’s cloud platform? - an on-premise DB to run SAP ERP (to replace Oracle)? - a full-stack proprietary web development platform? - a marketing term to solve all problems with SAP products? Does anyone have an hands on experience with HANA, beyond the usual marketing BS? Is is that revolutionary? If it runs on such specialized hardware, is the speed increase that impressive?
- lozaning 8y agoCouldn't tell you what it does, I only know that the advertising of it in airports is ubiquitous. In the dozen plus airports I've been in this year, they've all been plastered in adds for HANA.
- sacheendra 8y agoDon't know specifically about HANA. But, any database with high throughput would benefit from high bandwidth PCIe non-volatile storage. Databases with significant random accesses in their workload would see a large improvement in performance due to Optane. Nothing else apart from main memory matches Optane with its insane random access performance. ERP applications which use HANA typically do have significant random accesses in their workload. It is due to this reason that it was initially marketed as an in-memory database. The random access performance of RAM was necessary for high performance. If HANA was just being used in in-memory mode, the speed increase is probably not that impressive. But, if there is non-volatile storage involved, then yes the speed increase would be impressive.
- dijit 8y agoI have no practical experience with SAP HANA. But I’m a sysadmin and I’ve been to conferences where SAP has had stands and I have a tendency to ask too many questions. SAP HANA is an in memory database. That’s all it is. It’s similar to Qlikview if you’ve used that. It’s licensed per socket, so scaling up is better than scaling out. SAP will also host HANA for you, for a fee. See. The thing is: SAP is a company for HR departments. The kinds of departments that aren’t technical, do not have technical staff to develop thing for them. (It’s an example) but your HR department talks to SAP, they deliver some service, sometimes something that the barely tech-literate members of HR is able to build on and then suddenly your sysadmins have to support those things in perpetuity. Well. They market themselves as this anyway. And I have experience of our HR department buying all kinds of stuff from SAP and their child companies. (Concur, for example). So I understand your incredulous-ness. This company is not made for us. And most of their tech isn’t either.
- boulos 8y agoDisclosure: I work on Google Cloud. As I replied in a sub-thread, I think Intel's marketing diagram [1] is probably useful to help separate the Optane flavors. This is about the "near DRAM" variant. While the blog post highlights running SAP HANA (SAP's in-memory focused database), you can use them for whatever you want. The persistent part is that it's persistent across reboots. The hope is that this might make it easier to have tiered database/caching systems, since the gap between DRAM and this new "memory" is much closer than say DRAM and SSD. [1] https://newsroom.intel.com/wp-content/uploads/sites/11/2018/05/filling-the-gaps-between-memory-and-storage-after.png https://newsroom.intel.com/wp-content/uploads/sites/11/2018/...
- frakkingcylons 8y ago> The persistent part is that it's persistent across reboots. So does that mean it can be considered persistent like a regular SSD/HDD?
- ztorkelson 8y agoYes. It's durable storage with a RAM interface. Byte-addressable through conventional CPU load/store instructions. It follows the usual cache coherence protocols for memory, but once it gets flushed, it's stable—just like conventional storage devices.
- spiritcat 8y agoHow much of a problem is the gap for SSD's? Don't those have buffers with enough battery to write out if they lose power?
- georgewfraser 8y agoOne of the most interesting applications of NVM is databases. The design of most existing databases is predicated on needing to always persist writes to disk; the existence of non-volatile memory allows very different, potentially much faster designs. There's a great explanation of this in https://db.cs.cmu.edu/papers/2017/p1753-arulraj.pdf https://db.cs.cmu.edu/papers/2017/p1753-arulraj.pdf
- musiciangames 8y agoWould this be a good fit for Smalltalk, as it is image based? It seems it can remove a level of complexity, if you can treat your image, and so all your objects, as constantly persisted?
- scroot 8y agoCan't you already do this with an OODB? I'm asking sincerely, since I have never quite understood why OODBs aren't used more often.