4 ms·
I'm not sure I really understand why Optane went in the DIMM slots instead of the PCIe slots. I know PCIe storage is usually NVMe, but you could also expose the
by toast0 4y ago
I'm not sure I really understand why Optane went in the DIMM slots instead of the PCIe slots. I know PCIe storage is usually NVMe, but you could also expose the storage as a large memory mapped region. You wouldn't need explicit support from the cpu/chipset/motherboard, although you probably want resizable BAR support, cause the region would presumably be pretty large, and maybe you want to do something for older systems that can't handle a giant region. From the software side of things, PCIe mapped memory is more or less the same as physical memory; from the hardware side, you'd be taking a different path to get there, so you'd have different bottlenecks, but PCIe5 is here with 2x the bandwidth and there's a limited number of applications that really max out their PCIe lanes.
- lproven 4y ago[Article author here] FWIW I tried to explain why that was the important aspect of the tech in the article. I also covered it in some depth in my most recent FOSDEM talk, which is linked in the article. As is the script if you don't have time for the video.
- robin_reala 4y agoThey did do PCIe Optane (e.g. https://www.intel.com/content/www/us/en/products/sku/129833/intel-optane-ssd-905p-series-1-5tb-12-height-pcie-x4-20nm-3d-xpoint/specifications.html https://www.intel.com/content/www/us/en/products/sku/129833/...)
- toast0 4y agoSure, but looks like that was NVMe, not just a big bunch of bytes, so whatever magical future was enabled by having a big bunch of non-volatile bytes wasn't enabled by this. (unless there was a way to change it into big bunch of bytes mode?)
- MobiusHorizons 4y agoMy guess is that to get low latency you have to be interacting with a memory controller that expects low latency (like dram) I don’t know how many cycles it would take as a difference, but my impression is that pcie trades throughput for latency compared to dram
- formerly_proven 4y agoIIRC permissible lane skew in PCIe is like 30 ns, which means that the latency due to deskewing alone in PCIe already approaches main memory latency.
- NavinF 4y agoPCIe Optane was a thing and it achieved 10us latency whereas today's fastest SSDs get 40us. IIRC the DIMM version of Optane was <1us, literally an order of magnitude faster! I'd expect PCIe to contribute ~3us latency and interrupt/context-switch/kernel to contribute ~5us. You could eliminate the latter, but 3us is still kinda slow.
- usrusr 4y agoAn order of magnitude faster, but still a few cache hierarchies away from where the actual computing is happening. But the real dealbreaker was that there are also similar indirections in the other direction. It's just so very, very common in modern architectures to not permanently store data on the first drive it hits. That first level of persistence is usually just some queue log where the write takes a short breath, safe from power failure for the first time, before getting passed on to some more distributed persistence story. Optane wouldn't change this general setup at all, because those layers exist for more reasons than just the power failure scenario. Optane might change implementation details of that first layer, but it wouldn't question the layers. Because optane is only answer to power failure, not to the data center burning down.
- PaulHoule 4y agoOptane is either the fastest SSD or the slowest RAM. If it is doing the job of RAM it might hurt performance, not help it. It’s not clear adding another level to the cache is really going to help.
- funny_falcon 4y agoFor consumers notebook, Optane could be fast swap drive. It's pity manufacturers didn't get this done. Swap should be just fast to make application switch fast enough. Optane could do it.
- smoldesu 4y agoI mean... with modern memory compression and SSD speeds, is this really neccesary? If you've got an NVMe SSD, you've most likely got enough bandwidth to never notice the swap kicking in (especially with a generous swappiness value). Sure there might be a couple workloads that could benefit from that kind of upgrade, but I think most people building memory-optimized rigs would rather just buy more memory, since it goes in the same DIMM anyways.
- warpech 4y agoI guess it's because DIMM is physically closer to CPU than PCIe, so you get better bandwidth thanks to that (think that the data travels at the speed of light - the closer the faster).
- caf 4y agoProbably because the way they did it allowed it to participate in the caching hierarchy with MESI and all the rest of it.