4 ms·

But what market are they even trying to hit here? DDR4 prices are insane, but home consumers don't run large NUMA systems. Data center admins are going to be a

But what market are they even trying to hit here? DDR4 prices are insane, but home consumers don't run large NUMA systems. Data center admins are going to be able to get the capitol for real DDR4 over Optane. They'd need to be ordering a lot of servers before the price point becomes significant enough that they'd order an Optane configuration and decide to benchmark them. I'd be curious about the real benchmarking at that point. It there a performance boost to simply have more memory, even if some of it is Optane, or is it better to have a lower capacity without the Optane sticks? I have a feeling this will vary heavily by application too.

leetcrew 8y ago
> It there a performance boost to simply have more memory, even if some of it is Optane, or is it better to have a lower capacity without the Optane sticks? I would imagine it depends entirely on whether your working set fits in main memory already. if yes, you're not gonna get much speedup from more memory. but if you have to keep swapping from a traditional SSD, it's gonna be hard for faster main memory to make up for that.
naveen99 8y ago
I thought the real advantage of Optane was bypassing the pcie bus, with much larger bandwidth of dimm: 80GB/s instead of 12GB/s
- tgtweak 8y ago
  DDR4 dimm read throughput is about 20GB/s per dimm. PCIe 3.1 x4 is 16GB/s. I feel very comfortable claiming that these optane dimms will not reach 20GB/s of sustained performance. The advantage here being latency (and thus single queue throughput)
  - naveen99 8y ago
    But pcie bus is shared with other cards. So 8 dimms is 8x faster than 8 pcie cards.
    - saati 8y ago
      > But pcie bus is shared with other cards. No, it's not, that's the whole point of pcie, and dimms share the memory bus anyway, so the whole point is moot.
      - naveen99 8y ago
        Hmm. In that case never mind.
      - AnonymousR 8y ago
        Well, it depends. "True" pcie slots do not share the bandwidth, but depending on available CPU lanes, which is awfully few in the last years on Intel - 16 lanes in top of the line i7? thats one graphics card - the slots are switched to 8x/8x, 8x/4x/4x and so on when populated. M.2 slots are a whole another story, they are usually connected to the chipset instead of CPU on desktop motherboards (and finding this information for any particular board is difficult). The chipset is connected with CPU through DMI, which equals to pcie 4x and this is shared with everything on chipset - satas, gigabit ethernet, usb 3...
  - dragontamer 8y ago
    > PCIe 3.1 x4 is 16GB/s PCIe x4 is 3.94GB/s. PCIe x16 is 15.76GB/s. > DDR4 dimm read throughput is about 20GB/s per dimm Per channel, not per dimm. Two dimms typically go into a singular channel. > The advantage here being latency (and thus single queue throughput) Agreed
tgtweak 8y ago
The idea of putting ssds in memory dimms isn't new, the jedec standard for this is included in ddr3. It's referred to as an nvdimm(-f for SSD), sandisk and IBM both produced and sold them. These are supported since Linux kernel 4.1 and show up as a block device in /dev/pmem. The latest optane variants are nvdimm-p and are not treated like block storage devices but instead as persistent memory to the OS. The storage dimms have some notable benefits over pcie nvme ssds - latency to the processor is extremely low, they give extra total storage I/O since they're leveraging a different controller on the cpu with its own bandwidth, while still leaving pcie bandwidth for traditional storage. They can add capacity to a small system (a 1U server can hold 24 such ssds) As I see it though, these optane dimms are just fast swap drives which do not offer the same consistent throughput as true memory dimms. They are transparent to the OS which is normally in charge of deciding what stays in memory and what goes to swap. Use of an Intel sdk is required to leverage much of the benefits. It's also Intel-only as far as support goes for now. As usual the marketing doesn't mention the throughput or pricing and instead of comparing to memory they compare it to storage. http://www.admin-magazine.com/HPC/Articles/NVDIMM-Persistent-Memory http://www.admin-magazine.com/HPC/Articles/NVDIMM-Persistent... https://thessdguy.com/an-nvdimm-primer-part-1-of-2/ https://thessdguy.com/an-nvdimm-primer-part-1-of-2/ https://www.extremetech.com/extreme/270270-intel-announces-new-optane-dc-persistent-memory https://www.extremetech.com/extreme/270270-intel-announces-n...

leetcrew 8y ago

> It there a performance boost to simply have more memory, even if some of it is Optane, or is it better to have a lower capacity without the Optane sticks? I would imagine it depends entirely on whether your working set fits in main memory already. if yes, you're not gonna get much speedup from more memory. but if you have to keep swapping from a traditional SSD, it's gonna be hard for faster main memory to make up for that.

naveen99 8y ago

I thought the real advantage of Optane was bypassing the pcie bus, with much larger bandwidth of dimm: 80GB/s instead of 12GB/s

tgtweak 8y ago
DDR4 dimm read throughput is about 20GB/s per dimm. PCIe 3.1 x4 is 16GB/s. I feel very comfortable claiming that these optane dimms will not reach 20GB/s of sustained performance. The advantage here being latency (and thus single queue throughput)
- naveen99 8y ago
  But pcie bus is shared with other cards. So 8 dimms is 8x faster than 8 pcie cards.
  - saati 8y ago
    > But pcie bus is shared with other cards. No, it's not, that's the whole point of pcie, and dimms share the memory bus anyway, so the whole point is moot.
    - naveen99 8y ago
      Hmm. In that case never mind.
    - AnonymousR 8y ago
      Well, it depends. "True" pcie slots do not share the bandwidth, but depending on available CPU lanes, which is awfully few in the last years on Intel - 16 lanes in top of the line i7? thats one graphics card - the slots are switched to 8x/8x, 8x/4x/4x and so on when populated. M.2 slots are a whole another story, they are usually connected to the chipset instead of CPU on desktop motherboards (and finding this information for any particular board is difficult). The chipset is connected with CPU through DMI, which equals to pcie 4x and this is shared with everything on chipset - satas, gigabit ethernet, usb 3...
- dragontamer 8y ago
  > PCIe 3.1 x4 is 16GB/s PCIe x4 is 3.94GB/s. PCIe x16 is 15.76GB/s. > DDR4 dimm read throughput is about 20GB/s per dimm Per channel, not per dimm. Two dimms typically go into a singular channel. > The advantage here being latency (and thus single queue throughput) Agreed

tgtweak 8y ago

The idea of putting ssds in memory dimms isn't new, the jedec standard for this is included in ddr3. It's referred to as an nvdimm(-f for SSD), sandisk and IBM both produced and sold them. These are supported since Linux kernel 4.1 and show up as a block device in /dev/pmem. The latest optane variants are nvdimm-p and are not treated like block storage devices but instead as persistent memory to the OS. The storage dimms have some notable benefits over pcie nvme ssds - latency to the processor is extremely low, they give extra total storage I/O since they're leveraging a different controller on the cpu with its own bandwidth, while still leaving pcie bandwidth for traditional storage. They can add capacity to a small system (a 1U server can hold 24 such ssds) As I see it though, these optane dimms are just fast swap drives which do not offer the same consistent throughput as true memory dimms. They are transparent to the OS which is normally in charge of deciding what stays in memory and what goes to swap. Use of an Intel sdk is required to leverage much of the benefits. It's also Intel-only as far as support goes for now. As usual the marketing doesn't mention the throughput or pricing and instead of comparing to memory they compare it to storage. http://www.admin-magazine.com/HPC/Articles/NVDIMM-Persistent-Memory http://www.admin-magazine.com/HPC/Articles/NVDIMM-Persistent... https://thessdguy.com/an-nvdimm-primer-part-1-of-2/ https://thessdguy.com/an-nvdimm-primer-part-1-of-2/ https://www.extremetech.com/extreme/270270-intel-announces-new-optane-dc-persistent-memory https://www.extremetech.com/extreme/270270-intel-announces-n...