5 ms·

The problem with fast byte-addressable non-volatile memory isn't hardware IMO. They've been around for a while, with Optane DC being the better option at the mo

The problem with fast byte-addressable non-volatile memory isn't hardware IMO. They've been around for a while, with Optane DC being the better option at the moment. The real issue is that software and APIs aren't keeping up with the hardware. If you do IO with memory mappings or anything POSIX, you will not be able to saturate a modern NVME SSD. And I'm not even talking about Optane here. This means that there is no market for these crazy fast devices. Look at the storage-optimized instances of every major cloud provider, and you won't find them. They would be too expensive, and the software people would run on them wouldn't be any faster than the older, cheaper SSDs. It'll take time for the software to evolve and catch up. Initiatives like DAX, eBPF, and io_uring in the kernel are a big step forward in the right direction.

mhh__ 5y ago
What does eBPF have to do with storage?
- tenebrisalietum 5y ago
  eBPF is being or already is extended to hook into kernel facilities other than networking. I could see it being used to implement some abstraction onto storage other than POSIX block device or mmap().
- HippoBaro 5y ago
  You can implement stuff using eBPF and do IO and compute in without leaving the kernel. It’s crazy. For instance you can perform an on-disk b-tree lookup in the kernel. These devices are so fast that crossing the kernel boundary becomes the limiting factor.
The_rationalist 5y ago
That is no longer true, io_uring maintainer bought an Optane in 2021 and fine tuned io_uring to saturate said Optane https://www.phoronix.com/scan.php?page=news_item&px=Linux-IO_uring-10M-IOPS https://www.phoronix.com/scan.php?page=news_item&px=Linux-IO...
- wtallis 5y ago
  I'm pretty sure Intel loaned him the P5800X drives, starting before they were widely available for purchase. But those are a different product from the Optane DC persistent memory product that uses DIMM slots instead of PCIe and NVMe.
- HippoBaro 5y ago
  I mention io_uring in my original comment. The issue isn’t that the tech doesn’t exist, it’s that it’s not being used. And even before that there was Linux AIO etc.
  - The_rationalist 5y ago
    A nice counter-example are project loom green threads which will make use of it. But yeah it is very much underused
Someone 5y ago
> This means that there is no market for these crazy fast devices. If this becomes a real product, I expect that market will be created. My guess is that it would show up in Apple hardware very early on, if not first. It probably would increase battery life (if I understand this tech correctly, you won’t have to power the RAM while the system is sleeping), and they’ve shown they’re willing to completely redesign their hardware.
mdavis6890 5y ago
It can come up in use-cases with very large in-memory databases, caches or indexes that would otherwise have to be rebuilt/reloaded on reboot, taking hours. Imagine having, say 4TB of index in memory and needing to patch that server. With persistent memory you could be back up and running much more quickly. Especially helpful when you have many of these types of servers that have to all get patched in some sequence. This is not a home-use-case, but it certainly exists today in larger Enterprises. So it's not (necessarily) about having a very fast filesystem, it's about having persistent memory.
- tentacleuno 5y ago
  > This is not a home-use-case, but it certainly exists today in larger Enterprises. Part of me hopes someone in here proves you wrong :-) Us techies do mad things sometimes...
- Dylan16807 5y ago
  What kind of patching? Is the goal to say "okay, here is a persistent chunk of memory with a file or program in it, keep it preserved while the OS reboots, then if it's a program reconnect the files and sockets it was using"? Because you can just remove the word 'persistent' and do that today. Most hardware shouldn't wipe memory on a reboot, and if it does talk to your vendor for a BIOS option. And all the support code you need in your OS should be roughly the same.
cyber_kinetist 5y ago
Also in the desktop world, Microsoft’s DirectStorage API was made exactly to alleviate those concerns, although marketed primarily for games.

mhh__ 5y ago

What does eBPF have to do with storage?

tenebrisalietum 5y ago
eBPF is being or already is extended to hook into kernel facilities other than networking. I could see it being used to implement some abstraction onto storage other than POSIX block device or mmap().
HippoBaro 5y ago
You can implement stuff using eBPF and do IO and compute in without leaving the kernel. It’s crazy. For instance you can perform an on-disk b-tree lookup in the kernel. These devices are so fast that crossing the kernel boundary becomes the limiting factor.

The_rationalist 5y ago

That is no longer true, io_uring maintainer bought an Optane in 2021 and fine tuned io_uring to saturate said Optane https://www.phoronix.com/scan.php?page=news_item&px=Linux-IO_uring-10M-IOPS https://www.phoronix.com/scan.php?page=news_item&px=Linux-IO...

wtallis 5y ago
I'm pretty sure Intel loaned him the P5800X drives, starting before they were widely available for purchase. But those are a different product from the Optane DC persistent memory product that uses DIMM slots instead of PCIe and NVMe.
HippoBaro 5y ago
I mention io_uring in my original comment. The issue isn’t that the tech doesn’t exist, it’s that it’s not being used. And even before that there was Linux AIO etc.
- The_rationalist 5y ago
  A nice counter-example are project loom green threads which will make use of it. But yeah it is very much underused

Someone 5y ago

> This means that there is no market for these crazy fast devices. If this becomes a real product, I expect that market will be created. My guess is that it would show up in Apple hardware very early on, if not first. It probably would increase battery life (if I understand this tech correctly, you won’t have to power the RAM while the system is sleeping), and they’ve shown they’re willing to completely redesign their hardware.

mdavis6890 5y ago

It can come up in use-cases with very large in-memory databases, caches or indexes that would otherwise have to be rebuilt/reloaded on reboot, taking hours. Imagine having, say 4TB of index in memory and needing to patch that server. With persistent memory you could be back up and running much more quickly. Especially helpful when you have many of these types of servers that have to all get patched in some sequence. This is not a home-use-case, but it certainly exists today in larger Enterprises. So it's not (necessarily) about having a very fast filesystem, it's about having persistent memory.

tentacleuno 5y ago
> This is not a home-use-case, but it certainly exists today in larger Enterprises. Part of me hopes someone in here proves you wrong :-) Us techies do mad things sometimes...
Dylan16807 5y ago
What kind of patching? Is the goal to say "okay, here is a persistent chunk of memory with a file or program in it, keep it preserved while the OS reboots, then if it's a program reconnect the files and sockets it was using"? Because you can just remove the word 'persistent' and do that today. Most hardware shouldn't wipe memory on a reboot, and if it does talk to your vendor for a BIOS option. And all the support code you need in your OS should be roughly the same.

cyber_kinetist 5y ago

Also in the desktop world, Microsoft’s DirectStorage API was made exactly to alleviate those concerns, although marketed primarily for games.