4 ms·

I under understood what problem single-level storage was actually supposed to solve. In the real world you still need a notion of startup and shutdown because p

I under understood what problem single-level storage was actually supposed to solve. In the real world you still need a notion of startup and shutdown because programs crash, memory gets corrupted, things get screwed up, hardware changes, etc., so it's not like we were going to avoid those. So, from a technical standpoint, why exactly would I want to merge my RAM with my SSD? The only thing I can think of is that it lets you achieve genuine "zero-copy" usage of data. Which, I mean, is cool. But is the overhead of copying read-only data from secondary storage to RAM really such a big problem for >90% of users to be justification for changing everyone's very notion of computing?

threeseed 4y ago
> But is the overhead of copying read-only data from secondary storage to RAM really such a big problem It's not a problem that will affect every developer but it is a critical problem. Many uses cases need access to data in both a fast (~ memory) and durable (~ SSD) way. Copying data from SSD to memory helps with performance but the minute you modify that data then you've lost durability. Databases, file systems, caches etc are examples of critical use cases that are used by all developers that would benefit from something like Optane.
- dataflow 4y ago
  What percentage (say) speed improvement would you expect for typical workloads on database, file systems, etc. from a unified memory system?
  - threeseed 4y ago
    So there are only a few examples of existing systems being re-written for Optane. There was Redis [1] which was around ~10% slower than the memory only version but gained you durability. And there was an implementation of Spark Shuffle [2] which was 1.3x - 3.2x faster but that isn't really stressing I/O as much as other use cases. For a filesystem you can store the entire metadata in Optane so EXISTS/LIST type operations and anything involving a bloom filter would see the full benefit e.g. order of magnitude better than NVME. [1] https://github.com/pmem/pmem-redis https://github.com/pmem/pmem-redis [2] https://databricks.com/session_na20/accelerating-apache-spark-shuffle-for-data-analytics-on-the-cloud-with-remote-persistent-memory-pools https://databricks.com/session_na20/accelerating-apache-spar...
    - dataflow 4y ago
      Wow, the Redis case is even worse than I expected! I would've thought maybe a 2x improvement would be normal. Also I think you're conflating workloads with operations. Sure, the occasional operation like EXISTS and LIST operation might be faster, but surely most workloads do a lot more than checking a trillion things for existence? I feel like everything you wrote just makes the case against Optane better than even I could. There seems to be little if any clear performance benefit (being generous here, given slowdowns are also possible as you noted!) to most people's workloads to justify upending everyone's current model of computing. Something like this would probably need to deliver at least an order of magnitude of visible speedup in typical use cases for people to consider it. Which isn't to say some niche workloads might not see 2+ orders of magnitude performance improvement, but the rest of the world clearly won't; why should they have to pay the price for niche use cases?
      - ajmurmann 4y ago
        I didn't look into details for the Redis port to Optane, but work on GemFire which is a similar in-memory system. I wrote expect one big benefit not captured by the 10% number to be startup time. Redis uses a operations log. On startup you'd have to replay the operations which can take a while and you have to clean up the log periodically. So startup / recovery should be quicker and you now have a whole source of complexity and bugs you don't need to worry about because hardware and OS should solve it for you.
        dataflow 4y ago
        Sure, but again, that is clearly far, far too minor of a benefit to justify the radical change we are proposing to everyone's model of computing.
        SassyThrowaway 4y ago
        Are changes to everyones model of computing actually radical? I mean, I understand that taking advantage of the fact that memory actually persists is actually somewhat non-trivial when done via byte-addressable accesses, and that even with libraries that will be doing these things, developers gain a new set of properties to worry about, but... isn't this all optional? Why can't software treat persistent memory as just a lot denser RAM?
  - pbalcer 4y ago
    I recommend looking at this presentation from Oracle for an example of the benefits of PMem: "Under the Hood of an Exadata Transaction – How We Harnessed the Power of Persistent Memory" https://www.youtube.com/watch?v=ertF5ZwCHP0 https://www.youtube.com/watch?v=ertF5ZwCHP0
IshKebab 4y ago
> programs crash, memory gets corrupted... Yeah I think you would still need an "erase all caches and reboot" option but there's no theoretical reason you couldn't have that. The main reason you have to restart desktop OSes so often is because they're ancient and don't isolate components properly. How often do you have to restart your phone? A couple of times a year maybe? But I agree with your point - it does seem like a very cool idea but practically wouldn't make a huge amount of difference and basically requires an entirely new incompatible OS. I wonder if it would have been more successful in phones actually. iOS already doesn't have a user-accessible filesystem and Android is moving in that direction.
- dataflow 4y ago
  > How often do you have to restart your phone? A couple of times a year maybe? Me? Like once a month at least, probably twice. Could be as frequent as multiple times in an hour. Just depends on the reason. It could include anything from "battery ran out before I charged" (a few times a year maybe) to "my phone is crashing/behaving erratically" (could be every few weeks) to "Android didn't refresh its MTP database live and won't show the file I added till I reboot" (could be from 5 mins ago). Not an exhaustive set, just listing a few examples. > I wonder if it would have been more successful in phones actually. Interesting idea. Maybe? I wonder how much the performance improvement would be for the end-user.
  - NavinF 4y ago
    Ok that's not common by any means. Maybe a custom ROM or a new phone is in order? Personally I reboot my iphone once a year. It also reboots overnight every 3 months or so for software updates, but that's often not observable because apps restore their state.
    - deleted 4y ago
      [deleted]
    - dataflow 4y ago
      > Ok that's not common by any means. Pretty sure it's not uncommon on Android for people to reboot every couple weeks or so: https://www.reddit.com/r/GalaxyS21/comments/prdfpf/how_often_do_you_re_start_your_phone/ https://www.reddit.com/r/GalaxyS21/comments/prdfpf/how_often... > Maybe a custom ROM or a new phone is in order? Not due to the above (the reboot need isn't frequent enough to bother me), but for other reasons, possibly? It's not high priority for me but I've been thinking about it. Thing is, I love my phone otherwise. An insane amount of resources go into making hardware like this, and the planet's already trashed enough as is; I hate throwing out hardware that works fine just for random software glitches I can easily put up with. >Personally I reboot my iphone once a year. Maybe iPhone is better about this?
zozbot234 4y ago
> I under understood what problem single-level storage was actually supposed to solve. Briefly stated, it turns fsync() into a near-noop. That seems like it could be a pretty big deal for some workloads.

threeseed 4y ago

> But is the overhead of copying read-only data from secondary storage to RAM really such a big problem It's not a problem that will affect every developer but it is a critical problem. Many uses cases need access to data in both a fast (~ memory) and durable (~ SSD) way. Copying data from SSD to memory helps with performance but the minute you modify that data then you've lost durability. Databases, file systems, caches etc are examples of critical use cases that are used by all developers that would benefit from something like Optane.

dataflow 4y ago
What percentage (say) speed improvement would you expect for typical workloads on database, file systems, etc. from a unified memory system?
- threeseed 4y ago
  So there are only a few examples of existing systems being re-written for Optane. There was Redis [1] which was around ~10% slower than the memory only version but gained you durability. And there was an implementation of Spark Shuffle [2] which was 1.3x - 3.2x faster but that isn't really stressing I/O as much as other use cases. For a filesystem you can store the entire metadata in Optane so EXISTS/LIST type operations and anything involving a bloom filter would see the full benefit e.g. order of magnitude better than NVME. [1] https://github.com/pmem/pmem-redis https://github.com/pmem/pmem-redis [2] https://databricks.com/session_na20/accelerating-apache-spark-shuffle-for-data-analytics-on-the-cloud-with-remote-persistent-memory-pools https://databricks.com/session_na20/accelerating-apache-spar...
  - dataflow 4y ago
    Wow, the Redis case is even worse than I expected! I would've thought maybe a 2x improvement would be normal. Also I think you're conflating workloads with operations. Sure, the occasional operation like EXISTS and LIST operation might be faster, but surely most workloads do a lot more than checking a trillion things for existence? I feel like everything you wrote just makes the case against Optane better than even I could. There seems to be little if any clear performance benefit (being generous here, given slowdowns are also possible as you noted!) to most people's workloads to justify upending everyone's current model of computing. Something like this would probably need to deliver at least an order of magnitude of visible speedup in typical use cases for people to consider it. Which isn't to say some niche workloads might not see 2+ orders of magnitude performance improvement, but the rest of the world clearly won't; why should they have to pay the price for niche use cases?
    - ajmurmann 4y ago
      I didn't look into details for the Redis port to Optane, but work on GemFire which is a similar in-memory system. I wrote expect one big benefit not captured by the 10% number to be startup time. Redis uses a operations log. On startup you'd have to replay the operations which can take a while and you have to clean up the log periodically. So startup / recovery should be quicker and you now have a whole source of complexity and bugs you don't need to worry about because hardware and OS should solve it for you.
      - dataflow 4y ago
        Sure, but again, that is clearly far, far too minor of a benefit to justify the radical change we are proposing to everyone's model of computing.
        SassyThrowaway 4y ago
        Are changes to everyones model of computing actually radical? I mean, I understand that taking advantage of the fact that memory actually persists is actually somewhat non-trivial when done via byte-addressable accesses, and that even with libraries that will be doing these things, developers gain a new set of properties to worry about, but... isn't this all optional? Why can't software treat persistent memory as just a lot denser RAM?
- pbalcer 4y ago
  I recommend looking at this presentation from Oracle for an example of the benefits of PMem: "Under the Hood of an Exadata Transaction – How We Harnessed the Power of Persistent Memory" https://www.youtube.com/watch?v=ertF5ZwCHP0 https://www.youtube.com/watch?v=ertF5ZwCHP0

IshKebab 4y ago

> programs crash, memory gets corrupted... Yeah I think you would still need an "erase all caches and reboot" option but there's no theoretical reason you couldn't have that. The main reason you have to restart desktop OSes so often is because they're ancient and don't isolate components properly. How often do you have to restart your phone? A couple of times a year maybe? But I agree with your point - it does seem like a very cool idea but practically wouldn't make a huge amount of difference and basically requires an entirely new incompatible OS. I wonder if it would have been more successful in phones actually. iOS already doesn't have a user-accessible filesystem and Android is moving in that direction.

dataflow 4y ago
> How often do you have to restart your phone? A couple of times a year maybe? Me? Like once a month at least, probably twice. Could be as frequent as multiple times in an hour. Just depends on the reason. It could include anything from "battery ran out before I charged" (a few times a year maybe) to "my phone is crashing/behaving erratically" (could be every few weeks) to "Android didn't refresh its MTP database live and won't show the file I added till I reboot" (could be from 5 mins ago). Not an exhaustive set, just listing a few examples. > I wonder if it would have been more successful in phones actually. Interesting idea. Maybe? I wonder how much the performance improvement would be for the end-user.
- NavinF 4y ago
  Ok that's not common by any means. Maybe a custom ROM or a new phone is in order? Personally I reboot my iphone once a year. It also reboots overnight every 3 months or so for software updates, but that's often not observable because apps restore their state.
  - deleted 4y ago
    [deleted]
  - dataflow 4y ago
    > Ok that's not common by any means. Pretty sure it's not uncommon on Android for people to reboot every couple weeks or so: https://www.reddit.com/r/GalaxyS21/comments/prdfpf/how_often_do_you_re_start_your_phone/ https://www.reddit.com/r/GalaxyS21/comments/prdfpf/how_often... > Maybe a custom ROM or a new phone is in order? Not due to the above (the reboot need isn't frequent enough to bother me), but for other reasons, possibly? It's not high priority for me but I've been thinking about it. Thing is, I love my phone otherwise. An insane amount of resources go into making hardware like this, and the planet's already trashed enough as is; I hate throwing out hardware that works fine just for random software glitches I can easily put up with. >Personally I reboot my iphone once a year. Maybe iPhone is better about this?

zozbot234 4y ago

> I under understood what problem single-level storage was actually supposed to solve. Briefly stated, it turns fsync() into a near-noop. That seems like it could be a pretty big deal for some workloads.