11 ms·

Understanding the Intel Optane Shutdown

85 points by lichtenberger 4y ago

Veliladon 4y ago
Optane was an interesting solution that was still looking for a question. One of the things that drove mass SSD adoption was the experience was plainly better. Even the most illiterate user could see how fucking fast an SSD machine booted and how much more responsive they were. SSD to Optane? There's no practical difference. The high end desktop users and serious gamers that have serious cash for their rigs turned up their noses at Optane because it turns out SSD sequential read was good enough for most use cases. At least good enough to not pay 3x more per GB. Nobody really cared about insane sustained write speed when the SLC caches did almost as well in most use cases. Plus as the interfaces scaled up, plain old NAND drives were ready to flood the bus with as much sequential read bandwidth that the bus would take. What could have saved it? I dunno. If Intel wanted to be anti-competitive they could have slapped on 8 PCIe lanes direct to the CPU exclusively for Optane on their desktop products. As we've found out, 4 lanes of NVMe to the CPU and the rest having to go through the chipset is some sort of unfunny, market segmentation joke. It could have forced high end users to grudgingly accept Optane as the path to the absolute best I/O and an actual, tangible difference in performance.
- luma 4y ago
  I think one of the major missing factors is lack of OS support for the sorts of features that NVDIMM could potentially bring to the table. Super-fast wake from deep hibernation is just a starting point, the real work is re-writing every line of code written since the dawn of computing which starts from the two assumptions "one kind of storage is persistent and slow" and "the other kind of storage is volatile and fast". HPE made noise about challenging those assumptions with The Machine but predictably lost interest as their own memristor technology lost steam.
- orlp 4y ago
  > What could have saved it? Persistent memory was/is a big topic in the database research world. The idea was that you would not need to worry about ACID since all your data is still there in the event of a power loss/crash. It would massively simplify database code. However as it turns out... persistent memory doesn't buy you all that much. You still effectively need a replay log to restore the state you were in before a crash event, and you still need all the restoration code. In the end it in most systems it just behaved as a very fast (and expensive) SSD. The benefits vs. price just wasn't there as a modern DB setup on a fast SSD + more RAM would get you much more performance per dollar. What you'd really need is transactional memory, not just persistent memory. And you'd need a database engine written completely from the ground up to use this memory. Probably several hundred KLoC of tricky code (and all the joys that come with parallel programming), targeted at hardware that doesn't even exist yet. Sounds very expensive and risky to me. I think eventually we'll see something in this line of technology become commercially available. But I'm not surprised that Intel pulled out.
  - gavinray 4y ago
    > " And you'd need a database engine written completely from the ground up to use this memory." You mean like the one from one of CMU's research projects, that Pavlo et al have written about? https://db.cs.cmu.edu/projects/nvm/ https://db.cs.cmu.edu/projects/nvm/ "How to Build a Non-Volatile Memory Database Management System," in Proceedings of the 2017 ACM International Conference on Management of Data, 2017 "Spitfire: A Three-Tier Buffer Manager for Volatile and Non-Volatile Memory," in Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 2195-2207.
  - xyzzy123 4y ago
    Optane was pretty great for fs and db logs / journals. That was pretty much its killer (practical) use case.
- Melatonic 4y ago
  It could have made a big difference but it was adoption that was the problem. Swapping an SSD from a HDD is very easy and straight forward for most people - SATA cables are also well designed and easy to understand. Its almost like a USB cable in an L shape. Completely changing the format? That is a huge obstacle. NVME SSD's also came along soon enough we got something that was much, much faster than SATA SSD's and eventually I think it just became clear it was diminishing returns. That being said I am still hoping for an Optane replacement.
- paulmd 4y ago
  > One of the things that drove mass SSD adoption was the experience was plainly better. Even the most illiterate user could see how fucking fast an SSD machine booted and how much more responsive they were. > SSD to Optane? There's no practical difference. No, that's false. That's true of NVMe in general but Optane specifically is really good at the low-queue-depth random-4k workloads that characterize consumer use-cases. Optane actually is extremely fast at things like OS patches that normally take a few minutes even on NVMe flash let alone sata. They boot faster, applications launch faster, etc. Normally those things don't have lots of parallel threads running so queue depth is low and flash doesn't perform very well (as in, SATA and NVMe are nearly indistinguishable), but Optane really helps. Is that worth paying 10x as much for? No, probably not, but that's the real problem, not the general lack of performance. Optane is actually noticeable specifically for being the first thing that actually provides a noticeable improvement above and beyond SATA SSDs for consumer use-cases. It's just also too expensive to be justifiable.
  - PaulHoule 4y ago
    Although "client computing" is still the largest source of Intel revenue, and its scale was what made the x86 architecture able to crush legacy server architectures, I think Intel takes it for granted relative to the data center. It would have been very interesting to see an "all Optane" tablet device but it would have been hard to make work: a modest amount of storage would be affordable but it would take a clean sheet OS to maximize performance and economics. I was skeptical about Optane DIMMs. Optane SSDs were the fastest SSD you'd ever seen, but the DIMMS were the slowest you'd seen in a while. Although they were faster than Optane SSDs, they were slower than RAM if you replaced RAM 1-1. Optane was denser and cheaper than ordinary RAM so you could pack your machine with a huge quantity of slow RAM. If access patterns and cache behavior permit, you could build systems of a certain problem size that perform well with Optane. Advanced programming techniques can help. For every big problem there are many little problems (don't need Optane) and some problems are too big for Optane. Many problems in the size range for Optane can be implemented to conserve RAM and have access patterns that stream well to mass storage. Thus Optane faced fierce competition.
    - 4y ago
- christkv 4y ago
  They should have sold it as level 4 cache
  - skyde 4y ago
    you mean swap drive ?
    - cleansingfire 4y ago
      That's what I used the <$10 16gb Optane sticks for, when I had a batch of terrible small zero cache SSDs. Just to extend the performant life of the hardware I was stuck with. Worked ok.
- mort96 4y ago
  What's more, Optane was sold as... a cache to have in front of your spinning rust. In an age where SSDs were cheap enough that you could just get a few TB of SSD and have all your data access sped up, not just what happens to be in cache.
fiat_fandango 4y ago
Low latency optane ssd's are incredible for so many reasons - really hope those don't become discontinued as well :( Many here seem to miss how deeply complex SSD's are and how truly different optane is from most consumer and enterprise SSD offerings. Most consumer ssd's only optimize for sequential reads and writes and at that max throughput, not latency for random access. Glossing over a lot of detail, but this is where optane truly shines.
- paulmd 4y ago
  > Many here seem to miss how deeply complex SSD's are and how truly different optane is from most consumer and enterprise SSD offerings. Most consumer ssd's only optimize for sequential reads and writes and at that max throughput, not latency for random access. Glossing over a lot of detail, but this is where optane truly shines. Yep. Optane was much much closer to an "ideal" block device. There is no block write/page erase mechanism, no garbage collection, no interpolation layer between you and the hardware at all actually. Reads and writes onto a platonic block device. Latency is very low for xpoint so that's fine, on paper. It also means, that for ZFS, it's basically fine even without the battery backup. The xpoint isn't lying to you about a committed write so there's no reason to have a caching layer to manage the physical flash pages, in the way that flash SSDs need backup power for their controllers to flush the cache.
HarHarVeryFunny 4y ago
It was never just about Optane. The joint Micron/Intel 3DXPoint technology was meant to be a new type of non-volatile memory who's performance was closer to memory than disk, leading to all sorts of revolutionary changes in the way memory was used. The trouble is the performance goals were never realized, so the revolutionary non-volatile main memory use case was not realized. Optane/SSD-usage seems to have been an attempt to at least salvage something from the investment given that it was fast enough for that, but I suppose it the end it can't have been cost competitive with NAND-based SSDs. Perhaps the economy of scale would have been different if the technology had met it's performance goals and been more widely used ?
gavinray 4y ago
At least the PMDK (Persistent Memory Development Kit) code works interoperably with both Optane and CXL: https://github.com/pmem/pmdk https://github.com/pmem/pmdk
neverartful 4y ago
Surprised that there was not even a single mention of Micron! My understanding is that it was a joint venture of some variety between Intel and Micron. I'm sad to see Optane being discontinued. I had very good performance results with it being incorporated into SDS cluster architectures.
- anonymousDan 4y ago
  Interesting - do you mean Optane PMem (i.e. where it is connected as a DIMM)? Or the Optane SSDs?
  - tpetry 4y ago
    Optane SSDs had been great! They have the fastest write latency which made them e.g. perfect for databases or the ZFS ZIL which is a write of any new block to a special space (on that optane ssd) that will later be rewritten to the slower ssd.
- PAPPPmAc 4y ago
  IIRC Intel bought Micron out of the 3DXpoint/Optane project in 2018, and (based a some hallway conversations at conferences) folks at Micron were not exactly enthusiastic about the early results from the technology or Intel's overhyping marketing years before then.
- fluoridation 4y ago
  It had good performance, but part of the problem is that Intel way oversold it at the beginning. They said they would be "1000x faster, 1000x more enduring, and 10x more dense than NAND". In the end none of these promises materialized. They released these puny little drives that had a bit better latency than NVMe SSDs, that were locked to Intel systems, that OSs didn't know what to do with them, and that didn't quite fit anywhere in the memory hierarchy. Yeah, it found a few niche uses, but it was never going to be sustainable.
  - deleted 4y ago
    [deleted]

85 points by lichtenberger 4y ago

Optane was an interesting solution that was still looking for a question. One of the things that drove mass SSD adoption was the experience was plainly better. Even the most illiterate user could see how fucking fast an SSD machine booted and how much more responsive they were. SSD to Optane? There's no practical difference. The high end desktop users and serious gamers that have serious cash for their rigs turned up their noses at Optane because it turns out SSD sequential read was good enough for most use cases. At least good enough to not pay 3x more per GB. Nobody really cared about insane sustained write speed when the SLC caches did almost as well in most use cases. Plus as the interfaces scaled up, plain old NAND drives were ready to flood the bus with as much sequential read bandwidth that the bus would take. What could have saved it? I dunno. If Intel wanted to be anti-competitive they could have slapped on 8 PCIe lanes direct to the CPU exclusively for Optane on their desktop products. As we've found out, 4 lanes of NVMe to the CPU and the rest having to go through the chipset is some sort of unfunny, market segmentation joke. It could have forced high end users to grudgingly accept Optane as the path to the absolute best I/O and an actual, tangible difference in performance.

luma 4y ago
I think one of the major missing factors is lack of OS support for the sorts of features that NVDIMM could potentially bring to the table. Super-fast wake from deep hibernation is just a starting point, the real work is re-writing every line of code written since the dawn of computing which starts from the two assumptions "one kind of storage is persistent and slow" and "the other kind of storage is volatile and fast". HPE made noise about challenging those assumptions with The Machine but predictably lost interest as their own memristor technology lost steam.
orlp 4y ago
> What could have saved it? Persistent memory was/is a big topic in the database research world. The idea was that you would not need to worry about ACID since all your data is still there in the event of a power loss/crash. It would massively simplify database code. However as it turns out... persistent memory doesn't buy you all that much. You still effectively need a replay log to restore the state you were in before a crash event, and you still need all the restoration code. In the end it in most systems it just behaved as a very fast (and expensive) SSD. The benefits vs. price just wasn't there as a modern DB setup on a fast SSD + more RAM would get you much more performance per dollar. What you'd really need is transactional memory, not just persistent memory. And you'd need a database engine written completely from the ground up to use this memory. Probably several hundred KLoC of tricky code (and all the joys that come with parallel programming), targeted at hardware that doesn't even exist yet. Sounds very expensive and risky to me. I think eventually we'll see something in this line of technology become commercially available. But I'm not surprised that Intel pulled out.
- gavinray 4y ago
  > " And you'd need a database engine written completely from the ground up to use this memory." You mean like the one from one of CMU's research projects, that Pavlo et al have written about? https://db.cs.cmu.edu/projects/nvm/ https://db.cs.cmu.edu/projects/nvm/ "How to Build a Non-Volatile Memory Database Management System," in Proceedings of the 2017 ACM International Conference on Management of Data, 2017 "Spitfire: A Three-Tier Buffer Manager for Volatile and Non-Volatile Memory," in Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 2195-2207.
- xyzzy123 4y ago
  Optane was pretty great for fs and db logs / journals. That was pretty much its killer (practical) use case.
Melatonic 4y ago
It could have made a big difference but it was adoption that was the problem. Swapping an SSD from a HDD is very easy and straight forward for most people - SATA cables are also well designed and easy to understand. Its almost like a USB cable in an L shape. Completely changing the format? That is a huge obstacle. NVME SSD's also came along soon enough we got something that was much, much faster than SATA SSD's and eventually I think it just became clear it was diminishing returns. That being said I am still hoping for an Optane replacement.
paulmd 4y ago
> One of the things that drove mass SSD adoption was the experience was plainly better. Even the most illiterate user could see how fucking fast an SSD machine booted and how much more responsive they were. > SSD to Optane? There's no practical difference. No, that's false. That's true of NVMe in general but Optane specifically is really good at the low-queue-depth random-4k workloads that characterize consumer use-cases. Optane actually is extremely fast at things like OS patches that normally take a few minutes even on NVMe flash let alone sata. They boot faster, applications launch faster, etc. Normally those things don't have lots of parallel threads running so queue depth is low and flash doesn't perform very well (as in, SATA and NVMe are nearly indistinguishable), but Optane really helps. Is that worth paying 10x as much for? No, probably not, but that's the real problem, not the general lack of performance. Optane is actually noticeable specifically for being the first thing that actually provides a noticeable improvement above and beyond SATA SSDs for consumer use-cases. It's just also too expensive to be justifiable.
- PaulHoule 4y ago
  Although "client computing" is still the largest source of Intel revenue, and its scale was what made the x86 architecture able to crush legacy server architectures, I think Intel takes it for granted relative to the data center. It would have been very interesting to see an "all Optane" tablet device but it would have been hard to make work: a modest amount of storage would be affordable but it would take a clean sheet OS to maximize performance and economics. I was skeptical about Optane DIMMs. Optane SSDs were the fastest SSD you'd ever seen, but the DIMMS were the slowest you'd seen in a while. Although they were faster than Optane SSDs, they were slower than RAM if you replaced RAM 1-1. Optane was denser and cheaper than ordinary RAM so you could pack your machine with a huge quantity of slow RAM. If access patterns and cache behavior permit, you could build systems of a certain problem size that perform well with Optane. Advanced programming techniques can help. For every big problem there are many little problems (don't need Optane) and some problems are too big for Optane. Many problems in the size range for Optane can be implemented to conserve RAM and have access patterns that stream well to mass storage. Thus Optane faced fierce competition.
  - 4y ago
christkv 4y ago
They should have sold it as level 4 cache
- skyde 4y ago
  you mean swap drive ?
  - cleansingfire 4y ago
    That's what I used the <$10 16gb Optane sticks for, when I had a batch of terrible small zero cache SSDs. Just to extend the performant life of the hardware I was stuck with. Worked ok.
mort96 4y ago
What's more, Optane was sold as... a cache to have in front of your spinning rust. In an age where SSDs were cheap enough that you could just get a few TB of SSD and have all your data access sped up, not just what happens to be in cache.

fiat_fandango 4y ago

Low latency optane ssd's are incredible for so many reasons - really hope those don't become discontinued as well :( Many here seem to miss how deeply complex SSD's are and how truly different optane is from most consumer and enterprise SSD offerings. Most consumer ssd's only optimize for sequential reads and writes and at that max throughput, not latency for random access. Glossing over a lot of detail, but this is where optane truly shines.

paulmd 4y ago
> Many here seem to miss how deeply complex SSD's are and how truly different optane is from most consumer and enterprise SSD offerings. Most consumer ssd's only optimize for sequential reads and writes and at that max throughput, not latency for random access. Glossing over a lot of detail, but this is where optane truly shines. Yep. Optane was much much closer to an "ideal" block device. There is no block write/page erase mechanism, no garbage collection, no interpolation layer between you and the hardware at all actually. Reads and writes onto a platonic block device. Latency is very low for xpoint so that's fine, on paper. It also means, that for ZFS, it's basically fine even without the battery backup. The xpoint isn't lying to you about a committed write so there's no reason to have a caching layer to manage the physical flash pages, in the way that flash SSDs need backup power for their controllers to flush the cache.

HarHarVeryFunny 4y ago

It was never just about Optane. The joint Micron/Intel 3DXPoint technology was meant to be a new type of non-volatile memory who's performance was closer to memory than disk, leading to all sorts of revolutionary changes in the way memory was used. The trouble is the performance goals were never realized, so the revolutionary non-volatile main memory use case was not realized. Optane/SSD-usage seems to have been an attempt to at least salvage something from the investment given that it was fast enough for that, but I suppose it the end it can't have been cost competitive with NAND-based SSDs. Perhaps the economy of scale would have been different if the technology had met it's performance goals and been more widely used ?

gavinray 4y ago

At least the PMDK (Persistent Memory Development Kit) code works interoperably with both Optane and CXL: https://github.com/pmem/pmdk https://github.com/pmem/pmdk

neverartful 4y ago

Surprised that there was not even a single mention of Micron! My understanding is that it was a joint venture of some variety between Intel and Micron. I'm sad to see Optane being discontinued. I had very good performance results with it being incorporated into SDS cluster architectures.

anonymousDan 4y ago
Interesting - do you mean Optane PMem (i.e. where it is connected as a DIMM)? Or the Optane SSDs?
- tpetry 4y ago
  Optane SSDs had been great! They have the fastest write latency which made them e.g. perfect for databases or the ZFS ZIL which is a write of any new block to a special space (on that optane ssd) that will later be rewritten to the slower ssd.
PAPPPmAc 4y ago
IIRC Intel bought Micron out of the 3DXpoint/Optane project in 2018, and (based a some hallway conversations at conferences) folks at Micron were not exactly enthusiastic about the early results from the technology or Intel's overhyping marketing years before then.
fluoridation 4y ago
It had good performance, but part of the problem is that Intel way oversold it at the beginning. They said they would be "1000x faster, 1000x more enduring, and 10x more dense than NAND". In the end none of these promises materialized. They released these puny little drives that had a bit better latency than NVMe SSDs, that were locked to Intel systems, that OSs didn't know what to do with them, and that didn't quite fit anywhere in the memory hierarchy. Yeah, it found a few niche uses, but it was never going to be sustainable.
- deleted 4y ago
  [deleted]