5 ms·
It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good.
by hbogert 7mo ago
It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good. Some databases or journals for something like zfs really benefited from this.
- bombcar 7mo agoIt feels like everyone figured out what to do with them and how just about when they stopped making them.
- timschmidt 7mo agoSame for the Larabee / Knights architecture. Would sure be fun to play around with a 500 core Knights CPU with a couple TB of optane for LLM inference. Intel's got an amazing record of axing projects as soon as they've done the hard work of building an ecosystem.
- zozbot234 7mo ago> 500 core The newest fully E-core based Xeon CPUs have reached that figure by now, at least in dual-socket configs.
- timschmidt 7mo agoYup. And high end GPU compute now has on-package HBM like Knight's had a decade ago, and those new Intel CPUs are finally shipping with AVX reliably again. We lost a decade for workloads that would benefit from both.
- fc417fc802 7mo agoBut I'm surprised PCIe based CPU+RAM modules aren't a "thing" since that's basically what a GPU is if you ignore all the rather fundamental differences. Seems like it would be convenient to cheaply attach additional compute without worrying about all the other stuff. I suppose I'm just reinventing SXM at this point. The BC-250 comes close but despite the formfactor it isn't actually a PCIe card. Although if it integrated a 100 Gbit SFP slot it might actually be superior to a solution that resided in a host system. But the BC-250 is very much an anomaly as opposed to the norm.
- zozbot234 7mo agoYou need CXL to extend the cache coherency properties of actual RAM over a remote link. That's costly tech. Otherwise, you're relying on the OS (and even the compiler/basic libraries, since you need to make fences, etc. OS-visible) to paper over the differences by doing its own implementation of distributed shared memory (this is known as a 'SSI' or single-system-image approach) which has significant challenges and is closer to the spirit of setting up swap.
- fc417fc802 7mo agoI didn't mean anything like that. Just the equivalent of a GPU with the ability to run arbitrary CPU oriented programs. Of course GPUs do many tasks very well but there are also plenty of problems that aren't well suited to them. Well I suppose I've answered my own question at this point. There probably just aren't enough real world problems that aren't amenable to running on a GPU while also being either compute or memory bandwidth bound. Still the near-monoculture does strike me as odd. I guess GPUs have bifurcated into enterprise versus consumer at this point but otherwise all we've got is a single CPU example from over a decade ago and a single alternative take on the concept from Fujitsu. Is it just due to the obscene cost of masks for modern process nodes?
- akvadrako 7mo agoThings like that existed in the category of accelerator cards. Xeon Phi (Knights) is one example, focused on core count. Some from HP have soldered on SSDs too. You also had blade servers which is more focused on that use case, though that's going out of style. I don't think PCIe is really a good fit for general CPU tasks. You need big heatsinks and power and can't fit that much RAM on board.
- epistasis 7mo agoWhen most people are running databases on AWS RDS, or on ridiculous EBS drives with insanely low throughput and latency, it makes sense to me. There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease. Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.
- amluto 7mo ago> There are very few applications that benefit from such low latency Basically any RDBMS? MySQL and Postgres both benefit from high performance storage, but too many customers have moved into the cloud where you can’t get NVMe-like performance for durable storage for anything remotely close to a worthwhile price.
- epistasis 7mo agoI'm saying that there are very few downstream applications that use databases that benefit from reducing latency beyond the slow performance of the cloud. Running your database on VMs or baremetal gives better performance, but almost no applications built on databases bother to do it.
- p-e-w 7mo agoOptane was a victim of its own hype, such as “entirely new physics”, or “as fast as RAM, but persistent”. The reality felt like a failure afterwards even though it was still revolutionary, objectively speaking.
- zozbot234 7mo agoOptane didn't sell because they focused on their weird persistent DIMM sticks, which are a nightmare for enterprise where for many ordinary purposes you want ephemeral data that disappears as soon as you cut power. Thet should have focused on making ordinary storage and solving the interconnect bandwidth and latency problems differently, such as with more up-to-date PCIe standards.
- jauntywundrkind 7mo agoI don't think that would be my main complaint. Sticking optane in a dimm was just awkward as hell. You now have different bits of memory with very different characteristics, & you lose a ton of bandwidth. If CXL was around at the time it would have been such a nice fit, allowing for much lower latency access. It also seems like in spite of the bad fit, there were enough regular options drives, and they were indeed pretty incredible. Good endurance, reasonable price (and cheap as dirt if you consider that endurance/lifecycle cost!), some just fantastic performance figures. My conclusion is that alas there just aren't many people in the world who are serious about storage performance.
- hrmtst93837 7mo ago[flagged]
- PunchyHamster 7mo ago> and file systems built around NAND assumptions, a lot of the upside got shaved off before users ever saw it. What file systems ? Most common one you'd find would be ext4 or XFS and neither of them are
- ksec 7mo ago>Which is weird.... It isn't weird at all. I would be surprised if it ever succeed in the first place. Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.
- jauntywundrkind 7mo agoCost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD. For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.
- wtallis 7mo agoOptane was in the market during a time when the mainstream trend in the SSD industry was all about sacrificing endurance to get higher capacity. It's been several years, and I'm not seeing a lot of regrets from folks who moved to TLC and QLC NAND, and those products are more popular than ever. The niche that could actually make use of Optane's endurance was small and shrinking, and Intel had no roadmap to significantly improve Optane's $/GB which was unquestionably the technology's biggest weakness.
- raron 7mo ago> I'm not seeing a lot of regrets from folks who moved to TLC and QLC NAND, and those products are more popular than ever. That's interesting. Even TLC has huge limitations, but QLC is basically useless unless you use it as write-once-read-many memory. I wish I have bought a lot of SSDs when you could still buy MLC ones.
- cogman10 7mo agoIMO, the reason they didn't sell is the ideal usage for them is pairing them with some slow spinning disks. The issue Optane had is that SSD capacity grew dramatically while the price plummeted. The difference between Optane and SSDs was too small. Especially since the M.2 standard proliferated and SSDs took advantage of PCI-E performance. I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper. The ideal usage of optane was as a ZIL in ZFS.
- zozbot234 7mo agoThat may have been the ideal usage back in the day, but ideal usage now is just for setting up swap. Write-heavy workloads are king with Optane, and threshing to swap is the prototypical example of something that's so write-heavy it's a terrible fit for NAND. Optane might not have been "as fast as DRAM" but it was plenty close enough to be fit for purpose.
- mort96 7mo agoThat would be fine if I could put it in an M.2 slot. But all my computers already have RAM in their RAM slots, and even if I had a spare RAM slot, I don't know that I'd trust the software stack to treat one RAM slot as a drive... And their whole deal was making RAM persistent anyway, which isn't exactly what I want.
- amluto 7mo agoIntel did a spectacularly poor job with the ecosystem around the memory cells. They made two plays, and both were flops. 1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction). 2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently. These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.
- orion138 7mo agoOne of the many problems was trying to limit the use of Optane to Intel devices. They should have manufactured and sold Optane memory and let other players build on top of it at a low level.
- amluto 7mo ago> Optane memory Which “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.
- thesz 7mo agoIn "databases and journals" you rarely update just one byte, you do a transaction that updates data, several indexes and metadata. All of that needs to be atomic. Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streaming fashion. What is good, though, is that "blast radius" [1] of failure is smaller than usual - failed one byte write rarely corrupts more that one byte or cache line. SQLite has to deal with 512 (and even more) bytes long possible corruptions on most disks, with Optane it is not necessarily so. So, less data to copy, scan, etc. [1] https://sqlite.org/psow.html https://sqlite.org/psow.html
- PunchyHamster 7mo agoIt's not. You won't be writing one byte, ever (even if you had layers that actually supported less-than-block writes), because the overhead of instruction would be massive and you'd be murdering both latency and bandwidth for anything non-trivial
- tbrownaw 7mo ago> However small latency is, power failure is still faster. A fancy switching power supply with a friendly power factor (looks like a resistive load, rather than drawing more amps during the lower voltage parts of the waveform) actually will have non-zero fall time when suddenly unplugged.
- mort96 7mo agoI never understood what they're meant to do. Intel seemed to picture some future where RAM is persistent; but they were never close to fast enough to replace RAM, and the option to reboot in order to fix some weird state your system has gotten itself into is a feature of computers, not a problem to work around.
- trentnelson 7mo agoWhen the PDIMMs were used with an appropriate file system + kernel, it was pretty cool. NTFS + DAX + kernel support yielded a file system where mmap’ing didn’t page fault. No page faults because the file content is already there, instantly. So if you had mmap heavy read/write workloads… you could do some pretty cool stuff.