12 ms·

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extre

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this piece I had the immense pleasure of getting early access to one of the next generation Optane devices, from Intel. The entire blog post is complaining about how great engineers have misconception about modern storage technology and yet to prove it the author had to obtain benchmarks from early access to next-generation devices...?! And to top it off, from this we conclude "the disconnect" is due to the APIs? Not, say, from the possibility that such blazing-fast components may very well not even exist in users' devices? I'm not saying the conclusions are wrong, but the logic surely doesn't follow... and honestly it's a little tasteless to criticize people's understanding if you're going to base the criticism on things they in all likelihood don't even have access to.

arka2147483647 6y ago
I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.
- Miraste 6y ago
  Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.
  - mehrdadn 6y ago
    3 years I think: https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint > It was announced in July 2015 and is available on the open market under brand names Optane (Intel) and subsequently QuantX (Micron) since April 2017. For comparison, look at how many decades it took SSDs to become commonplace: https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_SSDs https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_...
  - stingraycharles 6y ago
    I know a few optane deployments in finance, but other than that, it seems incredibly difficult to justify the steep price.
    - touisteur 6y ago
      They everywhere in NUCs. Small size but hey :-) that's how I reboot some apps ultrafast... Mmap my whole app-state and gooo
      - Cullinet 6y ago
        Please may I pray your forgiveness first for my naivety and 2nd forgive please my disarray because I am keeping the first question unchanged because I think it's the way a lot of people will be thinking about Optane and I realised that this duality the memory and storage modes and it's not finished there either, needs far better marketing than I think Intel capable of. I immediately assumed that you are running Optane DIMMS and enjoying life with great sequential speed and acceptably OK latency memory capacity of multiples of the normal RAM address space for your machines. I forgot not only the fact that Optane M2 and AIC skus exist but in fact make incredible value improvement on smaller systems. It's up the desktop computational capacity scale where use cases aren't as elegant as the fairly easy to circumscribe NUC applications. (Least those I can imagine). Second I forgot that Intel shipped the Scale MP authored RAM to block driver for Optane that let you tell your OS to treat storage attached Optane as RAM. Because of unrelated factors the systems acquisition I was running in H218 is starting over now and as a consequence I never tested components not sure about relevance and 2nd order road map business effects. So since the only pixels I've read about the RAM to block driver option have been 9n the launch specifications and my 3 comments about this since I have no idea if you actually can use storage Optane as RAM. (Dropping the semantics "storage class" and "memory class" would be my first edict at the helm of Intel strategy. I bet this nomenclature actually confuses the internal planning and delivery itself it's so insidiously confusing I could write one of those hideous corporate communications "style guide" about the possibilities for disaster this way ) Below is my original question I still would dearly appreciate learning your answer to. But I owe you a apology for doubting you in the original because I originally thought that Optane could my to.a lot of beating in a 1l chassis and less. Check out Patrick at Serve the Home for a super series of reviews of inexpensive and bargain performance mini NUC style desktops I'm sure you get no better than with a stick of Optane added.(To a PCIE riser adapter n.b. some of these tiny models give you bifurcation on their only pcie slot which could transform your use cases. [Original below / Intel "new unit of computing" skus have block storage drivers for Optane??? touisteur I must beg your indulgence my desire for knowledge! Can you possibly post a sky / model number of the most capable NUC you have working with Optane?
  - echlebek 6y ago
    Consumer NVMe devices can deliver GB/s I/O and hundreds of thousands of iops. The article's point doesn't hinge on Optane at all.
    - mistrial9 6y ago
      please ask my several NVMe devices to take notice ! actual performance under Linux OS is far less than that, here
      - snovv_crash 6y ago
        I just tested my laptop with the Ubuntu benchmark tool on the partition editor. 3.5GB/s read on 100MB chunks.
        segfaultbuserr 6y ago
        > Ubuntu benchmark tool on the partition editor Known as "GNOME Disks". https://en.wikipedia.org/wiki/GNOME_Disks https://en.wikipedia.org/wiki/GNOME_Disks
      - magicalhippo 6y ago
        Single-thread, single-queue performance is much lower than the max with good NVMe devices. With increased concurrency and deeper queues, my Samsung 960 Pro which has been running my Windows 10 desktop for several years still can do 294k random 4k reads IOPS, and 2.5GB/s sequential read.
      - __d 6y ago
        I think that's the entire point of this article: the existing system software APIs that we use aren't a good abstraction for the capabilities of the underlying hardware, leading to poor performance.
      - deleted 6y ago
        [deleted]
  - vbezhenar 6y ago
    Optane in consumer devices is in a weird place. Write durability does not matter, SSD manufacturers reduce rewrite count with every new generation in the name of the cheaper prices and users are happy to accept that downgrade (see Samsung 980 Pro vs Samsung 970 Pro for a recent occasion). Speed matters to some extent. Many users don't observe performance improvements in their typical tasks when upgrading from SATA SSD to M.2 PCI-E SSD, so they definitely wouldn't notice speed improvement upgrading from M.2 to Optane. And yet Optane's cost is not cheap. I just don't see place for Optane in any user devices. It's either servers, but it's not obvious choice there or tiny market for benchmark users. May be someone like Apple with its vertical integration could exploit it extreme speed to achieve something awesome, but I doubt it, they don't like to depend on a single supplier.
    - imtringued 6y ago
      >Optane in consumer devices is in a weird place. Write durability does not matter, SSD manufacturers reduce rewrite count with every new generation in the name of the cheaper prices and users are happy to accept that downgrade (see Samsung 980 Pro vs Samsung 970 Pro for a recent occasion). The odd thing about rewrite count is that it shrinks very quickly the bigger your drive is. In the past you used to have a 128GB boot SSD and install a few games on it. This meant that you constantly deleted and reinstalled games on that low capacity SSD increasing the rewrite count. Nowadays you can get 500GB SSDs for a decent price. If you can install every game then you might never have to rewrite data on it through uninstallation/installation. A lot of files are only written once and then never changed. With greater capacity you can have more "write once" files.
      - rasz 6y ago
        You the consumer might not feel the need to rewrite, but the drive itself has to do this periodically as a matter of internal Flash maintenance. Even Read Disturb (think ram RowHammer) is a thing for NAND. Then you have MLC/TLC technology in conjunction with modern small geometries leading to faster leakage (data persistence) and you end up with failures like Samsung 840 slowing down to a crawl, or older driver forgetting all data after being unplugged for a while, leading to Samsung 840 firmware update forcing the drive to rewrite itself periodically in the background. https://www.techspot.com/news/60362-samsung-fix-slow-840-evo-ssds-periodic-refresh.html https://www.techspot.com/news/60362-samsung-fix-slow-840-evo...
  - mtgx 6y ago
    Yeah, I'd say ReRAM SSDs will have a better chance of becoming mainstream, and with better performance/$, too. I think Sony is planning on commercializing it next year. Probably Western Digital, too.
rtkwe 6y ago
Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.
- danuker 6y ago
  > network storage Do you mean cloud storage? as in, other people's computers?
  - karamanolev 6y ago
    I conjecture he means SANs, iSCSI, NFS, Fibre Channel and other on-prem, but still not local to the server where the compute is running.
  - aden1ne 6y ago
    It probably means NFS.
  - mikepurvis 6y ago
    It's a pretty common pattern to have a fleet of big beefy VM hosts all backed by a single giant SAN on a 10gbe switch. This lets you do things like seamlessly migrate a VM from one host to another, or do a high availability thing with multiple synchronized instances and automatic failover (VMWare called this all "vMotion"). In any case, lots of bandwidth to the storage, but also high latency, at least relative to a locally-connected SATA or PCIe SSD. So yeah, if that's your setup, you don't have much of an option in between your SAN and allocating an in-machine ramdisk, which will be super fast and low latency, but also extremely high cost.
    - tpurves 6y ago
      Why not consider nVME in this case then as cheaper than RAM, slower than RAM, but faster than network storage? I don't know how you handle concurrency btwn VMs or virtualize that storage, but there must be some standard for that?
Cullinet 6y ago
Intel has a storage layer running Optane for meta and logs called DAOS. ^0 Intel DAOS was news to me only last week, a oversight I'm profoundly embarrassed by given I'm responsible for a meaningful amount of storage iron due to be replaced with orders to spend to estimate eliminating dependencies if performance comes plenty with sufficient reduction of vendor lock in. Lock in is a relative thing in enterprise storage. Apple could have been forgiven a eternity of nannying for a good ZFS port but I have always believed that Steve knew Larry coveted SUN for ZFS alone, years before that forced wedding. I ran my business on VMS. RDB the management system Palmer gifted to Oracle was the upgrade option above licensing RMS Record Management System and RMS is plenty capable being foundational to the hallowed for good reason cluster capabilities VMS is remembered for in the same ways sclerotic drunks on their death beds remember their Sunday school teacher's good words on moderation. ZFS exposed to a new and enthusiastic development community unfettered with nuisance concepts precisely those which were cast off with exhilarating energy by the NO-SQL movement. would be a genuine threat provided with functional basic data integrity underpinning their efforts during the pertinent time.
Cullinet 6y ago
Intel DAOS isn't a file system but the feature set begs one imo: https://blocksandfiles.com/2019/11/28/intel-daos-high-performance-storage-file-system-explainer/ https://blocksandfiles.com/2019/11/28/intel-daos-high-perfor...

arka2147483647 6y ago

I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.

Miraste 6y ago
Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.
- mehrdadn 6y ago
  3 years I think: https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint > It was announced in July 2015 and is available on the open market under brand names Optane (Intel) and subsequently QuantX (Micron) since April 2017. For comparison, look at how many decades it took SSDs to become commonplace: https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_SSDs https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_...
- stingraycharles 6y ago
  I know a few optane deployments in finance, but other than that, it seems incredibly difficult to justify the steep price.
  - touisteur 6y ago
    They everywhere in NUCs. Small size but hey :-) that's how I reboot some apps ultrafast... Mmap my whole app-state and gooo
    - Cullinet 6y ago
      Please may I pray your forgiveness first for my naivety and 2nd forgive please my disarray because I am keeping the first question unchanged because I think it's the way a lot of people will be thinking about Optane and I realised that this duality the memory and storage modes and it's not finished there either, needs far better marketing than I think Intel capable of. I immediately assumed that you are running Optane DIMMS and enjoying life with great sequential speed and acceptably OK latency memory capacity of multiples of the normal RAM address space for your machines. I forgot not only the fact that Optane M2 and AIC skus exist but in fact make incredible value improvement on smaller systems. It's up the desktop computational capacity scale where use cases aren't as elegant as the fairly easy to circumscribe NUC applications. (Least those I can imagine). Second I forgot that Intel shipped the Scale MP authored RAM to block driver for Optane that let you tell your OS to treat storage attached Optane as RAM. Because of unrelated factors the systems acquisition I was running in H218 is starting over now and as a consequence I never tested components not sure about relevance and 2nd order road map business effects. So since the only pixels I've read about the RAM to block driver option have been 9n the launch specifications and my 3 comments about this since I have no idea if you actually can use storage Optane as RAM. (Dropping the semantics "storage class" and "memory class" would be my first edict at the helm of Intel strategy. I bet this nomenclature actually confuses the internal planning and delivery itself it's so insidiously confusing I could write one of those hideous corporate communications "style guide" about the possibilities for disaster this way ) Below is my original question I still would dearly appreciate learning your answer to. But I owe you a apology for doubting you in the original because I originally thought that Optane could my to.a lot of beating in a 1l chassis and less. Check out Patrick at Serve the Home for a super series of reviews of inexpensive and bargain performance mini NUC style desktops I'm sure you get no better than with a stick of Optane added.(To a PCIE riser adapter n.b. some of these tiny models give you bifurcation on their only pcie slot which could transform your use cases. [Original below / Intel "new unit of computing" skus have block storage drivers for Optane??? touisteur I must beg your indulgence my desire for knowledge! Can you possibly post a sky / model number of the most capable NUC you have working with Optane?
- echlebek 6y ago
  Consumer NVMe devices can deliver GB/s I/O and hundreds of thousands of iops. The article's point doesn't hinge on Optane at all.
  - mistrial9 6y ago
    please ask my several NVMe devices to take notice ! actual performance under Linux OS is far less than that, here
    - snovv_crash 6y ago
      I just tested my laptop with the Ubuntu benchmark tool on the partition editor. 3.5GB/s read on 100MB chunks.
      - segfaultbuserr 6y ago
        > Ubuntu benchmark tool on the partition editor Known as "GNOME Disks". https://en.wikipedia.org/wiki/GNOME_Disks https://en.wikipedia.org/wiki/GNOME_Disks
    - magicalhippo 6y ago
      Single-thread, single-queue performance is much lower than the max with good NVMe devices. With increased concurrency and deeper queues, my Samsung 960 Pro which has been running my Windows 10 desktop for several years still can do 294k random 4k reads IOPS, and 2.5GB/s sequential read.
    - __d 6y ago
      I think that's the entire point of this article: the existing system software APIs that we use aren't a good abstraction for the capabilities of the underlying hardware, leading to poor performance.
    - deleted 6y ago
      [deleted]
- vbezhenar 6y ago
  Optane in consumer devices is in a weird place. Write durability does not matter, SSD manufacturers reduce rewrite count with every new generation in the name of the cheaper prices and users are happy to accept that downgrade (see Samsung 980 Pro vs Samsung 970 Pro for a recent occasion). Speed matters to some extent. Many users don't observe performance improvements in their typical tasks when upgrading from SATA SSD to M.2 PCI-E SSD, so they definitely wouldn't notice speed improvement upgrading from M.2 to Optane. And yet Optane's cost is not cheap. I just don't see place for Optane in any user devices. It's either servers, but it's not obvious choice there or tiny market for benchmark users. May be someone like Apple with its vertical integration could exploit it extreme speed to achieve something awesome, but I doubt it, they don't like to depend on a single supplier.
  - imtringued 6y ago
    >Optane in consumer devices is in a weird place. Write durability does not matter, SSD manufacturers reduce rewrite count with every new generation in the name of the cheaper prices and users are happy to accept that downgrade (see Samsung 980 Pro vs Samsung 970 Pro for a recent occasion). The odd thing about rewrite count is that it shrinks very quickly the bigger your drive is. In the past you used to have a 128GB boot SSD and install a few games on it. This meant that you constantly deleted and reinstalled games on that low capacity SSD increasing the rewrite count. Nowadays you can get 500GB SSDs for a decent price. If you can install every game then you might never have to rewrite data on it through uninstallation/installation. A lot of files are only written once and then never changed. With greater capacity you can have more "write once" files.
    - rasz 6y ago
      You the consumer might not feel the need to rewrite, but the drive itself has to do this periodically as a matter of internal Flash maintenance. Even Read Disturb (think ram RowHammer) is a thing for NAND. Then you have MLC/TLC technology in conjunction with modern small geometries leading to faster leakage (data persistence) and you end up with failures like Samsung 840 slowing down to a crawl, or older driver forgetting all data after being unplugged for a while, leading to Samsung 840 firmware update forcing the drive to rewrite itself periodically in the background. https://www.techspot.com/news/60362-samsung-fix-slow-840-evo-ssds-periodic-refresh.html https://www.techspot.com/news/60362-samsung-fix-slow-840-evo...
- mtgx 6y ago
  Yeah, I'd say ReRAM SSDs will have a better chance of becoming mainstream, and with better performance/$, too. I think Sony is planning on commercializing it next year. Probably Western Digital, too.

rtkwe 6y ago

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

danuker 6y ago
> network storage Do you mean cloud storage? as in, other people's computers?
- karamanolev 6y ago
  I conjecture he means SANs, iSCSI, NFS, Fibre Channel and other on-prem, but still not local to the server where the compute is running.
- aden1ne 6y ago
  It probably means NFS.
- mikepurvis 6y ago
  It's a pretty common pattern to have a fleet of big beefy VM hosts all backed by a single giant SAN on a 10gbe switch. This lets you do things like seamlessly migrate a VM from one host to another, or do a high availability thing with multiple synchronized instances and automatic failover (VMWare called this all "vMotion"). In any case, lots of bandwidth to the storage, but also high latency, at least relative to a locally-connected SATA or PCIe SSD. So yeah, if that's your setup, you don't have much of an option in between your SAN and allocating an in-machine ramdisk, which will be super fast and low latency, but also extremely high cost.
  - tpurves 6y ago
    Why not consider nVME in this case then as cheaper than RAM, slower than RAM, but faster than network storage? I don't know how you handle concurrency btwn VMs or virtualize that storage, but there must be some standard for that?

Cullinet 6y ago

Intel has a storage layer running Optane for meta and logs called DAOS. ^0 Intel DAOS was news to me only last week, a oversight I'm profoundly embarrassed by given I'm responsible for a meaningful amount of storage iron due to be replaced with orders to spend to estimate eliminating dependencies if performance comes plenty with sufficient reduction of vendor lock in. Lock in is a relative thing in enterprise storage. Apple could have been forgiven a eternity of nannying for a good ZFS port but I have always believed that Steve knew Larry coveted SUN for ZFS alone, years before that forced wedding. I ran my business on VMS. RDB the management system Palmer gifted to Oracle was the upgrade option above licensing RMS Record Management System and RMS is plenty capable being foundational to the hallowed for good reason cluster capabilities VMS is remembered for in the same ways sclerotic drunks on their death beds remember their Sunday school teacher's good words on moderation. ZFS exposed to a new and enthusiastic development community unfettered with nuisance concepts precisely those which were cast off with exhilarating energy by the NO-SQL movement. would be a genuine threat provided with functional basic data integrity underpinning their efforts during the pertinent time.

Cullinet 6y ago

Intel DAOS isn't a file system but the feature set begs one imo: https://blocksandfiles.com/2019/11/28/intel-daos-high-performance-storage-file-system-explainer/ https://blocksandfiles.com/2019/11/28/intel-daos-high-perfor...