10 ms·

The Intel Optane SSD DC P4800X (375GB) Review: Testing 3D XPoint Performance

134 points by randta 9y ago

saosebastiao 9y ago
Impressive. On a semi-related note, the fact that cloud providers offer managed Postgres databases is great, but things like this keep pushing me to think about bare metal in colo. A $20,000 server/backup combo with a couple of these will give me 5x the performance of a server that costs me $70k/year on AWS before provisioned IOPS. That's a huge gap to play with for taking care of your own maintenance costs.
- dboreham 9y ago
  It has always been the case that owned metal is much cheaper than cloud. Cloud wins for things like very small machines (you can't buy 20% of a metal box); temporary deployments; and startups where the investors don't want to be on the hook for big $$ capex that they end up needing to shift on eBay post flame-out. Also where you just don't have the ability to run hardware (increasingly the case) and where for whatever reason the cost to deploy and run metal would add up to more than your $70k delta (easy if it needs one more person on staff). Also remember that funky storage card may have a firmware bug that costs you hundreds of hours to track down and get resolved. That time could have been burned by Jeff Bezos' guys instead..
  - Toast_ 9y ago
    >where you just don't have the ability to run hardware (increasingly the case) Softlayer has 'on demand' bare metal plans. I'm not sure how cost effective it would be compared compared to running your own box, but the simplicity is nice and they had me up and running in under 2 hours (assembled with customizations). http://www.softlayer.com/bare-metal-servers http://www.softlayer.com/bare-metal-servers
    - utternerd 9y ago
      As a customer, I can tell you we're very, very happy (Singapore SoftLayer, our Asian DC). No upfront capex, I can add or subtract hardware as necessary (I can say the same, they usually take about 2 hours to have it up and configured as requested), and my favorite benefit is when new hardware is available, we just spin up new servers and retire the old ones; bill ends up being the ~same. It runs substantially cheaper than our colo here in Irvine, but that's also not apples-to-apples. That said, I don't plan on continuing with colos unless there's a pressing reason, after almost 15 years of them being my default. Bare Metal is the sweet spot in my eyes, the benefits of cloud without the downside of shared/inconsistent resources.
    - toast0 9y ago
      How custom can you get with the hourly machines? The order page (without logging in) looks like you have to pick a ram size (and they don't go beyond 256GB), and you can only go up to 4 disks. That said, I know the order page for monthly has maximum ram sizes listed below some of the monthly machines I manage, so maybe the hourly page is a simplified version of the truth too.
      - Toast_ 9y ago
        I'm not really sure about hourly plans. When I was configuring my machine (~1-2 years ago) I was able to select hourly or monthly billing (hourly would've been outrageously expensive).
  - chx 9y ago
    I never understood this. Just rent a dedicated box or ten. There's no capex, it's way cheaper to have 3-5 times the capability you will need just sitting there instead of renting stuff from Amazon with all the I/O unpredictability, complexity, w/e. People talk as if colo and cloud is the only solution -- and I am like, rent dedicated is the only solution. I do not readily know how huge your site needs to be for colo to worth it but Examiner was ~75th on Quantcast and we ran it on a dozen dedicated boxes and colo would'v been ridiculously more expensive.
    - cookiecaper 9y ago
      Renting a dedicated box doesn't have that cool "we can scale INFINITELY!" vibe. ;)
      - chx 9y ago
        YAGNI.
        problems 9y ago
        Exactly. The only time the cloud services really pay off are when you need to scale up or down massive capacity overnight. The odds of that happening at most companies are very, very slim, even most of the web companies. If you're really concerned about that though, I'd say go for the cloud option - just make it so you can start small on a single physical server, but can scale onto the cloud, then migrate that to physical hardware as needed. With this set up, you save massive cash and avoid vendor lock-in.
        mrep 9y ago
        Or if you are a team like mine that develops/manages 200 servers, 15 RDS database instances, 500 TB of compressed S3 data, and provides accessibility to all of that through API's with only 5 developers because of how easy AWS makes it for us. While our server costs are probably higher than going bare metal, how many developers would it take you to manage all of that on bare metal hosts?
- mozumder 9y ago
  You don't even need to co-lo. Just build a system for your home on a static IP address with a business account from your ISP. It's so much cheaper than AWS, and you get unlimited bandwidth.
- zxcmx 9y ago
  Yep but the FTEs you need are 100k+ each, managing baremetal servers is almost certainly not your core business, and you're running the risk of gitlab style fat fingering dinking around with low level stuff that you did to save money.
mtanski 9y ago
Some people like to claim that old storage technologies go away. But in reality old storage technologies live on along side of new... all that happens is that we end up having more tiers to deal with. Drum magnetic memory has been replace but we still have spinning rust, tape, optical, DRAM, SRAM, SSD...
- ajross 9y ago
  Drum memory was still spinning rust, the only difference is the axis of rotation.
- agumonkey 9y ago
  My floppies would like to have a word with you.
  - mtanski 9y ago
    It lives on in the save icon. But in all seriousness you're right, floppy as a medium for data storage is dead. Unless you're the US gov. According to Wikipedia: "In May 2016 the United States Government Accountability Office released a report that covered the need to upgrade or replace legacy computer systems within Federal Agencies. According to this document, old IBM Series/1 minicomputers running on 8-inch floppy disks are still used to coordinate "the operational functions of the United States’ nuclear forces..." The government plans to update some of the technology by the end of the 2017 fiscal year." [1] 1. https://en.wikipedia.org/wiki/Floppy_disk#Use_in_the_early_21st_century https://en.wikipedia.org/wiki/Floppy_disk#Use_in_the_early_2...
    - agumonkey 9y ago
      Optical discs are relegated to back shelves too. Hard to beat cheap rewritable nand flash. ps: I'd be curious about a tiny, 2017 laptop friendly optical disc storage. Say a 2.5" RW BD layer in cartridge. A modern mini disc.
      - blacksmith_tb 9y ago
randta 9y ago
Better than NAND performance if ever so slightly for a first generation memory product but I still feel like it would not able to scale to DRAM speeds. The search for a universal memory goes on...
- Analemma_ 9y ago
  They're not advertising it as universal memory or a DRAM replacement though, so I'm not sure why you would make that comparison. They're advertising it as sitting between RAM and an SSD on the storage hierarchy, and it seems to mostly deliver on that promise.
  - endorphone 9y ago
    One of the proposed uses for 3D XPoint has been replacing system memory, owing to its extremely low latency (which it has already delivered), high throughput, where it's early, and supposed almost unimaginable reliability/rewrites. In DIMM form it wouldn't have the overhead of a storage controller, could hypothetically be very parallel, etc. It's extremely early in the technology, and I imagine we will get there. The first SSDs were terrible compared to SSDs now, and 3D XPoint as a technology is extremely scalable and refinable.
    - Dylan16807 9y ago
      > extremely low latency (which it has already delivered) Are we looking at the same numbers? "probably under 10 microseconds" is pretty terrible compared to DRAM.
      - endorphone 9y ago
        3D XPoint has extremely low latency (e.g. 7 usec), and has already been demonstrated as such. Putting it through a PCI slot and a storage controller is not the same. The discussion is about running it as DIMM memory through a normal memory controller.
        9y ago
ganfortran 9y ago
The random access performance is IMPRESSIVE! It is going to bring a storm in DB landscape, since now we probably don't need to force to use data structure that tailored to perform with sequential writes, but all the other data structures as well? Very exciting indeed!
gigatexal 9y ago
Jeebus, that's one performant drive. http://www.anandtech.com/show/11209/intel-optane-ssd-dc-p4800x-review-a-deep-dive-into-3d-xpoint-enterprise-performance/4 http://www.anandtech.com/show/11209/intel-optane-ssd-dc-p480... -- I can't believe those numbers. The QD1 numbers are impressive to say the least.
bbulkow 9y ago
Aerospike CTO here. We did a lot of testing with this drive, and the interesting bit is that performance doesn't degrade under write pressure. With a NAND drive, when you push writes high ( usually needed in front-edge / microservice apps ), the read latencies take a real hit, and often get into millisecond-average at the device. Optane simply doesn't behave that way. All of that means you have to code to it differently. It isn't DRAM, it's not Flash. We've done the work to integrate and test at Aerospike ( semi-plug sorry ), so the system works. The XPoint tech gets really interesting later in the year, when three things happen: the higher density drives show up, Dell ships their high-NVMe chassis, and more clouds support it. Regarding cloud - IBM's BlueMix has announced Optane, and the other providers have a wide variety of plans. I can't comment more. Finally, Intel has been clear about putting this kind of tech on the memory bus, and that really opens the doors to some interesting choices, some data structure changes. That requires new coding, we're on it. Here's an interesting ComputerWorld article about our experience with Optane: http://www.computerworld.com/article/3188338/data-storage/what-aerospike-learned-from-testing-intels-superfast-optane-ssds.html http://www.computerworld.com/article/3188338/data-storage/wh...
- bch 9y ago
  > All of that means you have to code to it differently. > That requires new coding, we're on it. Can you give a rough example to guide my thinking? I understand OSes are considering ways that this has both traditional disk and RAM properties, and are sorting out storage subsystems of drivers, but I assume you're talking about something closer to user-level structures and algorithms?
  - wtallis 9y ago
    Databases have traditionally gone to great lengths to bypass whatever heuristics the operating system is using for I/O scheduling and caching so that they can implement their own optimizations while maintaining their data consistency guarantees. Any changes to OS behavior you can imagine are also potentially applicable to databases that take this approach.
  - bbulkow 9y ago
strictnein 9y ago
Another nice review of the same part: https://www.pcper.com/reviews/Storage/Intel-Optane-SSD-DC-P4800X-375GB-Review-Enterprise-3D-XPoint https://www.pcper.com/reviews/Storage/Intel-Optane-SSD-DC-P4...
frozenport 9y ago
At $5 per gb, I would instead buy RAM. Edit: why the downvotes? I'm currently doing image processing and using a ram drive with ~200GBs.
- analognoise 9y ago
  That sounds interesting, can you tell us more a about the system?
- Crespyl 9y ago
  The reason you would use this over RAM is persistence, it doesn't need to stay powered to keep the data. If all you need is a massive amount of temporary storage for some algorithm, you'll still need RAM, but if you want a stupidly fast backing store for a huge amount of source and then output data, this is pretty incredible.
  - Dylan16807 9y ago
    When your computation fits in RAM, what's the point of a lower latency backing store that does fewer gbps per dollar?
  - frozenport 9y ago
    Many of these systems have write caching enabled, which means that on critical power failure, some data will be corrupted. And often such system have back up supplies. Let's be clear here. The real reason people used this stuff was that it was 3x/4x cheaper than RAM.
- scurvy 9y ago
  Have you considered write-back cache scenarios? Can't use RAM there. Stuff like this would be fantastic for those applications.
elorant 9y ago
Which are the use cases for an SSD like that? Would I see improvements if I were to load my db on it or is it for working with bigger files?
- mozumder 9y ago
  These are going to be absolute beasts for databases. Databases are best on drives that have reduced latencies. Data access is random and keeps jumping around the drive for joins. Your SELECT statements are going to speed up in proportion to latency improvements.
- unholythree 9y ago
  First thing I think of is a Cache drive for a SSD array.
benlwalker 9y ago
I'm the technical lead for the Storage Performance Development Kit (http://spdk.io http://spdk.io). I have one of these in my development system and we're hoping to post benchmarks fairly soon. SPDK further reduces the latency at QD 1 by several microseconds. It's an impressive device!
- justinclift 9y ago
  Any word on real world endurance? There's some speculation these devices are massively overprovisioned due to the er... cells (or equivalent) wearing out much faster than early hype/info/etc. So, it'd be interesting to get a real world idea if these things are likely to explode badly ;) 6 months after purchase or not (etc). :)
ksec 9y ago
I am rather hoping this make large capacity DRAM some price drop.
- scurvy 9y ago
  I'll settle for NAND pricing to drop. There's a bit of large-scale NAND shortage right now. Companies are bearing the brunt of the shortage right now, but they'll past it on to consumers if it continues much longer.

134 points by randta 9y ago

Impressive. On a semi-related note, the fact that cloud providers offer managed Postgres databases is great, but things like this keep pushing me to think about bare metal in colo. A $20,000 server/backup combo with a couple of these will give me 5x the performance of a server that costs me $70k/year on AWS before provisioned IOPS. That's a huge gap to play with for taking care of your own maintenance costs.

dboreham 9y ago
It has always been the case that owned metal is much cheaper than cloud. Cloud wins for things like very small machines (you can't buy 20% of a metal box); temporary deployments; and startups where the investors don't want to be on the hook for big $$ capex that they end up needing to shift on eBay post flame-out. Also where you just don't have the ability to run hardware (increasingly the case) and where for whatever reason the cost to deploy and run metal would add up to more than your $70k delta (easy if it needs one more person on staff). Also remember that funky storage card may have a firmware bug that costs you hundreds of hours to track down and get resolved. That time could have been burned by Jeff Bezos' guys instead..
- Toast_ 9y ago
  >where you just don't have the ability to run hardware (increasingly the case) Softlayer has 'on demand' bare metal plans. I'm not sure how cost effective it would be compared compared to running your own box, but the simplicity is nice and they had me up and running in under 2 hours (assembled with customizations). http://www.softlayer.com/bare-metal-servers http://www.softlayer.com/bare-metal-servers
  - utternerd 9y ago
    As a customer, I can tell you we're very, very happy (Singapore SoftLayer, our Asian DC). No upfront capex, I can add or subtract hardware as necessary (I can say the same, they usually take about 2 hours to have it up and configured as requested), and my favorite benefit is when new hardware is available, we just spin up new servers and retire the old ones; bill ends up being the ~same. It runs substantially cheaper than our colo here in Irvine, but that's also not apples-to-apples. That said, I don't plan on continuing with colos unless there's a pressing reason, after almost 15 years of them being my default. Bare Metal is the sweet spot in my eyes, the benefits of cloud without the downside of shared/inconsistent resources.
  - toast0 9y ago
    How custom can you get with the hourly machines? The order page (without logging in) looks like you have to pick a ram size (and they don't go beyond 256GB), and you can only go up to 4 disks. That said, I know the order page for monthly has maximum ram sizes listed below some of the monthly machines I manage, so maybe the hourly page is a simplified version of the truth too.
    - Toast_ 9y ago
      I'm not really sure about hourly plans. When I was configuring my machine (~1-2 years ago) I was able to select hourly or monthly billing (hourly would've been outrageously expensive).
- chx 9y ago
  I never understood this. Just rent a dedicated box or ten. There's no capex, it's way cheaper to have 3-5 times the capability you will need just sitting there instead of renting stuff from Amazon with all the I/O unpredictability, complexity, w/e. People talk as if colo and cloud is the only solution -- and I am like, rent dedicated is the only solution. I do not readily know how huge your site needs to be for colo to worth it but Examiner was ~75th on Quantcast and we ran it on a dozen dedicated boxes and colo would'v been ridiculously more expensive.
  - cookiecaper 9y ago
    Renting a dedicated box doesn't have that cool "we can scale INFINITELY!" vibe. ;)
    - chx 9y ago
      YAGNI.
      - problems 9y ago
        Exactly. The only time the cloud services really pay off are when you need to scale up or down massive capacity overnight. The odds of that happening at most companies are very, very slim, even most of the web companies. If you're really concerned about that though, I'd say go for the cloud option - just make it so you can start small on a single physical server, but can scale onto the cloud, then migrate that to physical hardware as needed. With this set up, you save massive cash and avoid vendor lock-in.
        mrep 9y ago
        Or if you are a team like mine that develops/manages 200 servers, 15 RDS database instances, 500 TB of compressed S3 data, and provides accessibility to all of that through API's with only 5 developers because of how easy AWS makes it for us. While our server costs are probably higher than going bare metal, how many developers would it take you to manage all of that on bare metal hosts?
mozumder 9y ago
You don't even need to co-lo. Just build a system for your home on a static IP address with a business account from your ISP. It's so much cheaper than AWS, and you get unlimited bandwidth.
zxcmx 9y ago
Yep but the FTEs you need are 100k+ each, managing baremetal servers is almost certainly not your core business, and you're running the risk of gitlab style fat fingering dinking around with low level stuff that you did to save money.

mtanski 9y ago

Some people like to claim that old storage technologies go away. But in reality old storage technologies live on along side of new... all that happens is that we end up having more tiers to deal with. Drum magnetic memory has been replace but we still have spinning rust, tape, optical, DRAM, SRAM, SSD...

ajross 9y ago
Drum memory was still spinning rust, the only difference is the axis of rotation.
agumonkey 9y ago
My floppies would like to have a word with you.
- mtanski 9y ago
  It lives on in the save icon. But in all seriousness you're right, floppy as a medium for data storage is dead. Unless you're the US gov. According to Wikipedia: "In May 2016 the United States Government Accountability Office released a report that covered the need to upgrade or replace legacy computer systems within Federal Agencies. According to this document, old IBM Series/1 minicomputers running on 8-inch floppy disks are still used to coordinate "the operational functions of the United States’ nuclear forces..." The government plans to update some of the technology by the end of the 2017 fiscal year." [1] 1. https://en.wikipedia.org/wiki/Floppy_disk#Use_in_the_early_21st_century https://en.wikipedia.org/wiki/Floppy_disk#Use_in_the_early_2...
  - agumonkey 9y ago
    Optical discs are relegated to back shelves too. Hard to beat cheap rewritable nand flash. ps: I'd be curious about a tiny, 2017 laptop friendly optical disc storage. Say a 2.5" RW BD layer in cartridge. A modern mini disc.
    - blacksmith_tb 9y ago

randta 9y ago

Better than NAND performance if ever so slightly for a first generation memory product but I still feel like it would not able to scale to DRAM speeds. The search for a universal memory goes on...

Analemma_ 9y ago
They're not advertising it as universal memory or a DRAM replacement though, so I'm not sure why you would make that comparison. They're advertising it as sitting between RAM and an SSD on the storage hierarchy, and it seems to mostly deliver on that promise.
- endorphone 9y ago
  One of the proposed uses for 3D XPoint has been replacing system memory, owing to its extremely low latency (which it has already delivered), high throughput, where it's early, and supposed almost unimaginable reliability/rewrites. In DIMM form it wouldn't have the overhead of a storage controller, could hypothetically be very parallel, etc. It's extremely early in the technology, and I imagine we will get there. The first SSDs were terrible compared to SSDs now, and 3D XPoint as a technology is extremely scalable and refinable.
  - Dylan16807 9y ago
    > extremely low latency (which it has already delivered) Are we looking at the same numbers? "probably under 10 microseconds" is pretty terrible compared to DRAM.
    - endorphone 9y ago
      3D XPoint has extremely low latency (e.g. 7 usec), and has already been demonstrated as such. Putting it through a PCI slot and a storage controller is not the same. The discussion is about running it as DIMM memory through a normal memory controller.
      - 9y ago

ganfortran 9y ago

The random access performance is IMPRESSIVE! It is going to bring a storm in DB landscape, since now we probably don't need to force to use data structure that tailored to perform with sequential writes, but all the other data structures as well? Very exciting indeed!

gigatexal 9y ago

Jeebus, that's one performant drive. http://www.anandtech.com/show/11209/intel-optane-ssd-dc-p4800x-review-a-deep-dive-into-3d-xpoint-enterprise-performance/4 http://www.anandtech.com/show/11209/intel-optane-ssd-dc-p480... -- I can't believe those numbers. The QD1 numbers are impressive to say the least.

bbulkow 9y ago

Aerospike CTO here. We did a lot of testing with this drive, and the interesting bit is that performance doesn't degrade under write pressure. With a NAND drive, when you push writes high ( usually needed in front-edge / microservice apps ), the read latencies take a real hit, and often get into millisecond-average at the device. Optane simply doesn't behave that way. All of that means you have to code to it differently. It isn't DRAM, it's not Flash. We've done the work to integrate and test at Aerospike ( semi-plug sorry ), so the system works. The XPoint tech gets really interesting later in the year, when three things happen: the higher density drives show up, Dell ships their high-NVMe chassis, and more clouds support it. Regarding cloud - IBM's BlueMix has announced Optane, and the other providers have a wide variety of plans. I can't comment more. Finally, Intel has been clear about putting this kind of tech on the memory bus, and that really opens the doors to some interesting choices, some data structure changes. That requires new coding, we're on it. Here's an interesting ComputerWorld article about our experience with Optane: http://www.computerworld.com/article/3188338/data-storage/what-aerospike-learned-from-testing-intels-superfast-optane-ssds.html http://www.computerworld.com/article/3188338/data-storage/wh...

bch 9y ago
> All of that means you have to code to it differently. > That requires new coding, we're on it. Can you give a rough example to guide my thinking? I understand OSes are considering ways that this has both traditional disk and RAM properties, and are sorting out storage subsystems of drivers, but I assume you're talking about something closer to user-level structures and algorithms?
- wtallis 9y ago
  Databases have traditionally gone to great lengths to bypass whatever heuristics the operating system is using for I/O scheduling and caching so that they can implement their own optimizations while maintaining their data consistency guarantees. Any changes to OS behavior you can imagine are also potentially applicable to databases that take this approach.
- bbulkow 9y ago

strictnein 9y ago

Another nice review of the same part: https://www.pcper.com/reviews/Storage/Intel-Optane-SSD-DC-P4800X-375GB-Review-Enterprise-3D-XPoint https://www.pcper.com/reviews/Storage/Intel-Optane-SSD-DC-P4...

frozenport 9y ago

At $5 per gb, I would instead buy RAM. Edit: why the downvotes? I'm currently doing image processing and using a ram drive with ~200GBs.

analognoise 9y ago
That sounds interesting, can you tell us more a about the system?
Crespyl 9y ago
The reason you would use this over RAM is persistence, it doesn't need to stay powered to keep the data. If all you need is a massive amount of temporary storage for some algorithm, you'll still need RAM, but if you want a stupidly fast backing store for a huge amount of source and then output data, this is pretty incredible.
- Dylan16807 9y ago
  When your computation fits in RAM, what's the point of a lower latency backing store that does fewer gbps per dollar?
- frozenport 9y ago
  Many of these systems have write caching enabled, which means that on critical power failure, some data will be corrupted. And often such system have back up supplies. Let's be clear here. The real reason people used this stuff was that it was 3x/4x cheaper than RAM.
scurvy 9y ago
Have you considered write-back cache scenarios? Can't use RAM there. Stuff like this would be fantastic for those applications.

elorant 9y ago

Which are the use cases for an SSD like that? Would I see improvements if I were to load my db on it or is it for working with bigger files?

mozumder 9y ago
These are going to be absolute beasts for databases. Databases are best on drives that have reduced latencies. Data access is random and keeps jumping around the drive for joins. Your SELECT statements are going to speed up in proportion to latency improvements.
unholythree 9y ago
First thing I think of is a Cache drive for a SSD array.

benlwalker 9y ago

I'm the technical lead for the Storage Performance Development Kit (http://spdk.io http://spdk.io). I have one of these in my development system and we're hoping to post benchmarks fairly soon. SPDK further reduces the latency at QD 1 by several microseconds. It's an impressive device!

justinclift 9y ago
Any word on real world endurance? There's some speculation these devices are massively overprovisioned due to the er... cells (or equivalent) wearing out much faster than early hype/info/etc. So, it'd be interesting to get a real world idea if these things are likely to explode badly ;) 6 months after purchase or not (etc). :)

ksec 9y ago

I am rather hoping this make large capacity DRAM some price drop.

scurvy 9y ago
I'll settle for NAND pricing to drop. There's a bit of large-scale NAND shortage right now. Companies are bearing the brunt of the shortage right now, but they'll past it on to consumers if it continues much longer.