7 ms·
Marvell Announces First PCIe 5.0 NVMe SSD Controllers: Up to 14 GB/S
- PinkPigeon 5y agoI mean, I love the insane GB/sec figures, but does anyone else mostly care about IOPS? These state 1.8M read and 1M write, which sounds quite impressive.
- CoastalCoder 5y agoRelated: anyone know of a good video or diagram for helping CS students get an intuition regarding the interplay of bandwidth and latency? Including how saturating bandwidth increases latency by causing queueing bottlenecks? I'm looking for something a bit more visual and dynamic than the old "station wagon full of tapes going down the highway" imagery. [EDIT: Just for clarification, I feel like I already have a pretty good grasp on these concepts. I'm looking for good ways to help others at the ~ undergrad level.]
- wtallis 5y agoI do some rather coarse measurements of random read throughput vs latency as part of my SSD reviews. See eg. the bottom of https://www.anandtech.com/show/16636/the-inland-performance-plus-2tb-ssd-review-phisons-e18-nvme-controller-tested/3 https://www.anandtech.com/show/16636/the-inland-performance-... Those graphs cut off the essentially vertical latency spike that results from enqueuing requests faster than the sustained rate at which the drive can serve them. For a different view in terms of queue depth rather than throughput, there are some relevant graphs from an older review that predates io_uring: https://www.anandtech.com/show/11930/intel-optane-ssd-dc-p4800x-750gb-handson-review/5 https://www.anandtech.com/show/11930/intel-optane-ssd-dc-p48... Generally speaking, latency starts increasing long before you've reached a drive's throughput limit. Some of this is inevitable, because you have a relatively small number of channels (eg. 8) and dies to access in parallel. Once you're up to the throughput range where you have dozens of requests in flight at a time, you'll have constant collisions where multiple requests want to read from the same plane/die/channel at once, and some of those requests have to be delayed. But that's mostly about contention and link utilization between the SSD controller and the NAND flash itself. The PCIe link is pretty good about handling transactions with consistently low latency even when on average it's mostly busy.
- bombcar 5y agoHere's a post on relative latencies that may be useful: https://danluu.com/infinite-disk/ https://danluu.com/infinite-disk/ There was another post I saw recently comparing the increase in disk SIZE over the last 30 years vs the increase in disk SPEED vs LATENCY (so size in GB, speed in GB/s, latency in IOPS) - and how size increases far outstripped speed which outstripped latency, though all had improved. Found it! The key is IOPS/GB as a metric. https://brooker.co.za/blog/2021/03/25/latency-bandwidth.html https://brooker.co.za/blog/2021/03/25/latency-bandwidth.html
- uyt 5y agoAre you referring to Little's Law?
- louwrentius 5y agoMaybe this doesn’t answer your question exactly but I addressed this topic in two blogpost, maybe it helps. https://louwrentius.com/understanding-storage-performance-iops-and-latency.html https://louwrentius.com/understanding-storage-performance-io... https://louwrentius.com/understanding-iops-latency-and-storage-performance.html https://louwrentius.com/understanding-iops-latency-and-stora...
- MrFoof 5y ago>...but does anyone else mostly care about IOPS IOPS helps, but for the average user, hundreds of thousands is already functionally infinite. What matters at this point is latency. Where you really feel that is Queue Depth 1. Where you read a file that points you to other files, that point you to other files, etc. That is the exact case where the computer is still making you wait. This happens when you start your operating system, it starts services, you launch apps, etc. Driving that latency down is the biggest improvement you'll ever see past where we are today in terms of IOPS and throughput. This is where the latest Optane actually shines. Optane doesn't win on IOPS or throughput, but where it shines is its crazy latency (delivered at relatively low power levels). Where latencies are 10% of that of even the highest end PCIe 4.0 NVMe SSDs. Do something like launch 20 applications at once, and it'll be done in a fraction of a time compared to even something like a Samsung 980 Pro because of latency being more around 10 μs instead of 100 μs. PCIe 5.0 SSDs will cut latencies down to where Optane is today, but driving latency under 1 μs is where we'll get into a new level of crazy.
- Dylan16807 5y agoIsn't launching 20 applications at once the realm where flash competes the best?
- labawi 5y agoTechnically, it would comparatively excel at launching 64K+ applications at once, but yes - flash is not quite amazing at launching 20 applications serially.
- jiggawatts 5y agoI can't upvote this enough. Related: Notice how the public cloud marketing material tends to focus on scalability over other metrics? That's because scaling horizontally for them is easy: They just plop down more "stamps" -- a set of clusters and controllers that is their unit of scale. Need 1,000 more servers in US East? Plop down 10 more stamps of 100 servers. Easy! Except of course this involves an awful lot of networking, with long cable runs and many hops and virtualisation layers. The end result is that you can't get anywhere near the underlying storage latency. Azure's Premium SSD has "write flush" latencies of about 4 milliseconds according to my measurements, which is easily 100x slower than what my laptop can do with a now very outdated Samsung NVMe SSD. Notice that if you go to their marketing page, they talk about "low latency" and "lowest latency", but they have no numbers? Meanwhile the MB/s and IOPS is stated with numbers: https://azure.microsoft.com/en-us/pricing/details/managed-disks/ https://azure.microsoft.com/en-us/pricing/details/managed-di...
- NikolaeVarius 5y agoIs there a theoretical "practical" limit for how fast these things can physically get (in this particular form factor) My understanding is that NVMe interface is pretty much as close as you can get to the CPU without being integrated. Is there a world where these things can operate as fast as RAM?
- baybal2 5y agoFlash chips themselves can't operate as fast as RAM. You can get as much bandwidth as RAM, but you can not get it as physically fast as RAM i.e. faster than 200ns
- anarazel 5y agoWhy? DMA can do transfers to/from CPU caches.
- baybal2 5y agoNAMD flash cells themselves can't charge/discharge faster
- m4rtink 5y agoWhat about those non volatile RAM technologies (IIRC called 3D XPoint) Intel is using for their Optane stuff ? It seems to be kinda in between RAM and flash spec wise.
- api 5y agoThat's getting to be RAM speeds, but of course not RAM latencies.
- dvfjsdhgfv 5y agoIt looks like we'll get fantastic speeds in a few years. Now it's time to take care of durability.
- harveywi 5y agoMarvell? I thought that DC had all the Flash controller rights.
- deadcore 5y agolol that made me chuckle
- FBISurveillance 5y agoI see what you did there.
- WesolyKubeczek 5y ago1) is the actual NAND flash faster, or are we talking about going from "awesome" to "abysmal" as soon as you run out of DRAM/SLC caches or what have you? Which, given this kind of bandwidth, is going to be sooner rather than later. 2) this plus QLC cells, which, durability-wise, make TLC look good, makes me anticipate headlines like "This new PCIe 5.0 SSD ran out of its rated TBW in a week!"
- wtallis 5y agoThese controllers are for enterprise drives where SLC caching is almost unheard of and all the performance specs are for sustained performance on a full drive. But the best performance may only be attainable on a drive with higher capacity than you can afford.
- derefr 5y ago> Which, given this kind of bandwidth, is going to be sooner rather than later. I don't see why — like NAND up until now, it keeps up by adding more separately-addressable chips to the board and striping writes across them. A 512GB SSD with this chip would hit the wall pretty soon, but they wouldn't waste this controller on a 512GB SSD. They'd use it for 16TB+ SSDs.
- vbezhenar 5y ago> "This new PCIe 5.0 SSD ran out of its rated TBW in a week!" If this drive supports claimed 9 GB/s write speed, you can write 324 TB at 10 hours. Samsung QLC warranty for 1 TB drive is 360 TBW.
- wtallis 5y ago1 TB of QLC can't get anywhere close to 9 GB/s write speed. The best is currently about 40 MB/s for a 1 Tbit die, so 320 MB/s for 1 TByte. The slow write speed of QLC generally prevents you from burning out a drive in less than a few weeks.
- NullPrefix 5y agoChia is a godsend for storage prosumers. All you have to do is look at the speed and check if the warranty is voided by Chia.
- ksec 5y ago20x20mm Package. That is quite large. ~10W Controller. Doesn't matter on a Desktop or Server. But we sort of hit the limit what could be done on a Laptop. Having said that it doesn't mention what node it was fabbed on. I assume there are more energy efficiency could be squeezed out. <6us Latency. Doesn't say what percentile it is and under what sort of condition. But Marvell claim this is 30% better than previous Marvell SSD Controller. I think ServeTheHome article [1] is probably better. ( Cant change it now >< ) We also have PCI-E 6.0 [1] finalised by the end of this year which we cant expect to be in 2023/2024. SSD Controller with 28GB/s. I am also wondering if we are approaching the end of S-Cruve. [1] https://www.servethehome.com/marvell-bravera-sc5-offers-2m-iops-and-14gbps-in-a-pcie-gen5-ssd/ https://www.servethehome.com/marvell-bravera-sc5-offers-2m-i... [2] https://www.anandtech.com/show/16704/pci-express-60-status-update-draft-071-coming-soon-final-release-by-end-of-year https://www.anandtech.com/show/16704/pci-express-60-status-u...
- bhouston 5y ago2023/2024 I suspect you meant to write. I do like the fact that SSD bandwidth may be approaching memory bandwidth.
- ksec 5y ago>2023/2024 I suspect you meant to write. ROFL. Thanks. Keep thinking about the good old days.
- CyberDildonics 5y agoSSD bandwidth is not really approaching modern memory bandwidth - a computer with one of these will probably have 4 to 8 channels of DDR4 at 25GB/s each with DDR5 released about a year ago. That is 100GB/s to 200GB/s currently and more by the time PCIe 5 becomes a reality. I am sure if you go back a few years you can find systems that have the same or less memory bandwidth than this has now, but they have both been moving forward enough that SSDs are still not close. That being said, between bandwidth and random access times, swapping out to disk backed virtual memory is very pragmatic and isn't the death of performance it used to be.
- vmception 5y agoWhich of your applications or your clients applications running at the same time are currently bottlenecked?
- Strom 5y agoEven something as simple as grep is bottlenecked by disk speeds right now.
- PedroBatista 5y agoThat ~10W tho ... I think right now the real money is in somebody who can implement PCI-E 5 efficiently, or soon we'll see every SSD with a mini fan on it. ( It's not all PCI-E 5's fault, these controllers have been more and more power hungry )
- wtallis 5y agoYou can always wish for more efficiency, but it's important to understand that SSDs have actually been getting steadily more efficient in the Joules per Byte transferred sense. We're just seeing a simultaneous concentration and consolidation of performance that is moving a bit faster, hence the need for enterprise SSDs to abandon the M.2 form factor in favor of the somewhat larger and easier to cool EDSFF family. One way of looking at things is to realize that until now, a storage controller handling 14 GB/s of traffic would have been a full-height RAID card demanding 300 LFM of airflow, and now we're putting that much throughput into a SSD with a heatsink that's roughly 1U by 1in in cross section.
- ChuckMcM 5y agoPretty impressive. And if it can really do 4 GB/s[1] of random write performance is super helpful in database applications. I am wondering where the "Cloud SSD" stuff takes it relative to general purpose use. Does anyone have any insights on that? [1] They quote 1 million IOPS and assuming a 4K block size, which is a good compromise between storage efficiency and throughput, gives the 4GB/s number.
- wtallis 5y agoHere's a version of the OCP NVMe Cloud SSD Spec from last year: https://www.opencompute.org/documents/nvme-cloud-ssd-specification-v1-0a-pdf https://www.opencompute.org/documents/nvme-cloud-ssd-specifi... It covers a lot of ground, but as far as I'm aware nothing in there really makes drives less suitable for general server use. It just tightens up requirements that aren't addressed elsewhere.
- ChuckMcM 5y agoYeah, pretty much. Thanks I've added it to my specifications archive.
- dstaley 5y agoObviously there's always a customer for faster speeds, but have we even hit the upper threshold of PCIe 4 in the consumer market?
- wtallis 5y agoThe first wave of consumer gen4 SSDs that all used the Phison E16 controller were only good for about 5 GB/s out of the ~7 GB/s possible (and 3.5 GB/s on PCIe gen3). But the newer gen4 consumer drives that started hitting the market last fall come a lot closer, and this summer a lot of those are getting refreshed with newer, faster NAND that will have PCIe 4.0 as thoroughly saturated as PCIe 3.0 has been for the past few years. Phison has already clearly stated that their next high-end controller after the recently-launched E18 will be a PCIe gen5 chip, and the E18 is fast enough to finish out the gen4 era.
- dstaley 5y agoAbsolutely wild that PCIe 4.0 was saturated just two years after introduction in the consumer market.
- wmf 5y agoSN850 is getting close to the limit of PCIe 4.0. https://www.anandtech.com/show/16505/the-western-digital-wd-black-sn850-ssd-review https://www.anandtech.com/show/16505/the-western-digital-wd-...
- louwrentius 5y agoTry to imagine with all these developments how much performance you can get from vertical scaling. I bet that many applications don’t need to care about horizontal scaling with all the burdens involved before you outgrow the performance of a ‘single’ box (aka stack overflow)
- walrus01 5y agoGiven a theoretical either/or choice between a PCIE 5.0 SSD, or more PCI-E lanes using the tech we have now, I would rather have a greater number of PCI-E 4.0 lanes in single socket consumer/workstation grade motherboards. Leaving open the possibility for dual NVME SSD in a workstation along with a x16 video card, and 10Gbps NICs.
- bmcahren 5y agoWith the location of modern NVME slots, you can't even exceed 500MB/s without a controller rate limiting due to heat.
- Synaesthesia 5y agoStill not good enough for PS5 hey Sony?
- RachelF 5y ago"16 Physical Functions, 32 Virtual Functions" Does anyone know what this means in the specs?