12 ms·
Is sequential IO dead in the era of the NVMe drive?
- sylware 3y agoHuh? PCIE is serial.
- antx 3y ago"Writes to the WAL are purely sequential and writes to long-term storage are purely sequential." Is that simply a typo or am I missing something? Shouldn't it be mostly random for long-term storage?
- jasonwatkinspdx 3y agoNo. In WAL based systems new writes are appended to the write ahead log, and the dirty database pages held in ram. Periodically, a checkpoint process writes out all those dirty pages to long term storage, then truncates the write ahead log. So both these writes can be sequential. This is one of the reasons databases have the performance they do vs just using lots of little files like a database naively.
- eatonphil 3y ago> So both these writes can be sequential. In the worst case though, the long-term storage can still have random writes though no (for btrees anyway)? I figure LSM tree writes can always be sequential, even during compaction. But btrees can only do solely sequential writes if the entire tree needs to be rewritten. And you don't want to always do that if only parts of the tree need to be updated?
- saltcured 3y agoI would take it as a simplification to really mean "sequential enough" when writing out the reorganized copy to the long-term storage. With all the different layers in modern storage, I think you can get asymptotically close to the sequential rate once you start hitting some of these other multi-block granularities, i.e. around the block size for flash erasure, encryption, redundancy coding, etc. These don't have to be that big, i.e. several megabytes.
- jasonwatkinspdx 3y agoThe dirty pages can be written out in physical address order no matter what indexing structure is used. What real databases do involves a bunch of complicated choices, but basically there's ways you can make checkpoints incremental such that it's nearly always large sequential writes.
- eatonphil 3y agoSure it can be written in order but it still can involve non-consecutive writes which I assumed is closer to random writes from the disk's perspective.
- cperciva 3y agoMoving from spinning rust to solid-state storage dramatically improved the performance of random reads, but random writes still carry a penalty compared to sequential writes. So sequential I/O is perhaps half-dead; there's no need to optimize layouts for sequential reads any more.
- GauntletWizard 3y agoThere's still a ton of performance benefit that can be gained by keeping your read pipeline full, and you can most easily keep your read pipeline full by making it guessable by doing readahead. It doesn't need to be strictly disk-layout sequential, file-layout sequential will usually do.
- Proven 3y agoFar from it, sequential IO optimization isn't dead. Sequential reads need to be optimized to produce optimal performance for selected workload. That usually means applying a lower level of inline compression. In some cases deduplication works better before, in other after, compression. Sometimes post-process dedupe is more suitable than inline. Then there's erasure coding and data protection methods that are still being optimized for NVMe with sequential workloads, including random workloads which are being sequentialized to work better with latest flash media. I would even say developments in sequential IO are becoming more important than random IO.
- jasonwatkinspdx 3y agoYes. And the Flash Translation Layer inside modern drives are quite sophisticated. This means you could start out with a write pattern that's causing a lot of copying and reorganization of pages in the back ground, typically because it's doing a lot of unaligned random writes smaller than the erase block size, and initially the performance will be fine. But as the drive wears in, and the capacity fills up, the FTL has less extra space to work with, forcing this garbage collection activity to become increasingly common. So you get performance degradation and premature wearing of the drive in a way that's opaque.
- bcrl 3y ago
- revelio 3y agoAmazing how much faster f2fs is than btrfs. Now I have some buyer's remorse over using btrfs...
- ilyt 3y agoIt was always slow-ish compared to traditional ones, same with ZFS in some workloads
- thewataccount 3y agoIt's because of the COW I think right?
- ilyt 3y agoYeah, and in general nice features cost IOPS. Not that it matters all that much on modern hardware, unless you're really trying to get all there is out of it
- thewataccount 3y agoNot sure about ZFS, but going off a graph in the link - BTRFS is like ~3x~4x slower then the other FS's which IMO that's really substantial. Of course it's workload specific but that type of a performance hit is going to effect a lot of workloads.
- rektide 3y agoWhat are you citing or talking about? Neither f2fs nor btrfs show up in the article.
- kardos 3y agoSee the reproduced Figure 9
- infamouscow 3y agoIf you like f2fs, you might also be excited about ssdfs: https://news.ycombinator.com/item?id=34939248 https://news.ycombinator.com/item?id=34939248
- deleted 3y ago[deleted]
- xmonkee 3y agoWhat a great and clear write-up. Thanks to the author.
- hawk_ 3y ago> There are different techniques for implementing OP (Over provisioning) yourself. You can simply leave a portion of the drive unpartitioned This doesn't sound right. Does this mean that the drive decides to arbitrarily move blocks between your partition and unpartioned spaces to manage GC?
- deleted 3y ago[deleted]
- Dalewyn 3y agoUnlike hard drives of yore where ordering the drive to write a bit at a given location results in the drive writing a bit at a given location, SSDs abstract it away as a matter of their design. An SSD will still provide a virtual bitmap so file systems can still order it to write a bit at a given location like ye olde days, but the SSD controller ultimately decides where to actually write that bit which more than likely has no relation to the specified location. Incidentally, yes: This also means defragmenting an SSD is not only harmful, it is outright misleading because the state of the file system is not related to the state of the bits on the drive.
- rasz 3y agoMechanical HDDs have same mechanisms available, its just used for defects https://www.dataclinic.co.uk/hard-drive-defects-table/ https://www.dataclinic.co.uk/hard-drive-defects-table/
- Dalewyn 3y agoIndeed! HDDs remap sectors to reserve sectors if their controller detects the original sectors have become unwritable. The result is quite similar to what SSDs do more broadly; the file system remains as-is, but the workings behind the scene differ from it.
- issafram 3y agoI'm still of the opinion that even with SSD/NVMe, the random reads/writes are still very slow.
- marginalia_nu 3y agoWith SSDs it's more like a discount on sequential/adjacent reads within a physical block that is of a size that varies with model and manufacturer.
- exabrial 3y agoIn my experience, depends on the SSD. Consumer grade ones appear really fast because they have a DRAM cache, and as soon as you start having a bunch of cache-misses you realize how slow they actually are. We got a bunch of enterprise grade Samsung SSDs and there'a large difference in sustained I/O. It's not "instant" by any means, but it there are other things slowing I/O.
- zamadatix 3y agoOnly the very best drives hit >200 MB/s in 4k write (even with DRAM as cache) and for 4k mixed not even Optane drives get that high.
- wtallis 3y agoDid you forget to stipulate that you're only talking about workloads that issue IO requests serially with zero parallelism?
- zamadatix 3y agoNo, I just didn't pick QD32T8 to pump the numbers either. Even in such workloads the gap to sequential speed is a decent ratio, just not painfully so as in most other mid to singular workloads.
- willis936 3y agoThat's why all of my minecraft servers are on RAM drives.
- layer8 3y agoTldr: No – “the benefits of sequential IO are alive and well, even in this new era of NAND flash.” (Betteridge's law of headlines strikes again.)
- rektide 3y agoIt's weird to me that there's so many promising reliefs that seem near at hand, but which have simply never been delivered. Zoned Storage has existed for a while, for Shingled Magnetic Recording & tape, and there was much hubbub in 2018-2020 about NVMe getting Zoned Namespaces, which happened. https://zonedstorage.io/ https://zonedstorage.io/ And there's now a bunch of fs implementations. f2fs and btfs both have some support. But there's still no products actually available to buy. We could be getting so much better at NVMe, so systematically making things better. But we've kind of been stalled out for a while, after a bunch of ceremony figuring out what we wanted to do.
- bcrl 3y agoSMR and zoned storage offer compete and total utter crap performance in the real world. They have all the glass jaws of early flash firmware (garbage collection that took seconds to complete during which time no additional I/Os completed) while running on a medium that has many orders of magnitude higher latency than flash. Take 2008 era USB flash drives and that's about what you'll get out of an SMR HDD when using a write workload that isn't 100% large sequential writes. SMR might have become more relevant if it had provided a useful increase in density, but that never materialized as was originally promised. Retail prices for SMR drives were never much better than CMR. The largest generally available HDDs are available in CMR flavours because that's what's needed in real world servers. There's no 10x density improvement (or even 2x). Meanwhile flash is marching relentlessly down the cost reduction path afford by Moore's law + 3D layer stacking. Really fast 1TB NVMe SSDs are under $100 now, and it doesn't look like flash cost reduction is going to slow down any time soon.
- rektide 3y agoI wasn't really talking about SMR though, just some protocols they had that are now being re-used. I'm talking about Zoned Namespace SSDs (ZNS). Let's go back in time almost a decade. A bunch of smart people had figured out that the cumbersome Flash Translation Layer (FTL) on SSDs made it really hard to get expectable & consistent performance. They were building possible specs to try to directly access each flash block, to be able to fill them up as they saw fit & clear them out as they saw fit. They wanted direct access, with far less work juggling complex data-mappings done on the SSD itself. Two examples of Open Channel SSD works: https://openchannelssd.readthedocs.io/en/latest/ https://openchannelssd.readthedocs.io/en/latest/ http://lightnvm.io/ http://lightnvm.io/ Note the huge banner on the second: "Zoned Namespace (ZNS) SSDs have replaced the work on OCSSDs, and is now a standardized interface." Everyone realized that the protocols we had for Zoned Storage were basically adequate to get us what we wanted. We'd just make a lot of 128kB or whatever sized zones on the SSD, and let people manage them themselves. Currently this means opening blocks & then appending to them. There's been outstanding hope, in the future, that we might go further and allow more random write patterns, but still with the same contract of no over-writes (just clears). That work seemed like it was ready to go in 2020 & 2021. SSDs were supposedly sampling/becoming available. https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-sampling/ https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-s... https://semiconductor.samsung.com/newsroom/news/samsung-introduces-its-first-zns-ssd-with-maximized-user-capacity-and-enhanced-lifespan/ https://semiconductor.samsung.com/newsroom/news/samsung-intr... And that was reaffirmed again a year latter. https://www.tomshardware.com/news/samsung-and-western-digital-team-up-for-zns-ssds https://www.tomshardware.com/news/samsung-and-western-digita... But here we are in 2023 & there's still no Zoned Namespace SSDs (ZNS) one can purchase.
- COGlory 3y agoNot to harp too much on the title (I read the article and found it very interesting), but are there any industries storing their data on all-flash? My intuition is that most institutions still have a giant array of spinning rust somewhere underpinning everything they do.
- birdman3131 3y agoOutside of backups I would expect many business to be all SSD. Maybe not the majority but for many use cases I see no good reason to go HDD for most small businesses. I know I went pretty much all SSD ages ago. I do run backups to a few 14TB externals but nothing outside of that is spinning rust.
- girvo 3y agoI ditched my final spinning rust (a WD Black 2.5” 7200RPM drive I had jammed into my home server) last night actually. It’s all SSDs in my server, and it’s all NVMe in my gaming desktop. My Macs and my partner iPad are all flash too obviously.
- deleted 3y ago[deleted]
- kccqzy 3y agoI do not think it is feasible to store all of YouTube videos on SSD. Google must be using spinning rust for the long tail of videos.
- rch 3y agoI see a fair number of all SSD clusters, but most have a mix of drive types. Interestingly, the ratio of drive types is aligned with optimal performance rather than cost.
- loeg 3y agoGobs. This is Pure Storage's entire market.
- tatersolid 3y ago
- loeg 3y agoEven setting aside internal GC, there is read-side benefit to having fewer contiguous sections: your large read request needs to be split into fewer IOs. This matters if you're reading the output of this workload. https://dl.acm.org/doi/abs/10.1145/3477132.3483593 https://dl.acm.org/doi/abs/10.1145/3477132.3483593 discusses it in a sciencey way.
- hot_gril 3y agoNo, because flash memory (and so-called RAM) still do sequential reads and writes faster, but for sure I'll tune a default Postgres config to give less of a penalty to random reads.
- numpad0 3y agoCompletely tangential, but Yangtze Memory/YMTC 232-layer NAND flash that was supposed to be used in iPhone 14 is on fire sale due to US import restrictions, to the point that some speculates that YMTC will be gone soon. The point is, DRAM-less 2TB M.2 NVMe disks are at $80 right now in select non-US markets. This is mere ~5x HDD, and slow as 4Gbps on writes at worst conditions.
- zamadatix 3y agoDRAM-less 2 TB m.2 NVMe drives are <$80 in the US market as well https://www.amazon.com/dp/B08CDM2HSS https://www.amazon.com/dp/B08CDM2HSS. There are a couple such drives in this price range, even from companies like Intel with their 670p.
- rektide 3y agoMinor detail, Intel sold their SSD business to SK Hynix for $7b. SK Hynix created a new company Solidigm out of the acquisition. So it is now a Solidigm 670p. https://www.solidigm.com/products/client/d6/670p.html https://www.solidigm.com/products/client/d6/670p.html Personally it really felt a bit rotten to me that Intel had created drives like the 670p that would simply stop working after a designated amount of writes. On the one hand, predictable life-span is kind of of a good thing, but sending drives to the landfill before they're actually done seems monstrous. Anyways though, yeah, DRAM-less drives are available for incredible prices.
- jscipione 3y agoCOW file systems are pretty amazing, especially on flash memory-based storage.
- gfody 3y agoyou'd need something byte addressable for access pattern not to matter.
- deleted 3y ago[deleted]
- jrockway 3y agoI always worry that hardware "locks in" software. SSDs showed up after everyone wrote their databases to be optimal on spinning rust, so SSDs were built to look at the flow of requests, assume that the application was written for spinning rust, and optimize the data layout accordingly. Whether or not this is as good as brand-new software that managed the layout itself is debatable, and nobody will write that software anyway because they won't have users that have SSDs that let the database manage the disk in full. So the software has to tune it for one manufacturer's reorganization algorithm and hope for the best. The path to a "global maximum" remains unclear because of this. CPUs are similar. People started writing programs in C, so CPUs started optimizing for the outputs of C compilers. If someone were to invent CPUs and compilers today, the list of optimizations would probably be different, and the performance characteristics of real-world software would probably be different. (It's not just C; people wanted virtual address spaces, and that was slow in software, so now there is a TLB, etc.) Actually, it goes even deeper. Half the startups I see on HN related to software operations have a quickstart like "don't worry, you don't have to rewrite your code, we'll magically do everything for you". Why not just refactor the code? It would take about an hour, and then you don't need a crazy Rube Goldberg machine to achieve your desired results. (If you want specifics, think service meshes, and how they now detect what language your app is written in so they can rewrite your HTTP handling functions to pass around a trace ID. Back in the day, you just set outgoing.Headers["x-trace-id"] to incoming.Headers["x-trace-id"]. Now we have layers and layers of OS-level machinery that still don't do as good a job as spending half a day adjusting your codebase. Billions of dollars invested in saving half a developer day! Wow!)
- JoeAltmaier 3y agoDeeper than that! Such fundamentals as 'the stack is accessible by code' goes back to Fortran. C doesn't need that. The call stack and the 'display' stack could be different things and no C programmer need care. This is the root of attacks that rewrite the return address of a kernel call. If the return address were managed/protected outside normal data operation this would not exist.
- pklausler 3y ago
- jonstewart 3y agoThe interesting question —- I’d appreciate any links —- is how to change our programming patterns to get the most performance out of NVMe SSDs. I’ve seen before that it takes a lot of parallel requests to saturate a fast NVMe drive —- but how to change our application architectures to generate such a workload?
- hakunin 3y agoWe had a use case at one of former jobs where we had to lookup entries in hundreds of megs of static key/value data (vendored with the app), but it didn't have to be super fast and we didn't want to waste any RAM on it. Since everything is on SSDs, I wrote a lib[0] that was just seeking sorted data in files with pread and binary search. Worked perfectly for our needs. [0]: https://github.com/maxim/wordmap https://github.com/maxim/wordmap
- bullen 3y agoSeems SSDs are reaching the end game with ~8TB on a 2.5" platter. Now you just need to nail the peak of $/GB, I'm looking at both 870 EVO / QVO, they have 4TB versions for $300. For "write once" or append only data these could work well, think user registry or sequential log. But you need to offload the active data on SLC drives or take the risk of the wearing driver firmware in the exact devices you buy/patch. The OS needs to be on SLC anyhow.
- _joel 3y agoDefinitely not, random access on PCIe5 is not too different (from user experience) from PCIe4 and not that stellar compared to sequential throughput of various block sizes.
- jillesvangurp 3y agoI think append only type logs like Kafka are more important from a semantic point of view than from an IO point of view. The importance of having an immutable, append only data store is that it is immutable. That gives you some nice properties with consistency in especially distributed setups and synchronizing state between different nodes. The IO write speed is a bit of a secondary concern. And since the data is immutable, you don't actually overwrite it all the time. And of course, SSDs have been around for quite long and spinning disk was already on the way out over a decade ago on high end servers. So, things like Kafka and event sourcing mostly became popular after that started happening; not before. This was never really about SSDs vs. spinning disks. Instead it is about the guarantees that come with having immutable data in a distributed setup. You see that in the Elasticsearch world as well (lucene uses append only data structures). Elasticsearch emerged around 2010. Almost from the beginning the consensus was that ssd was vastly preferable to spinning disks for scaling. Not because of the writes but for reads. Of course lots of people still used spinning disk around the time as well. So, sequential reads and writes are two things. Elasticsearch writes sequentially; but reading is random access. Which is why SSDs are nice to have for Elasticsearch clusters even though it uses append only storage.
- yuhong 3y ago[dead]
- yxhuvud 3y agoSequential IO is not dead for the simple reason that it is the simplest way of solving many problems. Often it will be good enough.
- randunel 3y agoFor some reason, reading this title made me remember Betteridge's Law, so I first answerd "no" and then clicked to read the article. > Betteridge's law of headlines "Any headline that ends in a question mark can be answered by the word no." https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- rbanffy 3y agoI find it strange to try to make the application deal with this kind of issue. It's the OS that's responsible to deal with the physical aspects of IO, not the application. Now, as for a write-optimized log, it's important from a durability perspective that data is persisted to media as soon as possible after the OS orders a flush. In the storage hierarchy, even the increasingly rare battery-backed DRAM drive has its place.
- moconnor 3y agoThink of it like this: the universe doesn't care what we want, or what we find to be clean abstraction. When our programs are run on real, physical hardware they will be faced with the physical limits of those systems. If that means sequential writes are significantly faster than random writes then no amount of OS abstraction will make an application that is structured to require random writes as fast as one that uses sequential ones. If you don't care about performance, then the abstraction layers we have let you ignore this level of detail already.
- rbanffy 3y ago> If you don't care about performance, then the abstraction layers we have let you ignore this level of detail already. I really care more about correctness, portability, and long-term maintainability. Optimizing for the idiosyncrasies of a given OS at a given point in time go against them all. Performance can often be increased by adding hardware (or waiting for the next Moore iteration). There are a few cases where performance is critical, in the sense that, if you don't achieve it, the system is not fit for purpose, but they are vanishingly rare.
- 1827163 3y agoCan't you just map the flash into the the host's PCIe memory space, at least for reading, then you can have random access to it? Though some speed penalty if accessing different rows from the flash rapidly in succession. You can just treat it as memory, and be able to mmap() it from userspace? The SSD controller will handle all the bad block remapping, ECC, etc. transparently in that case. It should be possible to make a PCIe x16 card for this, so the CPU will not be tied up as much waiting for the flash.
- thway15269037 3y agoEven on most overpriced and insane modern nvme consumer SSDs you'd see numbers like that: Seq Writes 7000 MiB/s; Rand 4k Writes: 44 MiB/s. And the last number haven't really improved over the years. Yeah, it's better compared to (1 MiB/s writes;8 MiB/s reads) on HDDs, but it's not that good.
- warrenm 3y agoif you were going to design a filesystem for solid-state storage (instead of linear or rotational media), could you implement wear-leveling by determining where to place the next segment of a file by hashing it, and using the hash as its address, updating the linked-list/tree/table/whathaveyounot that tracks where files exist?