19 ms·
What every programmer should know about solid-state drives (2014)
- metadat 4y agoWhat sorts of programmers should be concerned about these matters? Page cache doesn't seem too important or interesting in my day to day app and distributed systems development. Maybe it's useful if you want to make something like a more performant version of grep? (aka ripgrep?)
- Gordonjcp 4y agoBy the looks of the article? People writing SSD firmware, or SSD drivers. There is probably a small but non-zero number of these on here.
- jqcoffey 4y agoThe author appears to be an EM at Booking.com. It seems unlikely that anyone at Booking would be working on SSD firmware or drivers, but a CDN seems like a reasonable assumption and also a useful place to plumb the depths of SSD implementations.
- eschneider 4y agoPeople who read from disks and people who write to them. How SSDs organize data definitely had read and write performance implications and if you're writing to disk, some write habits that are perfectly reasonable on regular disks can cause catastrophically fast wear on SSDs.
- KennyBlanken 4y agoYes, but the number of people who need to be worried about aligning their writes and such is pretty small; certainly not "every" programmer. The author gets into the weeds about certain things application level programmers almost never need to know or concern themselves about. He really doesn't understand what's useful information and what isn't. If you're programming at enterprise scale, this sort of stuff is the responsibility of architect-level programmers and senior systems engineers. Even most linux sysadmins know all about block alignment (well, if they predate most of the various tools figuring out block size/alignment stuff for you.) It's nothing new - RAID arrays work best when properly aligned, for example.
- loxias 4y ago> Page cache doesn't seem too important or interesting in my day to day app and distributed systems development This is why we can't have nice things.
- the_only_law 4y agoI love how people say this, when the reality is, all the software from the oh-so-coveted is the biggest shit show I’ve seen. But it’s rarely because some developer didn’t understand page caches, and usually because it obviously didn’t revive enough QA or UX input.
- tshaddox 4y agoHow so? Isn't the only point of developing these systems and abstractions so that other people don't have to worry about them?
- chrisandchris 4y agoIMHO, today to many people think "don't have to worry about them" equals "don't need to know anything about it".
- tshaddox 4y agoI would argue that in most cases you "don't need to know anything about it" either. It's reasonable to deliberately treat abstractions as if they are not leaky, as long as you're aware that all abstractions in fact are leaky and you're equipped to investigate and learn about them if the leaks cause problems.
- dotopotoro 4y ago“don't need to know anything about it” is acceptable, but should not be encouraged. It’s not like reading 10 bullet points on the subject is “diving deep” and making huge time investment. It’s just getting the minimal context, so later on at least some keywords are known.
- yourapostasy 4y agoNot just programmers. Anyone using ZFS with SSD, whether as the pool itself or in various caches like slog(zil) is going to find this information of use when tuning for better SSD citizenship. Programmers treating SSD like faster spinning rust is like programmers treating S3 like another POSIX filesystem; you can do it, but you're trading away compounding future advantages for that one moment of expedience.
- dekhn 4y agoIn my career I have found that file system tuning for the devices an anti-pattern that almost always ends up causing more problems than it's worth.
- philjohn 4y agoAre you writing low-level software, such as filesystems, or raw block backed database storage engines? If not, then that's definitely a decent maxim to live by.
- golergka 4y agoDon't your distributed systems use databases of some sort?
- alpaca128 4y agoAnd why does a DB user need to know those details? Isn't it the whole point of DB systems to provide an optimized solution that allows users to focus on other things?
- dotopotoro 4y agoDatabases always try to flush something to disk after transaction, just in case unexpected reboot happens. So your writes to db have direct correlation to disk writes. Choice of db schema impacts physical layout on ssd. E.g. Different tables are more likely to be on different ssd pages resulting in random writes. Databases are insanely complex, but not magic.
- jeffbee 4y ago> doesn't seem too important or interesting in my day to day app and distributed systems development. Makes sense to me. At Google we were told to stop thinking about all this stuff, that the storage hardware and software people were responsible for hiding things like wearout from application developers. This article is really "things you should know if you plan to directly access an NVMe device" but there is a huge class of programmers who are better off not knowing.
- bombcar 4y agoThere was an article by varnish taking about how you should leave the caching and memory management to the OS - even if you can beat the virtual memory manager today you’ll stop improving your home grown solution while RAM and the kernel keep marching on.
- wmf 4y agohttps://varnish-cache.org/docs/trunk/phk/notes.html https://varnish-cache.org/docs/trunk/phk/notes.html
- rasz 4y ago>At Google we were told to stop thinking about all this and as a result Chrome slams SSD by writing cached Youtube videos to disk .... except Youtube never reuses cached video data (not even when rewinding more than couple minutes to already watched spot in same video), it explicitly generates hashed requests with custom URL parameters googlevideo.com/videoplayback?expire (~6hour shelf life) &range &sig &lsig. Heavy YT viewing results in wearing out your SSD by tens of gigabytes per day for no particular reason. This is just one small example of side effects from such brilliant decisions.
- MichaelZuo 4y agoYikes, is that really the case? No wonder the 128GB drive on my 2013 MacBook air wore out to 60% of its original performance...
- pavon 4y agoMy take: 1-13) General background info that informs the rest. 14-25) Important for any programmer that does enough file IO that they need to optimize it. 26-29) Important for any system admin to ensure they aren't inadvertently limiting the performance of their hardware.
- user3939382 4y agoWhat you should know is that I had an Apple OEM 1TB SSD in my late-2013 MBP and one day it failed so catastrophically under normal conditions that 2 of the best data recovery teams in the world told me there was nothing they could do. Backup your stuff
- samatman 4y agoI'll admit my memories of 2013 are hazy, but I do recall TRIM being an issue early in the Macbook's history†. Backup your stuff! I happen to also back up to an SSD these days, because the difference between minutes and hours is hard to argue with. †edit: history of shipping with an SSD standard, that is.
- lostlogin 4y ago> because the difference between minutes and hours is hard to argue with. If the backups are incremental it shouldn’t take hours.
- twofornone 4y agoSpeaking about backing up...if one were interested in long term archiving, do magnetic platters offer longer lasting data integrity than SSDs in cold storage?
- Melatonic 4y agoNot sure about that but I do know that the new sealed helium filled drives are much harder to take apart and do backup recovery on
- tenebrisalietum 4y ago> Splitting cold and hot data as much as possible into separate pages will make the job of the garbage collector easier. How do I tell my SSD to write stuff to specific pages? You can't really tell the SSD to do anything except read, write, or trim LBAs. Does NVMe support this with its queues? > 27. Over-provisioning is useful for wear leveling and performance I thought most if not all SSDs were already overprovisioned. Does additional overprovisioning help? > To ensure that logical writes are truly aligned to the physical memory, you must align the partition to the NAND-flash page size of the drive. I think this is false. This assumes there is a one-to-one mapping of LBA to SSD PBA which you don't know. LBA 2048 could go to any PBA on any page/block/flash line in the unit and as things are written and rewritten, any correspondence that might happen due to sequential assignment of PBAs->LBAs would gradually diminish, IF you knew for sure that was happening in the first place. Because you wouldn't really know what the SSD is doing without reverse engineering or seeing the source code of firmware, unless there's things going on in NVMe land that are new and I don't yet know.
- thfuran 4y ago>I thought most if not all SSDs were already overprovisioned. Does additional overprovisioning help? I think a big extra helping of overprovisioning is one of the major differences between consumer and enterprise SSDs.
- wtallis 4y agoI wrote a series of articles that covered the new features defined for NVMe drives. The general pattern is that there are now lots of optional hints that drives and host systems can exchange about data placement, alignment and lifetime. But there are also alternative paradigms available like Zoned Storage that break compatibility to offer explicit control. These features are mostly only implemented in enterprise SSDs, and often only if a big customer specifically asks for them. https://www.anandtech.com/show/11436/nvme-13-specification-published-new-features https://www.anandtech.com/show/11436/nvme-13-specification-p... https://www.anandtech.com/show/14543/nvme-14-specification-published https://www.anandtech.com/show/14543/nvme-14-specification-p... https://www.anandtech.com/show/16702/nvme-20-specification-released https://www.anandtech.com/show/16702/nvme-20-specification-r... https://www.anandtech.com/show/15959/nvme-zoned-namespaces-explained https://www.anandtech.com/show/15959/nvme-zoned-namespaces-e...
- tester756 4y agoare speeds of bleeding edge mem devices getting close to RAM?
- Scene_Cast2 4y agoIn terms of bandwidth or latency? All conditions, worst case, best case?
- cogman10 4y agoNot really. Max throughput is around 6gbps with a fairly high latency. DDR5 has speeds of 52gbps, lower latency, AND your CPU will almost undoubtedly have a cache on it to increase that speed further. This is all assuming you are putting your mem device on a pci-express bus.
- KennyBlanken 4y ago> Max throughput is around 6gbps with a fairly high latency. In the consumer market, a number of performance NVMe drives will hit over 5GB/sec, which would be 40 Gbps. The latency isn't anywhere near as good as even quite-old RAM, but modern SSDs are considerably less than an order magnitude off in transfer speed from even current, common ram (DDR4) and "only" about a hundred times higher in latency than RAM. That's pretty stunning from mass storage. So is well over 500,000 IOPS.
- fomine3 4y agoGP uses wrong unit for both, both GiB/s.
- MichaelZuo 4y agoYour forgetting Optane DIMMs for enterprise.
- bob1029 4y agoI've been thinking about the possibility of "dumb" SSD devices. All of the current HW-level performance hacks could actually get in the way if your software already enforces things like single writer, chunky writes and/or append-only log structures. Give me a drive that only writes in 1 linear direction (until its full) and has a big red button to clean the entire thing all at once (which would clearly require some offline processing time & multiple disks for a realistic system).
- bruce343434 4y agoSure! Go ahead and order some memory cells.
- jerdfelt 4y agoDoes the ZNS (Zoned Namespaces) spec come close enough? https://nvmexpress.org/new-nvmetm-specification-defines-zoned-namespaces-zns-as-go-to-industry-technology/ https://nvmexpress.org/new-nvmetm-specification-defines-zone...
- bob1029 4y agoYes, actually. This looks like a realistic/practical path. Had no idea this was a thing.
- mbjorling 4y agoThere is more technical information at zonedstorage.io which also offers drives for academia and open-source projects. https://zonedstorage.io/docs/community/devices https://zonedstorage.io/docs/community/devices
- rsaxvc 4y agoHave you seen SMR spinning disks? You can get them today in host-managed flavors.
- MichaelZuo 4y agoI think that's roughly what the flash storage modules on Apple's new Mac Studio are.
- NovemberWhiskey 4y agoI feel like maybe this is "what all filesystem developers should now about solid-state drivers"; not very obvious how most other developers would interact with a device at the level of abstraction where they have the kind of necessary control.
- B1FF_PSUVM 4y agoThis guy used to hammer a good point about databases: "In a time of SSD, multi-core/processor, two terabyte memory and Optane App Direct Mode machines, there is no reason not to build from BCNF data. Time to do what Dr. Codd demonstrated. Technology has finally caught up with the maths." https://drcoddwasright.blogspot.com https://drcoddwasright.blogspot.com (skip the distractions)
- mhh__ 4y agoIs anyone aware of a book-length equivalent of this?
- pkaye 4y agoIf you want to know about some internals of SSD, the only book I know is "Inside Solid State Drives (SSDs)" by G. Wong. Its an old book though.
- the_only_law 4y agoDoubt it, books on niche technical subjects don’t seem to be much of a thing anymore unless you’re willing to pay extortionist prices for university textbooks.
- mhh__ 4y agoThere is a book on DRAM, caches and hard drives by Bruce Jacobs. Basically I want what every programmer should know about storage but in the style of dreppers original article.
- wolverine876 4y ago> books on niche technical subjects don’t seem to be much of a thing anymore Why not? Blog posts aren't nearly as valuable.
- SketchySeaBeast 4y agoI assume someone would be writing that book in the hope they'd make money back and that's hard to do with a super niche subject few will be interested in and even fewer would be willing to pay for.
- wolverine876 4y agoHow does that differ from 10 years ago?
- 4y ago
- dekhn 4y agoI treat ssds like faster hard drives and I have never been disappointed.
- nonrandomstring 4y agoFrom a low level programmatic standpoint, managing size and alignment with (potentially unknown) page sizes throws the same challenges as for AV buffers and network packet MTU/sizes - either side of "just right" is suboptimal.
- deleted 4y ago[deleted]
- dang 4y agoRelated: What every programmer should know about solid-state drives - https://news.ycombinator.com/item?id=9049630 https://news.ycombinator.com/item?id=9049630 - Feb 2015 (31 comments)
- dtgriscom 4y agoA question: do you have a tool that searches the history for previous links, or do you just have a really good memory?
- mmmpetrichor 4y agoIf some typical write pattern from a typical app is wearing out the SSD really fast, I'd say that's the SSD firmware engineer's problem? And I think they've actually done a great job in general, judging by the typical lifespan of SSDs and the typically great performance. I'd argue that if the drive is designed correctly, most programmers shouldn't have to care about low level details. (I did say MOST).
- Sakos 4y agoI think you misspelled "it's the user's problem". I don't think most companies care until it becomes something that materially affects them. Until then, users are reliant on the developers of the applications they use to make up for the deficiencies in lower layers.
- lazide 4y agoA reputation for drives that fail faster than their competitors will definitely effect them!
- melissalobos 4y ago> A reputation for drives that fail faster than their competitors How can they get that if they stuff enough fake reviews, plus the legion of consumers who would have no idea that the drive was the issue and not "viruses".
- lazide 4y agoNeither of those apply to enterprise customers.
- dotancohen 4y agoThat's a completely different market, to which are marketed different drives.
- jbverschoor 4y ago> Cells are grouped into a grid, called a block, and blocks are grouped into planes. The smallest unit through which a block can be read or written is a page. Pages cannot be erased individually, only whole blocks can be erased. The size of a NAND-flash page size can vary, and most drive have pages of size 2 KB, 4 KB, 8 KB or 16 KB. Most SSDs have blocks of 128 or 256 pages, which means that the size of a block can vary between 256 KB and 4 MB. For example, the Samsung SSD 840 EVO has blocks of size 2048 KB, and each block contains 256 pages of 8 KB each. Very confusing and might be incorrect. What are planes. And are pages made out of blocks or vice-versa? If blocks are grouped in pages, with erasing it sounds very different.. Only whole blocks, which sounds like blocks are bigger than pages.
- jasonwatkinspdx 4y agoIt's correct. Planes reflect the physical structure of the storage chips: there's multiple layers that share a common vertical bus. Plane > Block > Page, that is to say Blocks are always made up of multiple pages (commonly 128 or 256 as the quote mentions). Pages are the unit of read and write, while blocks are the unit of erasure. The FTL tries to hide this page write vs block erase mismatch as best it can, but as the original article points out you may need to be aware of what it's doing in very high performance systems.
- wtallis 4y agoA single NAND die is only divided into two or four planes. It's a function of how many copies of the peripheral circuitry for accessing the array are included, not how many layers are in the 3D NAND array. More planes means the die can do more things in parallel (subject to constraints). A drive with 8 dies each having 512Gbit capacity divided into four planes per die will perform almost as well as one with 16 dies of 256Gbit divided into two planes, other things being equal (eg. number and speed of the channels between the SSD controller and the NAND, page and block sizes and access times, all of which are subject to change at the same time a generational change increases die capacity and number of planes).
- jbverschoor 4y ago
- wly_cdgr 4y agoHow relevant is this in 2022? What's changed and what still applies?
- wolverine876 4y agoA serious question: What has changed?
- rasz 4y agoGeometry got smaller, thus wear endurance got a LOT worse.
- fomine3 4y agoNo, 3D NAND helped a lot for durability.
- MichaelZuo 4y ago3D QLC NAND, which is what all cheap consumer SSDs are transitioning to, is pretty bad, like 1/10th the durability of common 3D TLC NAND from 3 years ago. And 1/100th the durability of even non 3d MLC NAND from 2014. Enterprise class 3D TLC NAND is relatively close to enterprise class non 3D MLC NAND, the gap is bigger for consumer drives. But I think as of 2022 only Apple still sells consumer desktops/laptops with entirely TLC NAND. Everyone else is racing to the bottom for their consumer stuff.
- amelius 4y agoFrom Wikipedia: > In December 2012, Taiwanese engineers from Macronix revealed their intention to announce at the 2012 IEEE International Electron Devices Meeting that they had figured out how to improve NAND flash storage read/write cycles from 10,000 to 100 million cycles using a "self-healing" process that used a flash chip with "onboard heaters that could anneal small groups of memory cells." So can I apply this myself by placing an SSD drive in an oven?
- rasz 4y agoYes, if you have manufacturer software to factory format blank drives. Heating up heals cells being written (probably filled as writing empties cells while erasing stores max charge value), but also speeds up data degradation in all the other cells not being written to.
- thehappypm 4y agoPersonally I feel like files are an abstraction that are too low-level for your typical new programmer. I find it odd that a typical script use case that you learn in Python 101 is reading a bunch of junk from a file and then write into another file. Files are finicky and we have much better abstractions than them, like databases.
- nine_k 4y agoLook at the file system as at a key-value store, only tree-shaped.
- thehappypm 4y agoA key value store has simple CRUD primitives, does a file system?
- SoftTalker 4y agoI'd say pretty much yes. If the file name is the "key" then most languages have straightforward ways to create and delete files, read data from them, and write data to them.
- thehappypm 4y agoNot really.. try editing a giant file or reading a file with an odd encoding
- yopawngungmstyl 4y agoSince code is typically stored in files, I hope that new programmers would be expected to learn what a file is.
- fuzzfactor 4y agoWell, there's still flashbench: https://github.com/bradfa/flashbench https://github.com/bradfa/flashbench Plus, alternatively, there's FlashBench: https://github.com/JonghyeokPark/FlashBench https://github.com/JonghyeokPark/FlashBench These might be found useful for determining the underlying structure.