7 ms·
Unbalanced reads from SSDs in software RAID mirrors in Linux
- wolf550e 9y agoAFAIK, there is no need to do "wear leveling" for reads, SSDs have "unlimited" life cycle reads. So the first SSD seeing ten times the reads will not make it fail sooner.
- rcthompson 9y agoEven if reads did wear out an SSD, wouldn't it be better to wear them out one at a time rather then wear them all out equally so they all fail in rapid succession once they reach the end of their life cycles?
- gist 9y ago> wouldn't it be better to wear them out one at a time rather then wear them all out equally Agree and this would seem to make sense. It is like having two rolls of paper towels in the kitchen. If you use both of them you are only extending the time till empty. If you use one of them only you can then replace that when it runs out running on the 2nd in the time it takes to replace.
- baruch 9y agoOn one hand this is true, on the other, if you have two devices that can work for 5 years and then you anyway replace them because they are now too small and never failed you have one device that you replaced and the other hardly in use and you have an increase TCO since you had to buy another device in the middle of the life cycle. Not that it is really critical, read-disturb exists but it is on a far lower scale compared to erase endurance.
- xorgar831 9y agoI would say no, you'd want the whole disk to fail then so you can just replace it vs. it getting smaller over time, which would make capacity planning pretty complex.
- rcthompson 9y agoWe're talking about RAID1 here, so multiple SSDs. I'm saying you don't want multiple disks to fail in rapid succession, which is what you'd maybe get if you exactly equalized the wear on them.
- xorgar831 9y agoThat makes sense, and you actually do see RAIDs fail like that in the real world sometimes even on non-SSDs.
- pkaye 9y agoIf the SSD has mechanisms to protect against read disturb, they should also factor that into the life remaining indicator to plan for replacement.
- wtallis 9y agoThere's no need. If a page has been read enough times to produce a read disturb, the remedy is to re-write the page, which moves the write endurance indicator (by the smallest possible increment).
- microcolonel 9y agoThis is not necessarily about NAND wear leveling, it could be about load balancing. SSDs have limited ability to service requests, and they dissipate heat when they do so. At high load, one of the disks is going to be hotter for longer, and wear out faster as a result. Then again, not sure if author was really thinking about these sorts of things, who knows.
- kabdib 9y agoNot actually true; SSDs experience "read disturbance" caused by application of read voltages. Essentially, the cells being read, and those nearby, experience a little wear on each read operation. It's a lot less wear than a write, but it's still there. When I last looked at this, devices could suffer (say) 3K writes, and once written a cell would have to be re-written after 3K reads in order to maintain proper voltage levels.
- ricardobeat 9y agoHow do you keep a counter for each cell? Who keeps a counter for the cell that stores the counter?
- gruez 9y agousually a more durable form of memory (SLC as opposed to TLC or MLC)
- baruch 9y agoSome of the meta data is only kept in ram and updated from time to time to special regions. These special regions can also suffer from endurance issues but are usually are SLC-use-MLC so they wear slower but it is a known failure mode for the metadata locations to wear and we've had it happen to us during testing, especially if we did lots of firmware upgrade/downgrade testing.
- kabdib 9y agoYou can't really individually address a cell; a single read will drag in a bunch of cells, if only for ECC. Write granularity varies from device to device, but is often on the order of 256 or 512 bytes (though "append only" operations to blocks are sometimes supported). Erase granularity is what really matters, with values on the order of 64K being popular. So really you just need to keep track of erase block counts, which is relatively easy, and just maintain a "guess" as to read counts. It's much, much harder to keep track of logical block location, especially if you need to handle surprise removal of power.
- 9y ago
- Avernar 9y agoOne benefit of spreading out reads is detecting bad sectors on the drives that were normally not going to see much reads. Then the good data can be copied from the other mirror.
- sn 9y agoYou're better off running a weekly SMART long test https://linux.die.net/man/5/smartd.conf https://linux.die.net/man/5/smartd.conf and/or a periodic RAID check https://raid.wiki.kernel.org/index.php/RAID_Administration https://raid.wiki.kernel.org/index.php/RAID_Administration
- barrkel 9y agoI wonder if there are cache effects involved as well - continuously using the same SSD most of the time may reuse cached metadata relevant to the SSD innards that the kernel FS cache can't help with.
- Avernar 9y agoThe FS cache sits above the raid driver.
- barrkel 9y agoWell, it could hardly be below, could it. Was there an insight you wanted to mention?
- Avernar 9y agoLet me clarify. All the VFS caches are above the md driver. The cache we care about, the buffer cache, is not below the md driver. You'd need some kind of cache below the md driver to cache "metadata relevant to the SSD innards". Since there isn't one therefore there can't be a penalty due to caching to switch between SSDs in a mirror. That answers your question. But since you want an insight, here's one for you. There is no "metadata relevant to the SSD innards" available. The md drivers do not have this information. I wish they did. When I was still playing with the md code I'd love to know the block and page sizes of the drives. But that information is two bytes. Not really something that would need a cache.
- barrkel 9y agoLet me clarify what I was saying: I was redundantly (for clarity) talking about cache on the SSD, trying to make it extra clear that I wasn't talking about any kernel cache. I'm trying to figure out how, exactly, you've read my attempt at excessive clarity; it's not clear to me yet what you misread, since it seems like you're trying to correct some misunderstanding, but it hasn't worked because you haven't said anything that isn't already blindingly obvious. There is no "metadata relevant to the SSD innards" available. The md drivers do not have this information This is not an insight. The cache I'm interested in is tied up in how the SSD presents as a block device, but isn't implemented as a block device, as block devices are normally understood (wear levelling and remapping, hidden parallelism / striping, etc.). Yes, the FS cache can't help with this. Yes, the RAID implementation can't help with this. Of course. It's internal to the SSD. It's innards. I don't see what information you added.
- bhouston 9y agoBalancing reads may lead to improved performance if an individual SSD is near it's ops or datarate limits.
- pantalaimon 9y agoBut if this were the case the disc would not be idle, right?
- c0l0 9y agoPro tip: use RAID10 instead of RAID1 with Linux md to have the same level of redundancy for arrays with only two legs. RAID10 will properly stripe/balance reads across them, effectively doubling linear read throughput. The only real downside is that you will have to find and actively choose a sensible chunk size that fits both your hardware and workload.
- ryan-c 9y agoI think you have to specify layout=far2 with two drive raid10 on linux if you want that.
- rkeene2 9y agoI tried to get a patch in to do round robin reads in Linux RAID1 a few years ago due to this, but there was too great a request for doing a lot of benchmarks to show that it was ever useful that I abandoned getting it upstream. http://www.rkeene.org/viewer/tmp/linux-2.6.35.4-2rrrr.diff.htm http://www.rkeene.org/viewer/tmp/linux-2.6.35.4-2rrrr.diff.h...
- revelation 9y agoIt's either that or because you consistently spelled it round-robinn.
- rkeene2 9y agoThere was discussion about the spelling of "robbin" versus "robin", but that was easily corrected and a new patch submitted. It was definitely the request from the Linux RAID maintainer Neil Brown to provide benchmarks that stopped me from pursuing it further. I created the patch when someonein the Linux IRC channel was reporting this imbalance on their SSDs, so I had no access to hardware SSD hardware to test it on. This was a long time ago.
- rasz 9y agoCounter to common intuition SSD reads arent completely non destructive (they deplete stored charge) and rolling over READ DISTURB counter triggers wear leveling on affected Block. Micron listed 100K READ cycles per block for MLC, 1Mil for SLC, probably 1K for TLC. Thus balancing reads is good for SSD endurance.
- CodeWriter23 9y ago> Thus balancing reads is good for SSD endurance Ergo, non-balanced reading is better for endurance of the array, as it reduces the probability of contemporaneous multi-disk failure. Like disk 2 failing under the read load encountered during a rebuild.
- Avernar 9y agoBack when I still ran Linux on my personal servers I noticed this. I was adding TRIM support to the md drivers before the maintainers finally got around to it. That's when I noticed the balancing code tended to favour one disk. I added round-robin code and experimented with splitting large sequental requests between idle disks.
- smegel 9y agoDoes it matter? Do reads affect lifetime of SSDs? It can't be affecting performance, as if the drives were busy then it would be balanced much more evenly.