12 ms·
Real world SSD wearout
- burntrelish1273 8y agoHmm.. in production, this makes sense. On a laptop, running an an OWC SSD for 6 years and zero errors so far (knock on silicon).
- spydum 8y agoSo I'm confused. I get the wear out level not going lower than 1%, but that doesn't mean the drive failed, right?
- craftyguy 8y agoI think technically it might still work (i.e. you could read/write to it without errors?), but the integrity of the data it is storing would be questionable.
- sebazzz 8y agoDoesn't it only mean that writes will fail?
- solotronics 8y agoI would think it would mean that your writes may be corrupted.
- wmf 8y agoSome SSDs go into read-only mode when they have used up their endurance but others will let you keep writing.
- kalleboo 8y agoIt should but due to crappy firmware many SSDs will just drop off the bus and become unreadable.
- mjb 8y agoThat's very dependent on the drive vendor, model, and even firmware version. In many cases, drives can go on working for a very long time after the SMART metrics show it has failed. Whether you want to take the "risk" depends a lot on how critical this particular device is in your infrastructure.
- tenebrisalietum 8y agoI have a 16GB SSD that used to be in a POS terminal before I bought it second hand. The SSD Wear Leveling Count attribute shows as failed. Still works. I have chosen to use it as part of a RAID1 for an OS install on a home server. Waiting to see how long it will last.
- WrtCdEvrydy 8y agoIt's kinda like the expiration date for foods, an estimate.
- deleted 8y ago[deleted]
- rb2k_ 8y agoIn production tests I've seen the synthetic "wear out level" go down to 1% and the machine still writing Petabytes of data without any issues. That being said, I would only do that on machines that are mostly disposable :)
- S_A_P 8y agoThis is a useful datapoint, but I feel the author could learn a thing or two from the backblaze disk report. It looks like there are a few use cases that cause high wear on SSDs. I dont see a lot of concrete numbers here. To me, it is more useful to say something like: these ssds will last 2 years given that their sustained write throughput is X GB/sec on average. From the SSD torture tests Ive seen, it is many petabytes of data that the average SSD must write before getting anywhere near "weared out".
- mtmail 8y agohttps://www.heise.de/newsticker/meldung/SSD-Langzeittest-beendet-Exitus-bei-9-1-Petabyte-3755009.html https://www.heise.de/newsticker/meldung/SSD-Langzeittest-bee... (German) did a long-term test of 12 drives in 2016. The best drive (Samsung SSD 850) managed 9.1PB, that took a year of constant writing, the worst (Crucials BX200) about 200TB. All drives managed multiple times more than specified.
- AstralStorm 8y agoThat is before quite amplification which can be pretty high. Real value depends on block size and write algorithms employed.
- milesvp 8y agoMy understanding with flash, having talked to someone who had to write his own wear-leveling code so his hardware could use cheap memory, is it's not so much the pure throughout that can kill a flash cell, but the write pattern that matters. My memory of the conversation is a bit hazy, but it's something like, in order to update even a single bit in a cell, it needs to be deleted and written to, and each cell only has so many write statistically before it's flagged no good. So to detroy a disk, you can strategically flip bits all over your disk, or with a disk with naive wear-leveling maybe just flip the same bit over and over. Good disks will make this harder with tricks like buffers (hopefully backed by batt/cap in the event of power outtage), and very clever wear-leveling algos, but you can definitely wear out flash memory with a fraction of the data and time you might expect if you have a truly pathological workload.
- gameswithgo 8y agoThe redis case is interesting. If the 1 minute interval was somewhat arbitrary you could almost double the life of the drive by just bumping it to 2 minutes.
- krallja 8y agoYes, depending on their use case for this frequent dump file, they may be happier using AOF or much-less-frequent saves instead. I believe a dump file is belt-and-suspenders secondary backup for almost all users.
- PeanutNore 8y agoI just can't trust a man who doesn't trust his suspenders
- pohl 8y agoMaybe it's the belt they don't trust.
- julienfr112 8y agoWhat really happens when the level is low ? Less capacity ? Data loss ? Performance decreases ?
- ddingus 8y agoThis varies by SSD. Some just fail hard and that ranges from bricking to loss of filesysten intregity. Others stop allowing writes. Still others will slow down, or lose capacity. I have had two die on me after a few years of work involving tons of writes. One cheap SSD just bricked. Gone. The Samsung I had started doing weird things with capacity. Personally, I just cycle them when I see a deal out there. Old ones do need to be powered up, from time to time just like USB thumb drives do, or the contents can be lost.
- dylan604 8y agoHow does one know how a particular model of SSD will behave? If I had the choice, I'd rather the drive enter a read-only state rather than becoming a brick. So when I'm shopping for a new drive, how do I use this a determining factor in choosing the new drive? Is it a known thing that Vendor X will brick, Vendor Y will read-only, or Vendor X Model XXX will brick while Vendor X Model YYY will read-only? If so, where can one find this level of information on the inner workings of a drive? Most tech reports are solely focused on read/write speeds as that's what the majority of people are concerned. Inquiring minds want to know...
- ddingus 8y agoDoing that as hard. With older ssds you can find reviews and other information. With new ones, I consider it a crapshoot.
- whitexn--g28h 8y agoIt's going to come down to the firmware and controller. That can even be different when the vendor and model are the same.
- maxxxxx 8y ago
- brokentone 8y agoThis article seems to have a trove of interesting data, but struggles to generalize many conclusions out of it.
- wyldfire 8y agoI think that's because it's a thinly veiled advertisement.
- deleted 8y ago[deleted]
- tru_pablo 8y agoI'm the author. We wanted to showcase some scenarios and that's it. I would love to hear any suggestions of what else we can gather and report.
- brokentone 8y agoSure... so, showing "here is a weird disk pattern -- they were running X on top of it -- consider not running X on SSD" with a sampleset of 1 is a logical fallacy and kinda a bizarre post. For small samplesets, going deep to understand unnecessary writes, tuning the clients and showing less SSD wear after tuning would be interesting. Or, assuming you have more than 1 client of each of these situations aggregating the data to show patterns would be far more useful. As has been mentioned elsewhere, for inspiration, Backblaze has really nice posts analyzing their device wear.
- tru_pablo 8y agoThanks. That makes sense. But while we do have a lot of clients, I really think that all of their setups are unique in some way. So starting from that we didn't find more Redises on ceph dumping a lot. > tuning the clients and showing less SSD wear after tuning would be interesting. Is just not an option for us, as we only do monitoring and don't have any ways of tuning.
- okl 8y agoSpyware as a service from a Muscovite company -- and you even have to pay for it. Which serious sys-admin would install this on his server?
- duskwuff 8y agoI think you posted this comment to the wrong thread. (Not that it would be any more appropriate of a comment elsewhere.)
- pheleven 8y agoNot to support his fairly trashy post in particular, but I believe his comment has utility. As an infrastructure person this post was concerning; this company is collecting a lot of data and has a lot of access, which I wouldn't trust. I also would not be thrilled if I was a paying customer having these details shared (even without attribution, as in the article), further reducing trust. I appreciate these kind of real-world detail posts, but it's not appropriate if it's not your infrastructure.
- tru_pablo 8y agoCould you please elaborate on the point? How we can improve? What potential problems do you see? Thanks
- pheleven 8y agoSure. First and foremost, do you have permission from your customers who you're researching and reporting on here? If you do, great, ignore me. If not you'd be breaching (my) trust if I was one of them. The data is not yours and it may be possible to infer who these datapoints belong to if so desired. If one could do that, they may be able to gain competitive advantage or otherwise exploit knowledge of infrastructure (social engineering for example). There is a big difference, IMO, in someone like backblaze releasing statistics. They own all of the hardware and they choose to release the data themselves. You (on the surface) appear to be harvesting data from your customers, digging through it, and presenting it. You also point out very specific cases, rather than aggregate pseudonymous data. You are collecting sensitive data from your customers environments. This doesn't inspire confidence that you treat it as such.
- NikolaNovak 8y agoIs there a recommended generalized tool (non-vendor specific) that can be reliably used to determine your SSD's health and outlook? (Unfortunately I have a mix of SSDs in use, even in single personal computer, and all the vendor individual software gets overly complex and bloated...)
- octosphere 8y agoFor Windows there is CrystalDiskInfo[1] [1] https://crystalmark.info/en/software/crystaldiskinfo/ https://crystalmark.info/en/software/crystaldiskinfo/ Also relevant: https://ef.gy/statistics:ssd-write-endurance https://ef.gy/statistics:ssd-write-endurance
- cm2187 8y agoAnd crystal disk info can be used as a CLI as well if you want to automate the monitoring.
- yjftsjthsd-h 8y agoOn unix-likes, I believe SMART tools works?
- duozerk 8y agoYes they do, see smartmontools.
- rincebrain 8y agosmartmontools also work great on Windows, OS X, ...
- ars 8y agoInstall smartmontools, then configure it to do a SHORT self test every night, and a LONG self test every week. You can do the same on Windows, but I don't know the tool name. The self tests are built in to the drives and do not require OS support, just a way to start them and read results. Better make sure the tool has a way to contact you if it finds an error! (Email by default - and test it!) The long test reads the entire disk looking for errors. Next, if you are using a Linux MD Raid array install mdadm and make sure it's configured to run a checkarray command (by default every month).
- api 8y agoWe had one SSD die in production, but it was on a system whose write load was enormous... like 50-60mb/sec sustained for over one year.
- whitepoplar 8y agoDo you remember what model it was?
- api 8y agoIntel I think, but not positive. It was a higher grade SSD. Could have been an isolated incident but it was definitely abused.
- antisthenes 8y agoOut of curiosity, could some of that write load be shifted over to a temporary RAM drive? What did the server do?
- otterpro 8y agoIs the SMART attribute “media wearout indicator” only available on enterprise-grade SSD and not on consumer-grade SSD? I checked my Samsung SSD as well as NVMe drives and I didn't find it on CrystalDiskInfo. EDIT: It looks like "wear leveling count" attribute is available on Intel (#233) and some high-end Samsungs (#177), as well as other SSD, but it's not on any of my SSD.
- paavoova 8y agoI have both "177 Wear_Leveling_Count" and "233 Media_Wearout_Indicator" on an old Samsung SATA II consumer SSD. On a newer one, there's only Wear_Leveling_Count, but there's also "241 Total_LBAs_Written", which gives an interesting metric: $ sudo smartctl -A /dev/sda | awk '/177/ {print $2,$4} /241/ {printf "%.3f\n", $10 * 512 / 1024^3}' Wear_Leveling_Count 097 4151.677 Just 4.1TiB written so far causes a 3% decrease in drive "endurance".
- vthriller 8y agoIf you're using smartmontools, try running `update-smart-drivedb` first: those fancy attribute names don't come from drives themselves but are always interpreted from (model, firmware, attribute id) tuple by the software you're using to query disks. Or you can just look for [0] directly if you just want to know what drives Media_Wearout_Indicator is currently defined for. [0] https://raw.githubusercontent.com/mirror/smartmontools/master/drivedb.h https://raw.githubusercontent.com/mirror/smartmontools/maste...
- apankrat 8y agoPretty much all modern SSDs support an attribute called "Lifetime remaining", which is basically the same thing as the wearout indicator. It's attribute 231 for many vendors, 169 for several others, 202 for Crucials and Micron.
- rzzzt 8y ago"Media wearout indicator" was recorded for OCZ drives, IIRC.
- true_tuna 8y agoYou can enable it. I’ll dig through my notes for the command.
- tru_pablo 8y agoAs an author I would greatly appreciate any suggestions in what kind of stats you would want us to gather and share.
- ploxiln 8y agodon't just trust the drive's SMART attributes - try to independently confirm when and how the drive is failing techreport did a long-term experiment on this (but with limited samples): https://techreport.com/review/27062/the-ssd-endurance-experiment-only-two-remain-after-1-5pb https://techreport.com/review/27062/the-ssd-endurance-experi...
- labarna 8y agoJust FYI, "to wear (out)" has an irregular past tense, and past participle forms: "wore" and "worn". So: 1) These disk, because of constant throughput, wore out. 2) These disk, because of constant throughput, are worn out.
- tru_pablo 8y agothnx
- nickjj 8y agoIt's interesting how long SSDs tend to last on development boxes. I've had a 256GB Crucial SSD running on my primary dev box and it's been powered on for 3.6 years. During that time it's been powered on 94 times. I run a ton of Dockerized apps and other things. The "wear leveling count" is at 163 which according to Crucial means the drive is at about 95% healthy.
- cjensen 8y agoWorth pointing out: good manufacturers like Micron/Crucial publish endurance figures for their drives. Endurance is the number of bytes you can write before the drive is officially worn out. Endurance numbers can vary by two orders of magnitude. So if you have a write-intensive app like a Jenkins server building a C++ project, you might want to pony up for a server-class drive with better endurance.
- NullPrefix 8y ago>building a C++ project Build in a ram disk and then copy whatever you need for permanent storage?
- gruez 8y agonot everyone has 64GB of ram.
- NullPrefix 8y ago64GB is cheaper than enterprise SSD.
- cmurf 8y agoIt does very much depend on the "SSD" in question, e.g. SD Card and eMMC don't tolerate nearly as many writes as a SATA SSD or NVMe, and for sure the former dislike having power cut. I've had a few name brand SD Cards go permanently read only as a result: neat I can still read my data, but also not neat that I can't erase them before sending them back under warranty.
- Fnoord 8y agoThe advice to consider to disable swap applies equally on (micro)SD and USB sticks. Just make sure you have enough RAM. Also, consider mounting directories such as /var/log as tmpfs (if you don't need the logs after reboot) or use a remote syslog.
- ScoJoh 8y agoPlace I work got hit HARD by this. Year and half before I started at this company they setup 4 SSD's in a RAID 5 config. One day while working on things I noticed a message on our server indicating a PDR1001 Error (Predictive Failure). So we ordered a new one. The new SSD arrived. We popped in the new one and the RAID started to rebuild.... Lo and behold during that operation Drive 1 threw the same error and the whole thing came crashing down... We ended up losing the whole array. I had NO idea we had SSD's in the system. I had no idea that no one was monitoring their life.... The moment I saw we had this issue I saw the writing on the wall. 4 SSD's in a raid 5 all installed at the same time... means all SSDs end up with the same approximate critical end of life. All I could do was shake my head at the whole thing... Pretty sure those in the charge who setup the array still don't understand why this situation was 100% avoidable...
- arminiusreturns 8y agoBesides the ssds, who in their right mind uses raid 5 anymore? It's been dead to me for years... since the first time I had to rebuild an r5 of 4tb disks and did the math on the time window for cascades. Also, not to nitpick, but you shouldn't wait to order a replacement that gets shipped to rebuild, you should have spares ready to go at all times for all raid systems. That might have been the difference between a cascade and a normal rebuild.
- simcop2387 8y agoYea any time I'm using a raid setup (as opposed to a cluster or other more intelligent system like zfs) I insist that there be at least one hot spare and a cold one ready. And never let anyone convince you a raid is a backup
- Narkov 8y agoHot and cold spares are rather wasteful. Just RAID1 and then use software (i.e. ceph) to make that redundant across multiple enclosures and ultimately, geographic zones/DC's.
- apankrat 8y ago"Media wearout indicator" is a vendor-specific attribute. In fact, there are NO standard (or as the article coyly calls them "basic") SMART attributes at all. A lot of vendors stick certain attributes into same slots, granted, but there's literally no industry-wide spec for any of the attributes. You have to go not by vendor, and not even by device, but by firmware revision to accurately interpret an attribute. The wearout indicator often sits at slot 233, but sometimes it will be in 230, sometimes - some other slot. Moreover, in some drives (e.g. OCZ or ADATA) 233 is undocumented at all and it will grow from 0 normalized. So this: > we implemented collection not of all the attributes, but only basic and not vendor-specific ones is misleading and inaccurate.
- jarym 8y agoI read the related article about Postgres SELECTs causing writes sometimes - I've used Postgres for years and never knew this! Anyway, I spend most of my time in the software world and occasionally have to setup hardware (or more commonly, diagnose hardware issues). Is there a good guide for how to setup things like Postgres or Redis with recommendations on swap/cache setup, RAID-level selection, etc?
- true_tuna 8y agoI did a similar thing and found even lower failure rates (across thousands of drives in a high write database environment). One thing intel recommended was to overprovision the drives. Setting the maxlba at 80% of drive capacity preserves optimal wear leveling. I also used a different drive layout etc. the long and short of it is nand wearout is kinda like quicksand. You grow up expecting it to be a way bigger problem than it actually is in your adult life.
- Rafuino 8y agoFor those of you deploying lots of SSDs for work, how often are you having to replace them due to wear out? Many have 5 years warranties with multiple PB written (or drive writes per day), so I wonder how things are in practice.
- true_tuna 8y agoWearout is very uncommon. Biggest thing to do is make sure you don’t fill them up. Full drives are unable to do their optimal wear leveling.
- Rafuino 8y agoWould you do anything different with your storage software or SSD procurement if you didn't have to worry about filling up SSDs or wear leveling? Wondering how big of a pain point it is (disclosure: I work on new SSD tech at Intel).
- khana 8y agoInsightful. Thank you for posting!
- rl3 8y agoThis is why Optane storage is amazing. Wears out about ten times slower, and has incredible 4K random read performance on top of it. It almost feels like Intel is squandering a massive advantage by not pushing it harder. I know Optane DIMMs are coming soon, but M.2 form factor would be nice as well.
- Rafuino 8y agoWhat would you use it for yourself? (disclosure: I work at Intel on Optane-related topics.)
- rl3 8y agoThe fast 4K random read is handy for delta backups to hard disk; I can read half a million files in about a minute, determine what changed, and send whatever small changes off to disk. Another great use case is for web/npm projects with a million dependencies, all flat file. Working with those is usually miserable otherwise. In the future I intend to crunch time series data with it.
- ksec 8y agoI accidentally discovered something similar as well, I saw my Disk usage was 30TB in the past weeks on my MacBook, I thought that is abnormal. Most of my work are browsing, and Editor coding. Turns out I had too many Tabs in Safari, and my Mac was out of memory constantly doing swapping. I was doing anywhere from 100GB to few hundred GB write per day. If anyone has a easy solution to this would be much appreciated. Right now I a just checking every few days to see if I did something silly that causes lots of paging. There was a time I don't think I need 32GB of Memory, now I think there is a real need.
- firethief 8y agoOn Linux I set the vm.min_free_kbytes on my laptops so they crash something instead of swapping until pull the plug. Maybe XNU has an analogous knob?
- rasz 8y ago>If anyone has a easy solution to this would be much appreciated. -use different browser, Vivaldi supports suspending and lazy tab loading -add more ram, in your apple case it probably means buying new laptop