3 ms·
I recently had a similar SSD failure, although it wasn't in a "new fileserver" but my daily use 2013 desktop. It was working, then it was producing write error
by loeg 8y ago
I recently had a similar SSD failure, although it wasn't in a "new fileserver" but my daily use 2013 desktop. It was working, then it was producing write errors corrupting my filesystem, then the whole system died, very quickly. Fortunately for me, some data was recoverable from the corrupted disk; I had a local backup from 12h prior, and a tarsnap backup from about the same time back.
(Um, here's where I have to be critical of tarsnap: their recovery performance is absolutely abysmal for small files. They're latency bound between you, their EC2 instance, and the backing S3 store. Think single or double digit kB/s and then think about how much data you back up with tarsnap. I can't recommend any other backup provider better, but this is an experience where tarsnap left me very disappointed.)
Looking at that SSD and my other SSDs' SMART data, they report extra blocks remaining in SMART, and you can monitor that as it goes down. Ideally you replace the drive before it gets to zero.
My primary mistake was simply not monitoring that data in an effective way.
I don't think anyone who monitors HDDs has any real expectation that the high-level SMART yes/no is going to protect them from data loss. Instead they look at highly predictive factors like "Reallocated_Sector_Ct" or "Raw_Read_Error_Rate" (or even plain old "Power_On_Hours").
For SSDs it's quite similar: Reallocated_Sector_Ct, Power_On_Hours_and_Msec, Available_Reservd_Space, Uncorrectable_Error_Cnt, Erase_Fail_Count, Workld_Media_Wear_Indic, Media_Wearout_Indicator. Maybe NAND_Writes_1GiB.
NVME SSDs provide SMART-like data on logpage 2 ("Available spare", "Percentage used", "Power on hours"). For some reason the NVME spec does not require media to accept host-initiated self-checks, so most NVMe drives don't have the same functionality as smartctl --test. :-(