3 ms·

Slightly related: Anyone noticed how sloppy Samsung has become with the reliability of their NVMe firmware? I learned a lesson when I nearly lost a ton of data

Slightly related: Anyone noticed how sloppy Samsung has become with the reliability of their NVMe firmware? I learned a lesson when I nearly lost a ton of data in a ZFS mirror of two Samsung Pro 990 4TB NMVes, they might have been from the same batch and had the same firmware version. First one NVMe abandoned the system, then I made a ZFS send of a snapshot of the remaining one, followed by an rsync to have the raw data directly accessible as well, and during the rsync the second NVMe also abandoned ship. A reboot fixed it, and all data was still there, but only upgrading the firmware (to one which now has been superseded again) fixed the problem. I'm really considering no longer buying Samsung Pro NVMes.

fouc 1y ago
Been hearing of drive failures lately with Samsung, Sandisk, and even Crucial. Possibly because the bulk of recent drive production is getting reserved by the AI datacenters?
rbranson 1y ago
The 990 Pros have a temperature sensor issue. See https://github.com/openzfs/zfs/discussions/14793#discussioncomment-13130186 https://github.com/openzfs/zfs/discussions/14793#discussionc...
- qwertox 1y ago
  I think this was either a `number of TBW (terabytes written)` or `% of space used` issue, since both got removed by the OS within 9 hours delta and the same usage (couple of months, ~500GB), because they were in a mirror since the beginning. If it were a sensor issue, SMART data should have shown this. Not saying that a sensor issue does not exist, but I doubt that this was my problem (mine don't have a preinstalled heatsink, I prefer to use my own). I now use a 3-way mirror and am mixing brands. One very nice thing: the Samsung Pro 990 4TB has the exact same space (down to the byte) as the WD_BLACK SN850X 4TB, so they can be replaced without any issues. This rarely was the case with SSDs and HDDs and probably other NVMes. Looks like they learned.
pkulak 1y ago
I lost about a month of family chats to a Samsung NVME. Yes, I was taking daily backups of the DB, but the damn drive didn't fail catastrophically and instantly. It just started silently corrupting the database, causing little errors until I noticed a month later. By then, my only option was to restore from my backup right before the first error. The irony is that I'm a huge BTRFS fan, and use it on all my desktops. But this was a database on a server, so of course use EXT4 and be fine with silent data corruption. :/

fouc 1y ago

Been hearing of drive failures lately with Samsung, Sandisk, and even Crucial. Possibly because the bulk of recent drive production is getting reserved by the AI datacenters?

rbranson 1y ago

The 990 Pros have a temperature sensor issue. See https://github.com/openzfs/zfs/discussions/14793#discussioncomment-13130186 https://github.com/openzfs/zfs/discussions/14793#discussionc...

qwertox 1y ago
I think this was either a `number of TBW (terabytes written)` or `% of space used` issue, since both got removed by the OS within 9 hours delta and the same usage (couple of months, ~500GB), because they were in a mirror since the beginning. If it were a sensor issue, SMART data should have shown this. Not saying that a sensor issue does not exist, but I doubt that this was my problem (mine don't have a preinstalled heatsink, I prefer to use my own). I now use a 3-way mirror and am mixing brands. One very nice thing: the Samsung Pro 990 4TB has the exact same space (down to the byte) as the WD_BLACK SN850X 4TB, so they can be replaced without any issues. This rarely was the case with SSDs and HDDs and probably other NVMes. Looks like they learned.

pkulak 1y ago

I lost about a month of family chats to a Samsung NVME. Yes, I was taking daily backups of the DB, but the damn drive didn't fail catastrophically and instantly. It just started silently corrupting the database, causing little errors until I noticed a month later. By then, my only option was to restore from my backup right before the first error. The irony is that I'm a huge BTRFS fan, and use it on all my desktops. But this was a database on a server, so of course use EXT4 and be fine with silent data corruption. :/