6 ms·

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway, ZFS is a great choice either way. Definitely implement snapshots and a regular backup habit with plain zfs send/recv or syncoid.

CaliforniaKarl 5y ago
I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed. I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.
- hulitu 5y ago
  This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.
  - codechad 5y ago
    > Never use in a RAID disks from the same vendor. Wrong. The advice is to use the same vendor/model but from a different production run (batch).
  - wheybags 5y ago
    I very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(
    - lostlogin 5y ago
      If you had tried to get two that close in production, I think you’d have struggled. I didn’t bother and got drives off the stack and bought both, and they are a long way apart.
flyinghamster 5y ago
I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to put them back where they were, and the pool came up fine. whew I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
- Filligree 5y ago
  > I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS. If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.
- XorNot 5y ago
  You were never at any risk in this scenario. The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).
  - Filligree 5y ago
    Mostly true, but I've found this doesn't work if one of the drives is missing. It doesn't handle renames simultaneously with drive failure, or at least it didn't four years ago. Using /dev/disk/by-uuid is always safe, though.
- vero2 5y ago
  > were not in their expected locations on the new box This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/
vidarh 5y ago
My home fileserver has drives from four different manufacturers after I 20 years ago had bad experiences with closely correlated drive failures (the infamous IBM Death Star batch) I've never deployed anything without at a minimum mixing batches, and usually vendors since. We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to replace drives and restore would have been massively costly)
- mst 5y ago
  Those were actually some of my favourite drives cost wise ... -after- the problems happened. Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.
taneq 5y ago
Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D ) Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback would be annoying but acceptable.
- hddherman 5y ago
  Disclaimer: I'm a bumbling idiot who does this type of thing as a hobby. Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place. In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless. My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.
  - mst 5y ago
    Caveat that you probably already know but others reading may not: 'zfs send' while absolutely awesome when used for things it's designed for will stream the file system byte for byte so if there's corruption brewing you will absolutely transfer the corruption along with everything else. (taking a snapshot, mounting the snapshot, and then backing up from -that- via something 'cp -a' like is probably the safest option, IMO, though it's always possible I've missed something here in which case I hope somebody will reply correcting me)

CaliforniaKarl 5y ago

I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed. I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.

hulitu 5y ago
This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.
- codechad 5y ago
  > Never use in a RAID disks from the same vendor. Wrong. The advice is to use the same vendor/model but from a different production run (batch).
- wheybags 5y ago
  I very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(
  - lostlogin 5y ago
    If you had tried to get two that close in production, I think you’d have struggled. I didn’t bother and got drives off the stack and bought both, and they are a long way apart.

flyinghamster 5y ago

I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to put them back where they were, and the pool came up fine. whew I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.

Filligree 5y ago
> I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS. If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.
XorNot 5y ago
You were never at any risk in this scenario. The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).
- Filligree 5y ago
  Mostly true, but I've found this doesn't work if one of the drives is missing. It doesn't handle renames simultaneously with drive failure, or at least it didn't four years ago. Using /dev/disk/by-uuid is always safe, though.
vero2 5y ago
> were not in their expected locations on the new box This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/

vidarh 5y ago

My home fileserver has drives from four different manufacturers after I 20 years ago had bad experiences with closely correlated drive failures (the infamous IBM Death Star batch) I've never deployed anything without at a minimum mixing batches, and usually vendors since. We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to replace drives and restore would have been massively costly)

mst 5y ago
Those were actually some of my favourite drives cost wise ... -after- the problems happened. Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.

taneq 5y ago

Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D ) Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback would be annoying but acceptable.

hddherman 5y ago
Disclaimer: I'm a bumbling idiot who does this type of thing as a hobby. Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place. In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless. My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.
- mst 5y ago
  Caveat that you probably already know but others reading may not: 'zfs send' while absolutely awesome when used for things it's designed for will stream the file system byte for byte so if there's corruption brewing you will absolutely transfer the corruption along with everything else. (taking a snapshot, mounting the snapshot, and then backing up from -that- via something 'cp -a' like is probably the safest option, IMO, though it's always possible I've missed something here in which case I hope somebody will reply correcting me)