4 ms·

Ask HN: How can I create the most reliable and durable NAS today? I have a lot of very sentimental, very-important files, such as family photos and videos. And

by dannyw 6y ago

Ask HN: How can I create the most reliable and durable NAS today? I have a lot of very sentimental, very-important files, such as family photos and videos. And I simply like to hoard data. I currently have 8TB of data stored on a Synology DS218+ with RAID1, and monthly data scrubs (verifying checksums). It is backed up remotely to Google Drive (in encrypted form), and I also maintain an infrequently-updated, once-per-quarter disk clone with an external HDD. My biggest concern with my current setup is that the memory is non-ECC. Even though the files are checksummed, I am concerned that specific memory corruption / bit-flips could propagnate into the checksums, and hence result in data corruption. I am considering: * Building my own FreeNAS box using AMD Ryzen (which semi-officially support ECC memory). My concerns here are the semi-official nature of support: how do I know if ECC works, before a rare cosmic bit-flip? * Purchasing a Synology DS1621+. This is AUD$1400 which is a tough pill to swallow, for the equivalent of a first-gen Ryzen quad core and 4GB of memory. Any options or recommendations is appreciated!!

wojciii 6y ago
I just built my Truenas box using Ryzen. Let me know if you want a part list for inspiration or any tips.
- dannyw 6y ago
  I would love your part list! Did you go with ECC memory? Did you have any way of verifying the ECC is working and actually detecting/correcting bitflips? The other thing I am interested in is minimizing idle power consumption. Just to be more environmentally friendly.
  - wojciii 6y ago
    I don't know if you want to build the same kind of system but at least you can get a list of parts that work together. I use my Truenas box for storage using ZFS, VMs and NFS server for different PCs. I bought ECC memory as I understand this is more or less a requirement for ZFS. I found out that FreeBSD which Truenas is based on can give you info about what type of RAM is present. The command is: # dmidecode -t memory According to this I have ECC RAM. :) I did a build with the following parts: Case: SST-CS380 V2 (space for 8 3.5" drives, 2 x ). Mainboard: ASRock X470D4U2-2T Power supply: Seasonic Focus Plus 550W Gold 80 Plus Full Modular Power Supply RAM: Kingston Server Premier, DDR4, 16 GB, KSM26ED8/5 M CPU: AMD Ryzen 5 3600X Wraith Spire CPU Cooler: Arctic Liquid Freezer II NVMe to PCI bridge: ASUS Hyper M.2 x16 Gen 4 (PCIe 4.0/3.0) supports 4X NVMe M.2 devices (2242/2260/2280/22110) up to 256 Gbps for AMD TRX40 / X570 PCIe 4.0 NVMe RAID and Intel® RAID platform - CPU features I bought all this from Amazon.de. The main board is a server main board with 10 GE Ethernet and a console for flashing bios and remoting into the machine - no graphics card needed. This was expensive and you can most likely save a lot using a consumer grade main board. The RAM was taken from the list provided at https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=X... The CPU models supported are here: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#CPU https://www.asrockrack.com/general/productdetail.asp?Model=X... Be careful not to buy an AMD APU - this doesn't support ECC RAM for some insane reason. An APU would have build in graphics and the CPU. I use both SATA drives (long term storage) and SDD drives (for speed). I created two ZFS volumes (I don't remember the proper terms). One for rotating discs (which could sleep most of the time) and one using SDDs for fast storage which doesn't use much power. I have 6 KW solar cells with battery so I don't really care if the box uses a lot of power. During daylight its more or less free when sun is shining. I get next to nothing when selling the electricity that I generate and would like to use as much as possible locally.
Tepix 6y ago
Write the really important data for long-term storage to MDISC BD XL media (100GB each). If it's pictures you care about, i'm sure it's far less than 8TB that need the VIP treatment
- sedatk 6y ago
  See my other replies about reliability (or lack thereof) of M-DISC.
GauntletWizard 6y ago
You’ll know ECC works when it matters; When you encounter a flipped bit. With an 8TB RAID, that is likely to be within the next 24 months. Go with option 1, and Raid-Z2+. With RaidZ2, you’ll be able to not only detect but correct a flipped bit - Even if that flip happens when writing out the data. Pay attention to the counters. Your ZFS scrubs will report how many resilvers they have. You’re likely to encounter 1 in a scrub. You’re unlikely to encounter more than 2. If you see that, that’s when you check for memory errors. A single bad sector is likely the hard drive. Even a single flipped bit is likely a transient error; It could be your memory, or your disk, or anything in between. It happens at scale, and 8TB, read repeatedly, is a lot of bits. Look into rasdaemon and memtest86 - They’re the tools you use to debug what happens when The other advice I can give you: Don’t be paranoid. Your photos are likely to acquire bit rot. You will have dozens, even hundreds of bit flips that will happen in your lifetime. Of the many thousands of photos you will take, the chances that you will ever notice the discoloration or bad line that a bit flip will have in a photo are pretty small. Bit rot happens. Your photos are important to you, and you should treasure them, but treasure them for what they are: Things that you protect and are under your care, not things that must be twenty nines of correct. You can realistically achieve 10, even 12 nines of correct reads on your data. You don’t need more.
- amelius 6y ago
  But what if a bit flips in a header or in the OS's file length field?
  - GauntletWizard 6y ago
    ZFS checksums metadata, too.
stiray 6y ago
Regarding ECC and ZFS: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... "Next, you read a copy of the same block – this copy might be a redundant copy, or it might be reconstructed from parity, depending on your topology. The redundant copy is easy to visualize – you literally stored another copy of the block on another disk. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk. No data has been corrupted. A later scrub will attempt to read all copies of that block and validate them just as though the error had never happened, and if this time either copy passes, the error will be cleared and the block will be marked valid again (with any copies that don’t pass validation being overwritten from the one that did)." (dont just read the quote, read the link) I am using ZFS since 2007/2008 and I have never had any issues (except with those damn Seagate 3tb DeathStar hdds, where I was barely able to replace them fast enough - in 3 months 3 were gone - i will never buy seagate again) Co-creator of ZFS: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I am having a asus microatx board with 16gb of non ecc ram, additional SAS HBA and 3x3Tb Toshiba into zraid and additional 10TB HGST He3 disks + Ultrium 3000 (LTO-5, they are quite cheap today and are designed for 24/7 operation which is barely what I will ever encounter while tapes can be restored on latest LTOs if needed, while you can get cartridges on some sale for peanuts) for backups. There is no way in hell to go for my important data (like images) to disk only and tape is nice. You take the cartridge and store it at parents/gf/worspace drawer/... Anyway if I remembered correctly, google has lost 150k of user accounts in 201x and restored them from tapes. So even for cloud opinionated people, it still makes sense to shovel important data to tapes if you dont use the data in everyday processing (and just an info - even shelved disks die)

by dannyw 6y ago

wojciii 6y ago

I just built my Truenas box using Ryzen. Let me know if you want a part list for inspiration or any tips.

dannyw 6y ago
I would love your part list! Did you go with ECC memory? Did you have any way of verifying the ECC is working and actually detecting/correcting bitflips? The other thing I am interested in is minimizing idle power consumption. Just to be more environmentally friendly.
- wojciii 6y ago
  I don't know if you want to build the same kind of system but at least you can get a list of parts that work together. I use my Truenas box for storage using ZFS, VMs and NFS server for different PCs. I bought ECC memory as I understand this is more or less a requirement for ZFS. I found out that FreeBSD which Truenas is based on can give you info about what type of RAM is present. The command is: # dmidecode -t memory According to this I have ECC RAM. :) I did a build with the following parts: Case: SST-CS380 V2 (space for 8 3.5" drives, 2 x ). Mainboard: ASRock X470D4U2-2T Power supply: Seasonic Focus Plus 550W Gold 80 Plus Full Modular Power Supply RAM: Kingston Server Premier, DDR4, 16 GB, KSM26ED8/5 M CPU: AMD Ryzen 5 3600X Wraith Spire CPU Cooler: Arctic Liquid Freezer II NVMe to PCI bridge: ASUS Hyper M.2 x16 Gen 4 (PCIe 4.0/3.0) supports 4X NVMe M.2 devices (2242/2260/2280/22110) up to 256 Gbps for AMD TRX40 / X570 PCIe 4.0 NVMe RAID and Intel® RAID platform - CPU features I bought all this from Amazon.de. The main board is a server main board with 10 GE Ethernet and a console for flashing bios and remoting into the machine - no graphics card needed. This was expensive and you can most likely save a lot using a consumer grade main board. The RAM was taken from the list provided at https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=X... The CPU models supported are here: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#CPU https://www.asrockrack.com/general/productdetail.asp?Model=X... Be careful not to buy an AMD APU - this doesn't support ECC RAM for some insane reason. An APU would have build in graphics and the CPU. I use both SATA drives (long term storage) and SDD drives (for speed). I created two ZFS volumes (I don't remember the proper terms). One for rotating discs (which could sleep most of the time) and one using SDDs for fast storage which doesn't use much power. I have 6 KW solar cells with battery so I don't really care if the box uses a lot of power. During daylight its more or less free when sun is shining. I get next to nothing when selling the electricity that I generate and would like to use as much as possible locally.

Tepix 6y ago

Write the really important data for long-term storage to MDISC BD XL media (100GB each). If it's pictures you care about, i'm sure it's far less than 8TB that need the VIP treatment

sedatk 6y ago
See my other replies about reliability (or lack thereof) of M-DISC.

GauntletWizard 6y ago

You’ll know ECC works when it matters; When you encounter a flipped bit. With an 8TB RAID, that is likely to be within the next 24 months. Go with option 1, and Raid-Z2+. With RaidZ2, you’ll be able to not only detect but correct a flipped bit - Even if that flip happens when writing out the data. Pay attention to the counters. Your ZFS scrubs will report how many resilvers they have. You’re likely to encounter 1 in a scrub. You’re unlikely to encounter more than 2. If you see that, that’s when you check for memory errors. A single bad sector is likely the hard drive. Even a single flipped bit is likely a transient error; It could be your memory, or your disk, or anything in between. It happens at scale, and 8TB, read repeatedly, is a lot of bits. Look into rasdaemon and memtest86 - They’re the tools you use to debug what happens when The other advice I can give you: Don’t be paranoid. Your photos are likely to acquire bit rot. You will have dozens, even hundreds of bit flips that will happen in your lifetime. Of the many thousands of photos you will take, the chances that you will ever notice the discoloration or bad line that a bit flip will have in a photo are pretty small. Bit rot happens. Your photos are important to you, and you should treasure them, but treasure them for what they are: Things that you protect and are under your care, not things that must be twenty nines of correct. You can realistically achieve 10, even 12 nines of correct reads on your data. You don’t need more.

amelius 6y ago
But what if a bit flips in a header or in the OS's file length field?
- GauntletWizard 6y ago
  ZFS checksums metadata, too.

stiray 6y ago

Regarding ECC and ZFS: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... "Next, you read a copy of the same block – this copy might be a redundant copy, or it might be reconstructed from parity, depending on your topology. The redundant copy is easy to visualize – you literally stored another copy of the block on another disk. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk. No data has been corrupted. A later scrub will attempt to read all copies of that block and validate them just as though the error had never happened, and if this time either copy passes, the error will be cleared and the block will be marked valid again (with any copies that don’t pass validation being overwritten from the one that did)." (dont just read the quote, read the link) I am using ZFS since 2007/2008 and I have never had any issues (except with those damn Seagate 3tb DeathStar hdds, where I was barely able to replace them fast enough - in 3 months 3 were gone - i will never buy seagate again) Co-creator of ZFS: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I am having a asus microatx board with 16gb of non ecc ram, additional SAS HBA and 3x3Tb Toshiba into zraid and additional 10TB HGST He3 disks + Ultrium 3000 (LTO-5, they are quite cheap today and are designed for 24/7 operation which is barely what I will ever encounter while tapes can be restored on latest LTOs if needed, while you can get cartridges on some sale for peanuts) for backups. There is no way in hell to go for my important data (like images) to disk only and tape is nice. You take the cartridge and store it at parents/gf/worspace drawer/... Anyway if I remembered correctly, google has lost 150k of user accounts in 201x and restored them from tapes. So even for cloud opinionated people, it still makes sense to shovel important data to tapes if you dont use the data in everyday processing (and just an info - even shelved disks die)