17 ms·

Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish

Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server. Any questions or feedback about the post are more than welcome.

srinathkrishna 4y ago
Just wanted to share my appreciation for not just this post but all your work in recent times! Been following your trail since your post about Google promos and the set of useful projects you've been working on since then.
- mtlynch 4y ago
  Thanks so much for the kind words and for following along!
dhzhzjsbevs 4y ago
Did similar recently. Some suggestions for anyone else looking to do the same: i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.
- giantrobot 4y ago
  I'd be really careful about SMRs in a RAID. You can end up with no end of performance issues. It's all the downsides of a single SMR drive multiplied by however many drives are in the pools.
  - bcook 4y ago
    I think @dhzhzjsbevs meant that 8TB and higher is CMR. A quick google search seems to support that.
    - giantrobot 4y ago
      You actually have to be careful. There's disk sizes where the manufacturer will do say 8TB CMR and essentially the same drive with different firmware as a 10TB SMR. They'll also have a 10TB CMR model. You have to pay close attention to the model numbers. It's even more of a crapshoot if you shuck drives. You have to carefully research what externals are known to have CMRs. SMRs are a fucking blight.
      - dhzhzjsbevs 4y ago
        Yup, sorry got them mixed up. 8tb and up WD blues are all cmr. The Seagate ones are all hit and miss unless you go for Nas drives.
InvaderFizz 4y ago
A few points from someone with years managing raid and ZFS in arrays all the way up to 50 disks: RAID-Z1 is something I never consider without a solid backup to restore from and a plan to execute that process at least once in the lifecycle of an array. If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. The good news is that ZFS will tell you exactly which files you have lost data for and cannot rebuild. If you have those files, you can overwrite them with the backups to get your integrity back. The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file. For this reason, you need a strong(easily accessible, consistent, current) backup strategy and an acceptance of downtime with Z1. As for ECC, it's better, but your absolute worse case scenario is that you get a bit flip before the sync and hash happens, and now that bit flipped data is committed to disk and you think it's OK. I prefer ECC to avoid this, but you are still reaping a multitude of benefits from ZFS without ECC. The only valid rule for RAM and ZFS is that more RAM = more caching of recently read data. Single, or very few user appliances will see little benefit past 8GB even with 100TB unless you happen to be reading the same data over and over. Where ZFS shines is having hundreds of gigabytes of RAM and tens or more concurrent users mostly accessing the same data. That way the vast majority of reads are from RAM and the overall disk IOPS remain mostly idle. Most of the ZFS RAM myths come from Deduplication, which should be disregarded as a ZFS feature until they allow storing of the DDT on a Optane-like latency device. Even better would be offline deduplication, but I doubt that will be a thing in ZFS this decade.
- anarcat 4y ago
  > If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. [...] The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file. Wait, what? If a RAID-(z)1 ZFS array loses one disk, there's data loss? I've ran so many RAID-1 and RAID-10 arrays with mdadm that I can't even being to count them, and I had many drive failures. If any of those arrays would have corrupted data, I would have been mad as hell. What I am missing here? How is this even remotely acceptable?
  - InvaderFizz 4y ago
1500100900 4y ago
- "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does, but cheaper. ZFS snapshots obviously don't cover other failure modes like natural disasters or a break in, so RAID is still not a backup. - for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This can be tricky to check, so perhaps think about a UPS - if you have two M.2 slots you could use them to mirror two partitions from two different disks for your data pool's SLOG. The same could be done to form a new mirrored ZFS pool for the OS. In my case I even prefer the performance that a single-copy SLOG gives me at the risk of losing the most recent data before it's moved from the SLOG to the pool.
- walrus01 4y ago
  > - "RAID is not a backup" primarily because "you could rm -rf". or your house could burn down or somebody could steal the computer while you're away on vacation or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage or your house could flood lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.
- DarylZero 4y ago
  > - "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally. (Also fire, compromise, police confiscation, etc.)
  - aqfamnzc 4y ago
    > You need to be synchronizing to a _write-only_ backup archive. For my write-only backup needs, this free service known as S4 works wonderfully: http://www.supersimplestorageservice.com/ http://www.supersimplestorageservice.com/ (This is a joke.)
deleted 4y ago
[deleted]
sneak 4y ago
The slog is only used for synchronous writes, which most writes are not (as I understand it). Most workloads (ie non-db server) won't see much improvement with one.
XelNika 4y ago
> Performance topped out at 111 MiB/s (931 Mbps), which is suspiciously close to 1 Gbps. That's because of overhead in TCP over IPv4. You're testing the payload throughput, not the physical throughput. The theoretical maximum performance without jumbo frames is around 95%. https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficiency https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficien...
- mtlynch 4y ago
  Ah, good to know. Thanks!
AdrianB1 4y ago
I have a couple of home built TrueNAS systems for many years (since FreeNAS, ~ 10 years ago), here is some feedback: - with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster. - you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed - unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else * - * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it. - that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports - TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA - your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%. - that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU. - GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it. - unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network

srinathkrishna 4y ago

Just wanted to share my appreciation for not just this post but all your work in recent times! Been following your trail since your post about Google promos and the set of useful projects you've been working on since then.

mtlynch 4y ago
Thanks so much for the kind words and for following along!

dhzhzjsbevs 4y ago

Did similar recently. Some suggestions for anyone else looking to do the same: i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.

giantrobot 4y ago
I'd be really careful about SMRs in a RAID. You can end up with no end of performance issues. It's all the downsides of a single SMR drive multiplied by however many drives are in the pools.
- bcook 4y ago
  I think @dhzhzjsbevs meant that 8TB and higher is CMR. A quick google search seems to support that.
  - giantrobot 4y ago
    You actually have to be careful. There's disk sizes where the manufacturer will do say 8TB CMR and essentially the same drive with different firmware as a 10TB SMR. They'll also have a 10TB CMR model. You have to pay close attention to the model numbers. It's even more of a crapshoot if you shuck drives. You have to carefully research what externals are known to have CMRs. SMRs are a fucking blight.
    - dhzhzjsbevs 4y ago
      Yup, sorry got them mixed up. 8tb and up WD blues are all cmr. The Seagate ones are all hit and miss unless you go for Nas drives.

InvaderFizz 4y ago

A few points from someone with years managing raid and ZFS in arrays all the way up to 50 disks: RAID-Z1 is something I never consider without a solid backup to restore from and a plan to execute that process at least once in the lifecycle of an array. If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. The good news is that ZFS will tell you exactly which files you have lost data for and cannot rebuild. If you have those files, you can overwrite them with the backups to get your integrity back. The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file. For this reason, you need a strong(easily accessible, consistent, current) backup strategy and an acceptance of downtime with Z1. As for ECC, it's better, but your absolute worse case scenario is that you get a bit flip before the sync and hash happens, and now that bit flipped data is committed to disk and you think it's OK. I prefer ECC to avoid this, but you are still reaping a multitude of benefits from ZFS without ECC. The only valid rule for RAM and ZFS is that more RAM = more caching of recently read data. Single, or very few user appliances will see little benefit past 8GB even with 100TB unless you happen to be reading the same data over and over. Where ZFS shines is having hundreds of gigabytes of RAM and tens or more concurrent users mostly accessing the same data. That way the vast majority of reads are from RAM and the overall disk IOPS remain mostly idle. Most of the ZFS RAM myths come from Deduplication, which should be disregarded as a ZFS feature until they allow storing of the DDT on a Optane-like latency device. Even better would be offline deduplication, but I doubt that will be a thing in ZFS this decade.

anarcat 4y ago
> If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. [...] The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file. Wait, what? If a RAID-(z)1 ZFS array loses one disk, there's data loss? I've ran so many RAID-1 and RAID-10 arrays with mdadm that I can't even being to count them, and I had many drive failures. If any of those arrays would have corrupted data, I would have been mad as hell. What I am missing here? How is this even remotely acceptable?
- InvaderFizz 4y ago

1500100900 4y ago

- "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does, but cheaper. ZFS snapshots obviously don't cover other failure modes like natural disasters or a break in, so RAID is still not a backup. - for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This can be tricky to check, so perhaps think about a UPS - if you have two M.2 slots you could use them to mirror two partitions from two different disks for your data pool's SLOG. The same could be done to form a new mirrored ZFS pool for the OS. In my case I even prefer the performance that a single-copy SLOG gives me at the risk of losing the most recent data before it's moved from the SLOG to the pool.

walrus01 4y ago
> - "RAID is not a backup" primarily because "you could rm -rf". or your house could burn down or somebody could steal the computer while you're away on vacation or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage or your house could flood lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.
DarylZero 4y ago
> - "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally. (Also fire, compromise, police confiscation, etc.)
- aqfamnzc 4y ago
  > You need to be synchronizing to a _write-only_ backup archive. For my write-only backup needs, this free service known as S4 works wonderfully: http://www.supersimplestorageservice.com/ http://www.supersimplestorageservice.com/ (This is a joke.)

deleted 4y ago

[deleted]

sneak 4y ago

The slog is only used for synchronous writes, which most writes are not (as I understand it). Most workloads (ie non-db server) won't see much improvement with one.

XelNika 4y ago

> Performance topped out at 111 MiB/s (931 Mbps), which is suspiciously close to 1 Gbps. That's because of overhead in TCP over IPv4. You're testing the payload throughput, not the physical throughput. The theoretical maximum performance without jumbo frames is around 95%. https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficiency https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficien...

mtlynch 4y ago
Ah, good to know. Thanks!

AdrianB1 4y ago

I have a couple of home built TrueNAS systems for many years (since FreeNAS, ~ 10 years ago), here is some feedback: - with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster. - you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed - unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else * - * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it. - that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports - TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA - your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%. - that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU. - GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it. - unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network