6 ms·

How I installed TrueNAS on my new ASUSTOR NAS

80 points by ingve 3y ago

0x0000000 3y ago
> No warranty is given for faults caused by alternate operating systems. Sure, but the burden is on the manufacturer to prove the alternate OS caused the fault. They can't just void your warranty for installing another OS. https://en.m.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty_Act https://en.m.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warran...
- WirelessGigabit 3y ago
  The problem here is litigation. While you can take them up to court you will need a lot of $. And the Magnuson Moss Warranty Act does not contain a free-shifting statute. (unlike the Lemon Law which is a more specific version of Magnuson Moss). What I don't understand however is how we deal with features that aren't used by ASUS, say some form of encryption in the CPU. TrueNAS uses it and your NAS locks up. And Intel actually releases an errata for that CPU. Who is responsible? Should they give you a new one?
  - toast0 3y ago
    My perspective is if they're selling you hardware + software, the warranty covers the hardware in the context of running their software. If the accelerated encryption doesn't work on OtherOS, but they don't use it on their OS, too bad, that's not part of what they sold you. In other words, it's not unreasonable for them to request that you (re)install their software and show them the system is broken there. Only if you manage to actually break the device with OtherOS, does burden of proving who broke it matter. I'd personally argue, in most cases, the hardware is defective if it can be broken by doing wrong things with software; but it kind of depends on how far you go; if you go mucking about in the flash roms, it's debatable; extreme overclocking is on the user as well, etc.
bhauer 3y ago
I note the author added an external USB disk to act as the TrueNAS boot disk. I wonder if treating one of the M.2 disks as a boot disk is feasible. There are twelve M.2 slots in that kit, after all.
- geerlingguy 3y ago
  You can do that too, I just chose to use an external drive to keep things simple and compare apples to apples in terms of NVMe performance (part of my reason for installing TrueNAS was to compare its performance to ADM built into the NAS, running with the same 12-drive storage layouts).
- wmf 3y ago
  IMO it would make more sense to use the eMMC (even with the caveats mentioned in the article).
- voytec 3y ago
  I was setting up FreeNAS, TrueNAS or something similar (FreeBSD-based) for a client years ago and they required utilizing all SATA ports for actual storage and booting from RAID flash drives. They made A LOT of backup copies of these USB sticks and were told to make new copies after system upgrades and reconfiguration. Last time I had any info from that site - it worked well for them.
donmcronald 3y ago
This is a cool article. I wonder why all the content creators use TrueNAS. I evaluated TrueNAS Scale for a storage server and decided not to use it. I had two major complaints. First, when you create a ZFS pool, it partitions drives and puts swap on all of them. IIRC it may have even set up some mdadm RAID mirrors. It's a bunch of complexity and can leave you with swap on spinning disks if you don't notice and override the behavior. For me, whenever I see a bad default like that, where it might have made sense 10+ years ago, I get freaked out about what other poor defaults exist that I'm not noticing or what else may have been neglected for a decade. I use it as an negative indicator and often avoid the whole product at that point. Second, every time I'd boot (or maybe stop) a VM it would trample my ZFS tunables, specifically zfs_arc_max. I couldn't figure out exactly what was triggering it, but I don't want a system that doesn't play nice with the default CLI tooling. I actually had a really tough time finding a decent NAS and I'm going to end up building a plain old Linux system running Ubuntu 22.04. If anyone has any suggestions (not QNAP or Synology) for something that can do a good job of ZFS and 2-6 VMs, I'd love to hear them.
- mattpallissard 3y ago
  Plain old Linux system is the way to go. Mount your volumes, set `/etc/exports`, configure a keytab if you're into that sort of thing, and move on with life. It's honestly less messing around than any of the off the shelf tools. Depending on what you're doing you may want to mess with some kernel tunables. But even then, most of the time the defaults on a modern kernel are good enough for small stuff.
- voytec 3y ago
  > If anyone has any suggestions (not QNAP or Synology) for something that can do a good job of ZFS and 2-6 VMs, I'd love to hear them. I'm using FreeBSD and its native hypervisor called bhyve. It's due to overall OS preference, not features Linux lacks regarding ZFS.
  - lazide 3y ago
    Don’t use QNAP. It’s super buggy and you can’t trust it. I’ve had a ton of near misses with every piece of QNAP hardware and software I’ve used, and the only reason I haven’t lost data is due to sheer paranoia. Synnology is much better put together, but also tends to be a lot more conservative software stack wise. I’ve had no ‘wtf’ moments with Synnology equipment, unlike QNAP - but I have had a few sigh, that’s lame moments due to older/less efficient software approaches or the like. Anything custom, you’ll have to spend your time figuring everything out, which has pros and cons. Personally I’ve had zero issues with a multi-pool NAS that has 20 (spinning rust) + 6 SSDs totaling around 150TB on Ubuntu with some pretty standard intel server hardware. Been running that for 2 years, zero data loss, currently about 80% full. I’ve also learned a ton about ZFS and done some odd things for my own amusement. I wouldn’t recommend it if you want low maintenance though.
jamesu 3y ago
I was planning on doing this, but wasn't entirely sure how much of a bottleneck the CPU would be especially in some sort of encrypted raid setup...
- favorited 3y ago
  iXsystems sells turnkey TrueNAS boxes with Atom CPUs, which are comparable in terms of TDP, clock speed, core count, etc. Though the Atom CPUs support 16x more RAM, which ZFS will definitely take advantage of, and 50% more PCIe lanes. So they're not exactly apples-to-apples, but I don't think encrypted datasets would be the bottleneck.
- zamadatix 3y ago
  There is generally plenty of CPU. These cores support AES offload and the like. The main bottleneck comes from eac drive only getting 1x PCIe 3.0, from behind a switch. As long as you span at least two drives in some way the next limitation you hit is the 10G nic. There is something to be said for those $100 PCIe 8 to 4x NVMe cards. Stick 3 of those on a 13600k (2 in the split direct to CPU lanes, one in the indirect lanes) and you'll have more bandwidth per drive, a heck of a lot more cores with more power, the ability to slot a 25g nic, and the ability to load a ton more ram (officially the n5105 only supports 16GB, 32 works fine as far as I've been able to tell, 64 causes errors). The power usage, size, and cost will all be a little higher but if you're looking at a 12 SSD NAS it's probably worth thinking about IMO. You also get an extra x4 for the main os drive, instead of 8GB emmc which can only really be /boot for most realistic setups.
phren0logy 3y ago
Related tot he topic but not the article: As of very recently UnRaid also supports ZFS. Worth a look if you are evaluating TrueNAS and similar options.
- Ruthalas 3y ago
  Is this first-class/native support? Could you provide a link where I can read more? I am interested in this, but came up empty the last time I looked into it.
  - phantomathkg 3y ago
    https://unraid.net/blog/6-12-0-rc1 https://unraid.net/blog/6-12-0-rc1
hendersoon 3y ago
I'm looking forward to a bright future when NVMe drives are competitive with magnetic disks and we can all afford to put 12 of 'em in a tiny silent box like this for blazing fast local storage. It'll happen! Also, bravo to Asus for not only opening their hardware to make installing an alternate OS possible, but documenting it on their site. This makes me vastly more likely to consider them for my next purchase.
- Havoc 3y ago
  > silent From memory this does have a fan
jprd 3y ago
My first thought after the page loaded was that I was looking at some sort of old Playstation.
dfox 3y ago
As a guy who had installed vanilla Debian on a bunch of QNAP and Synology NAS boxes I don't understand what exactly is the reason why anyone would want ZFS. The thing is completely unintuitive when something breaks and also there are slight incompatibilities between the current implementations (which will bite you hard when you are trying to recover an failed array). The whatever filesystem you like on top of LVM2 on top of mdraid is a setup that has reasonable layering and one can understand each layer well enough to troubleshoot or performance tune that particular layer. With ZFS, you just have this neat, shiny, complete black box that does something in slightly incompatible ways.
- donmcronald 3y ago
  > I don't understand what exactly is the reason why anyone would want ZFS It makes snapshots, clones, and replication very simple to deal with.
SamuelAdams 3y ago
Looks like this device is for NVME drives. Is there any special tuning required for an all-ssd array versus rotational HDD’s? I’m also curious how long SSD’s are supposed to last relative to an equivalently sized HDD.
- spdy 3y ago
  https://www.backblaze.com/blog/ssd-edition-2022-drive-stats-review/ https://www.backblaze.com/blog/ssd-edition-2022-drive-stats-... https://www.backblaze.com/blog/backblaze-drive-stats-for-2022/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202... With there current sample its 0.89% SSD vs 1.39% HDD AFR but the SSD sample is to low atm.
- zamadatix 3y ago
  I have this and SSDs in it, running arch btw ;). One thing I did was go classic and use RAID due to the limited RAM support by the CPU (16GB officially, 32 works fine as far as I e been able to tell, 64 creates memory errors according to all that have tried). When I did this I went RAID4 since it makes more sense than RAID5 when you have SSDs. I was surprised to find out even cheap SSDs have better endurance than the typical high capacity drives you'll see used in a home NAS. Price was still ~4x per GB though.
frankharv 3y ago
6 NVMe on a single x8 Link Proving if you build it they will buy it.
- metadat 3y ago
  PCIe Gen 3 bandwidth x1: 1 GB/s, x8: 8 GB/s PCIe Gen 4 bandwidth x1: 2 GB/s, x8: 16 GB/s Yeah, good point.. but 8GB/s is still plenty to saturate a single 10gb ethernet link.
zamadatix 3y ago
One thing I did different is use the 8 gig emmc as /boot. This allows all of the data drives to be fully used and identically partitioned.

80 points by ingve 3y ago

> No warranty is given for faults caused by alternate operating systems. Sure, but the burden is on the manufacturer to prove the alternate OS caused the fault. They can't just void your warranty for installing another OS. https://en.m.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty_Act https://en.m.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warran...

WirelessGigabit 3y ago
The problem here is litigation. While you can take them up to court you will need a lot of $. And the Magnuson Moss Warranty Act does not contain a free-shifting statute. (unlike the Lemon Law which is a more specific version of Magnuson Moss). What I don't understand however is how we deal with features that aren't used by ASUS, say some form of encryption in the CPU. TrueNAS uses it and your NAS locks up. And Intel actually releases an errata for that CPU. Who is responsible? Should they give you a new one?
- toast0 3y ago
  My perspective is if they're selling you hardware + software, the warranty covers the hardware in the context of running their software. If the accelerated encryption doesn't work on OtherOS, but they don't use it on their OS, too bad, that's not part of what they sold you. In other words, it's not unreasonable for them to request that you (re)install their software and show them the system is broken there. Only if you manage to actually break the device with OtherOS, does burden of proving who broke it matter. I'd personally argue, in most cases, the hardware is defective if it can be broken by doing wrong things with software; but it kind of depends on how far you go; if you go mucking about in the flash roms, it's debatable; extreme overclocking is on the user as well, etc.

bhauer 3y ago

I note the author added an external USB disk to act as the TrueNAS boot disk. I wonder if treating one of the M.2 disks as a boot disk is feasible. There are twelve M.2 slots in that kit, after all.

geerlingguy 3y ago
You can do that too, I just chose to use an external drive to keep things simple and compare apples to apples in terms of NVMe performance (part of my reason for installing TrueNAS was to compare its performance to ADM built into the NAS, running with the same 12-drive storage layouts).
wmf 3y ago
IMO it would make more sense to use the eMMC (even with the caveats mentioned in the article).
voytec 3y ago
I was setting up FreeNAS, TrueNAS or something similar (FreeBSD-based) for a client years ago and they required utilizing all SATA ports for actual storage and booting from RAID flash drives. They made A LOT of backup copies of these USB sticks and were told to make new copies after system upgrades and reconfiguration. Last time I had any info from that site - it worked well for them.

donmcronald 3y ago

This is a cool article. I wonder why all the content creators use TrueNAS. I evaluated TrueNAS Scale for a storage server and decided not to use it. I had two major complaints. First, when you create a ZFS pool, it partitions drives and puts swap on all of them. IIRC it may have even set up some mdadm RAID mirrors. It's a bunch of complexity and can leave you with swap on spinning disks if you don't notice and override the behavior. For me, whenever I see a bad default like that, where it might have made sense 10+ years ago, I get freaked out about what other poor defaults exist that I'm not noticing or what else may have been neglected for a decade. I use it as an negative indicator and often avoid the whole product at that point. Second, every time I'd boot (or maybe stop) a VM it would trample my ZFS tunables, specifically zfs_arc_max. I couldn't figure out exactly what was triggering it, but I don't want a system that doesn't play nice with the default CLI tooling. I actually had a really tough time finding a decent NAS and I'm going to end up building a plain old Linux system running Ubuntu 22.04. If anyone has any suggestions (not QNAP or Synology) for something that can do a good job of ZFS and 2-6 VMs, I'd love to hear them.

mattpallissard 3y ago
Plain old Linux system is the way to go. Mount your volumes, set `/etc/exports`, configure a keytab if you're into that sort of thing, and move on with life. It's honestly less messing around than any of the off the shelf tools. Depending on what you're doing you may want to mess with some kernel tunables. But even then, most of the time the defaults on a modern kernel are good enough for small stuff.
voytec 3y ago
> If anyone has any suggestions (not QNAP or Synology) for something that can do a good job of ZFS and 2-6 VMs, I'd love to hear them. I'm using FreeBSD and its native hypervisor called bhyve. It's due to overall OS preference, not features Linux lacks regarding ZFS.
- lazide 3y ago
  Don’t use QNAP. It’s super buggy and you can’t trust it. I’ve had a ton of near misses with every piece of QNAP hardware and software I’ve used, and the only reason I haven’t lost data is due to sheer paranoia. Synnology is much better put together, but also tends to be a lot more conservative software stack wise. I’ve had no ‘wtf’ moments with Synnology equipment, unlike QNAP - but I have had a few sigh, that’s lame moments due to older/less efficient software approaches or the like. Anything custom, you’ll have to spend your time figuring everything out, which has pros and cons. Personally I’ve had zero issues with a multi-pool NAS that has 20 (spinning rust) + 6 SSDs totaling around 150TB on Ubuntu with some pretty standard intel server hardware. Been running that for 2 years, zero data loss, currently about 80% full. I’ve also learned a ton about ZFS and done some odd things for my own amusement. I wouldn’t recommend it if you want low maintenance though.

jamesu 3y ago

I was planning on doing this, but wasn't entirely sure how much of a bottleneck the CPU would be especially in some sort of encrypted raid setup...

favorited 3y ago
iXsystems sells turnkey TrueNAS boxes with Atom CPUs, which are comparable in terms of TDP, clock speed, core count, etc. Though the Atom CPUs support 16x more RAM, which ZFS will definitely take advantage of, and 50% more PCIe lanes. So they're not exactly apples-to-apples, but I don't think encrypted datasets would be the bottleneck.
zamadatix 3y ago
There is generally plenty of CPU. These cores support AES offload and the like. The main bottleneck comes from eac drive only getting 1x PCIe 3.0, from behind a switch. As long as you span at least two drives in some way the next limitation you hit is the 10G nic. There is something to be said for those $100 PCIe 8 to 4x NVMe cards. Stick 3 of those on a 13600k (2 in the split direct to CPU lanes, one in the indirect lanes) and you'll have more bandwidth per drive, a heck of a lot more cores with more power, the ability to slot a 25g nic, and the ability to load a ton more ram (officially the n5105 only supports 16GB, 32 works fine as far as I've been able to tell, 64 causes errors). The power usage, size, and cost will all be a little higher but if you're looking at a 12 SSD NAS it's probably worth thinking about IMO. You also get an extra x4 for the main os drive, instead of 8GB emmc which can only really be /boot for most realistic setups.

phren0logy 3y ago

Related tot he topic but not the article: As of very recently UnRaid also supports ZFS. Worth a look if you are evaluating TrueNAS and similar options.

Ruthalas 3y ago
Is this first-class/native support? Could you provide a link where I can read more? I am interested in this, but came up empty the last time I looked into it.
- phantomathkg 3y ago
  https://unraid.net/blog/6-12-0-rc1 https://unraid.net/blog/6-12-0-rc1

hendersoon 3y ago

I'm looking forward to a bright future when NVMe drives are competitive with magnetic disks and we can all afford to put 12 of 'em in a tiny silent box like this for blazing fast local storage. It'll happen! Also, bravo to Asus for not only opening their hardware to make installing an alternate OS possible, but documenting it on their site. This makes me vastly more likely to consider them for my next purchase.

Havoc 3y ago
> silent From memory this does have a fan

jprd 3y ago

My first thought after the page loaded was that I was looking at some sort of old Playstation.

dfox 3y ago

As a guy who had installed vanilla Debian on a bunch of QNAP and Synology NAS boxes I don't understand what exactly is the reason why anyone would want ZFS. The thing is completely unintuitive when something breaks and also there are slight incompatibilities between the current implementations (which will bite you hard when you are trying to recover an failed array). The whatever filesystem you like on top of LVM2 on top of mdraid is a setup that has reasonable layering and one can understand each layer well enough to troubleshoot or performance tune that particular layer. With ZFS, you just have this neat, shiny, complete black box that does something in slightly incompatible ways.

donmcronald 3y ago
> I don't understand what exactly is the reason why anyone would want ZFS It makes snapshots, clones, and replication very simple to deal with.

SamuelAdams 3y ago

Looks like this device is for NVME drives. Is there any special tuning required for an all-ssd array versus rotational HDD’s? I’m also curious how long SSD’s are supposed to last relative to an equivalently sized HDD.

spdy 3y ago
https://www.backblaze.com/blog/ssd-edition-2022-drive-stats-review/ https://www.backblaze.com/blog/ssd-edition-2022-drive-stats-... https://www.backblaze.com/blog/backblaze-drive-stats-for-2022/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202... With there current sample its 0.89% SSD vs 1.39% HDD AFR but the SSD sample is to low atm.
zamadatix 3y ago
I have this and SSDs in it, running arch btw ;). One thing I did was go classic and use RAID due to the limited RAM support by the CPU (16GB officially, 32 works fine as far as I e been able to tell, 64 creates memory errors according to all that have tried). When I did this I went RAID4 since it makes more sense than RAID5 when you have SSDs. I was surprised to find out even cheap SSDs have better endurance than the typical high capacity drives you'll see used in a home NAS. Price was still ~4x per GB though.

frankharv 3y ago

6 NVMe on a single x8 Link Proving if you build it they will buy it.

metadat 3y ago
PCIe Gen 3 bandwidth x1: 1 GB/s, x8: 8 GB/s PCIe Gen 4 bandwidth x1: 2 GB/s, x8: 16 GB/s Yeah, good point.. but 8GB/s is still plenty to saturate a single 10gb ethernet link.

zamadatix 3y ago

One thing I did different is use the 8 gig emmc as /boot. This allows all of the data drives to be fully used and identically partitioned.