7 ms·

This article seems right up my alley, so here are some thoughts: - ZFS is pretty amazing in it's abilities, with it ushering in the age of software RAID over h

This article seems right up my alley, so here are some thoughts: - ZFS is pretty amazing in it's abilities, with it ushering in the age of software RAID over hardware RAID - ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. - TrueNAS is more targeted towards enterprise applications. If you want good utility as a home user then give Proxmox or the like a look. Then you can make it into more than just a NAS (if you're open to it). - If you want to make things even more simple then consider something like UnRAID. - ZFS' snapshotting can really shine on a virtualization server application, with the ability to revert KVM VMs to a previous state in a matter of seconds. Lookup Jim Salter's (great dude) Sanoid project to see a prime example. - I don't recall why, but I've heard that RAIDZ should be avoided, in favor of stripped mirrors.

sngz 4y ago
> If you want good utility as a home user then give Proxmox or the like a look. Isn't proxmox just virtualization? didn't know it can be used as a NAS too.
- thefunnyman 4y ago
  IMO NAS setup is much more straightforward on truenas. It’s doable in Proxmox inside a VM or LXC, but true as exposes this all directly via a nice UI. I personally use Proxmox with a simple Debian NFS VM, but for less technical users that just want stuff to work I tend to recommend they stick with truenas.
  - sngz 4y ago
    how well does it handle running the NFS VM and using the NAS to serve files to jellyfin / plex running in a VM?
- AviationAtom 4y ago
  As thefunnyman pointed out, TrueNAS will make for a more user friendly sharing experience, but you could also just load a container up with Samba on Proxmox, to give you the NAS feature you seek. The heavy lifting of software RAID/ZFS would then be handled by Proxmox.
vetinari 4y ago
> I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. I would advise using ZFS only with distros that come with it (i.e. Ubuntu, Proxmox), especially if you plan to have your / on it. I wasted too much time on CentOS with ZFS, would not do it again.
- AviationAtom 4y ago
  ZFS on Root just sounds like pain to me. I opt for MD RAID on root and then ZFS my other volumes. I would also say Ubuntu is probably the better choice for Linux ZFS, as CentOS seems to be lacking good support.
  - GekkePrutser 4y ago
    ZFS on root is really amazing on FreeBSD and the advantage is that you can snapshot your boot drive.
    - mvanbaak 4y ago
      have a look at Boot Environments. It really is amazing.
      - GekkePrutser 4y ago
        Yes I know, bectl. I use it. I just didn't want to mention it because the discussion was mainly about Linux. But FreeBSD has a really strong toolchain for this indeed.
        indigodaddy 4y ago
        Also a Solaris thing, yeah? (beadm etc)
  - mustache_kimono 4y ago
    Once you try it, you're never going back. Snapshots are made for things like system administration. Upgrade borked your system? Just rollback. Want to use the last version of your firewall config? I wrote a utility you might like to try, httm[1], which allows you to restore from your snapshot-ed unique versions. If you like ZFS, then trust me you have to have ZFS on root. [1]: https://crates.io/crates/httm https://crates.io/crates/httm
mardifoufs 4y ago
I know software raid is better overall but are there any advantages to hardware raid anymore? Is it just worse at everything?
- simonjgreen 4y ago
  I would go as far as to say "hardware raid" these days is limiting, expensive, and less performant that can be achieved with software RAID.
  - AviationAtom 4y ago
    I think the worst point is vendor lock-in. If your controller fails, and a replacement isn't available, then you may be dead in the water. That kind of goes against the very point of RAID.
    - GekkePrutser 4y ago
      But this is why businesses spend so much on their RAID controllers. To make sure they're in warranty and that kind of thing doesn't happen. Incidentally its also pretty great because no business buys them second hand without warranty. So they're usually available for half nothing. I don't use raid cards right now but I do use fibre channel which is also dirt cheap second hand
      - AdrianB1 4y ago
        3 weeks ago I had a controller failure in a manufacturing plant in Latin America. The contract with the manufacturer was to provide a replacement on site in 4 hours. Guess what, 8 hours later the technician with the replacement controller was still on the way. With TrueNAS I can move my drives to any other computer with the right interface and they will just work. I did this in the past 10 years of using TrueNAS.
        GekkePrutser 4y ago
        I can imagine but companies want to offload responsibility. If the manufacturer is late they can blame them, after all your IT manager paid for 4hr support so they've covered their ass. If your TrueNAS fails at work, it's your ass on the line :P I totally agree these special drive formats are really annoying. In this case I'd probably keep a spare on hand myself.
SkyMarshal 4y ago
> - ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. Agreed. Also for anyone using NixOS, I've found its ZFS support is first class and easy to set up: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_to_the_nixos_team_for_the_excellent/ https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...
KennyBlanken 4y ago
> I don't recall why, but I've heard that RAIDZ should be avoided, in favor of stripped mirrors. Most people care about random IO (also once your filesystem has been populated and in use for a while, true linear IO really ceases to be due to fragmentation.) Striped arrays lose random IO performance as drive count goes up; an array of mirrored pairs gains random IO performance. This is less of an issue with tiered storage and cache devices, especially given you almost have to work to find an SSD less than 256GB these days. You can only upgrade a zdev by upgrading all its drives; it's a lot nicer cash-flow-wise to gradually upgrade a mirrored pair here and there, or upgrade exactly how many pairs you need to for the space you need. With RAID-Z you have a drive fail and pray a second doesn't fail during the resilver. With RAID-Z2 you can have any two drives fail. With mirrors you can lose 50% of your drives (provided that they're the right drives.)
- magicalhippo 4y ago
  > also once your filesystem has been populated and in use for a while, true linear IO really ceases to be due to fragmentation Enough concurrent clients doing sequential IO also looks like random IO to a storage server.
- Nursie 4y ago
  > You can only upgrade a zdev by upgrading all its drives This is no longer the case, or at least, should no longer be the case soon. The ability to add drives to a zpool has been announced, and will trickle through to stable before too long.
agilob 4y ago
>- ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. FreeBSD migrated from own ZFS to OpenZFS so you have single ZFS implementation in BSD and Linux https://openzfs.github.io/openzfs-docs/Getting%20Started/FreeBSD.html https://openzfs.github.io/openzfs-docs/Getting%20Started/Fre...
AviationAtom 4y ago
I also forgot to mention, your guidance on ZFS memory requirements is outdated. From what I have heard recent releases have drastically reduced the size of ARC cache necessary. One person reported it working phenomenally on a newer Raspberry Pi.
- GekkePrutser 4y ago
  True but COW is even harder on an SD than ext4 is so I would really not use it on a pi unless it's not using SD storage :)
  - AviationAtom 4y ago
    I do think many folks using such applications are booting from disk. IIRC, Raspberry Pi 4 supports disk booting natively.
  - agapon 4y ago
    Been using ZFS on eMMC with things like Orange Pi-s and Rock64-s for a few years, so far works good for me.
barrkel 4y ago
Raidz (or preferably raidz2) is good for archival / media streaming / local backup and has good sequential read/write performance, while striped mirrors - raid10 - are better for random access read and write and are a little bit more redundant (i.e. reliable), but costs more in drives for the same usuable space. Raidz needs to read all of every drive to rebuild after a drive replacement while a striped mirror only needs to read one. However if you're regularly scrubbing zfs then you read it all regularly anyway. Raidz effectively has a single spindle for random or concurrent I/O since a whole stripe needs to be read or written at a time. Raidz also had a certain amount of wastage owing to how stripes round out (it depends on how many disks are in the array), but you still get a lot more space than striped mirrors. For a home user on a budget raidz2 usually makes more sense IMO, unless you need more concurrent & random I/O, in which case you should probably build and benchmark different configurations. I've been using zfs for over 10 years, starting with Nexenta, a defunct oddity with Solaris kernel and Ubuntu userland. These days I use Zfs on Linux. I've never lost data since I started.
- eatbitseveryday 4y ago
  > Raidz needs to read all of every drive to rebuild after a drive replacement while a striped mirror only needs to read one. However if you're regularly scrubbing zfs then you read it all regularly anyway. Not quite. Each vdev rebuild uses disks within it. A pool with multiple vdev each being raidz does not need to read all disks to rebuild a single raidz vdev. Your statement compares one vdev vs many vdevs. It just happens to be the case folks assume one large vdev with raidz and multiple vdev with mirroring. If you have a 3- or 4- way mirror wouldn’t ZFS read from all disks in the vdev to rebuild any added disks to the mirror (there can be more than one)?
  - barrkel 4y ago
    Sure, you can build striped raidz and 3+ way mirrors and other more exotic variants but the two most typical corners of the configuration space are plain raidz2+ for all the drives vs striped mirrors. You lose usable space not putting all your drives into a single parity array and maximizing usable space is usually why you go with parity. Mixing multiple arrays only makes sense from this perspective if you have a mix of drive sizes.
gjulianm 4y ago
> - TrueNAS is more targeted towards enterprise applications. If you want good utility as a home user then give Proxmox or the like a look. Then you can make it into more than just a NAS (if you're open to it). I have questions about this. I'm thinking of building my own NAS server, and I don't know which OS to use. On the one hand it looks like people recommend TrueNAS a lot, which is nice now that they have a Linux version, but I'm not really sure what does it offer over a raw Debian apart from web/configuration and some extra tools? I have quite some experience in running Debian systems and managing RAIDs (not with ZFS but doesn't seem too much of a jump) and I worry that TrueNAS, while nice at the beginning, might end up being limiting if I start to tweak too much (I plan on using that NAS for more things than just storage).
- AviationAtom 4y ago
  If you want it to be strictly a NAS then TrueNAS should suffice. If you want to do anything more then I'd consider Proxmox or Ubuntu.
- aborsy 4y ago
  What you miss with raw Debian compared to TrueNAS is compatibility. TrueNAS makes sure that all pieces are compatible with one another so that when you update each piece or OS, the storage doesn’t break. The whole package is tested throughly before release. Also, TrueNAS makes setup painless: users, permissions, shares, vdevs, ZFS tuning, nice dashboard etc. With Debian, you get a lot of config files and ansible playbooks that become hard to manage. Ideally you won’t run other stuff on a NAS, outside Docker.
  - AviationAtom 4y ago
    There's been a movement in the industry to bring storage back onto servers. They use the fancy buzz term of "hyper convergence" now though. I will definitely argue that TrueNAS gives stability and ease of management. Some of that can be found with Proxmox too though. I think it just really depends on which medium you prefer. Perhaps trying both is the best option?
cm2187 4y ago
Is there any reason not to use zfs for a NAS? I am thinking of a NAS hosting multimedia files, surely the ability to roll back must comes at a cost in term of disk space for large files.
- simcop2387 4y ago
  Because it's a copy on write file system The snapshots don't take up any extra space until you do any writing. The common parts between snapshots are shared between them in storage so you only pay the cost for any changes (rounded to some block size).
  - cm2187 4y ago
    But if I remux a large video, won't I have two versions of that video taking space on the disk? If I do that often, that will eat the capacity very quickly.

sngz 4y ago

> If you want good utility as a home user then give Proxmox or the like a look. Isn't proxmox just virtualization? didn't know it can be used as a NAS too.

thefunnyman 4y ago
IMO NAS setup is much more straightforward on truenas. It’s doable in Proxmox inside a VM or LXC, but true as exposes this all directly via a nice UI. I personally use Proxmox with a simple Debian NFS VM, but for less technical users that just want stuff to work I tend to recommend they stick with truenas.
- sngz 4y ago
  how well does it handle running the NFS VM and using the NAS to serve files to jellyfin / plex running in a VM?
AviationAtom 4y ago
As thefunnyman pointed out, TrueNAS will make for a more user friendly sharing experience, but you could also just load a container up with Samba on Proxmox, to give you the NAS feature you seek. The heavy lifting of software RAID/ZFS would then be handled by Proxmox.

vetinari 4y ago

> I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. I would advise using ZFS only with distros that come with it (i.e. Ubuntu, Proxmox), especially if you plan to have your / on it. I wasted too much time on CentOS with ZFS, would not do it again.

AviationAtom 4y ago
ZFS on Root just sounds like pain to me. I opt for MD RAID on root and then ZFS my other volumes. I would also say Ubuntu is probably the better choice for Linux ZFS, as CentOS seems to be lacking good support.
- GekkePrutser 4y ago
  ZFS on root is really amazing on FreeBSD and the advantage is that you can snapshot your boot drive.
  - mvanbaak 4y ago
    have a look at Boot Environments. It really is amazing.
    - GekkePrutser 4y ago
      Yes I know, bectl. I use it. I just didn't want to mention it because the discussion was mainly about Linux. But FreeBSD has a really strong toolchain for this indeed.
      - indigodaddy 4y ago
        Also a Solaris thing, yeah? (beadm etc)
- mustache_kimono 4y ago
  Once you try it, you're never going back. Snapshots are made for things like system administration. Upgrade borked your system? Just rollback. Want to use the last version of your firewall config? I wrote a utility you might like to try, httm[1], which allows you to restore from your snapshot-ed unique versions. If you like ZFS, then trust me you have to have ZFS on root. [1]: https://crates.io/crates/httm https://crates.io/crates/httm

mardifoufs 4y ago

I know software raid is better overall but are there any advantages to hardware raid anymore? Is it just worse at everything?

simonjgreen 4y ago
I would go as far as to say "hardware raid" these days is limiting, expensive, and less performant that can be achieved with software RAID.
- AviationAtom 4y ago
  I think the worst point is vendor lock-in. If your controller fails, and a replacement isn't available, then you may be dead in the water. That kind of goes against the very point of RAID.
  - GekkePrutser 4y ago
    But this is why businesses spend so much on their RAID controllers. To make sure they're in warranty and that kind of thing doesn't happen. Incidentally its also pretty great because no business buys them second hand without warranty. So they're usually available for half nothing. I don't use raid cards right now but I do use fibre channel which is also dirt cheap second hand
    - AdrianB1 4y ago
      3 weeks ago I had a controller failure in a manufacturing plant in Latin America. The contract with the manufacturer was to provide a replacement on site in 4 hours. Guess what, 8 hours later the technician with the replacement controller was still on the way. With TrueNAS I can move my drives to any other computer with the right interface and they will just work. I did this in the past 10 years of using TrueNAS.
      - GekkePrutser 4y ago
        I can imagine but companies want to offload responsibility. If the manufacturer is late they can blame them, after all your IT manager paid for 4hr support so they've covered their ass. If your TrueNAS fails at work, it's your ass on the line :P I totally agree these special drive formats are really annoying. In this case I'd probably keep a spare on hand myself.

SkyMarshal 4y ago

> - ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. Agreed. Also for anyone using NixOS, I've found its ZFS support is first class and easy to set up: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_to_the_nixos_team_for_the_excellent/ https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...

KennyBlanken 4y ago

> I don't recall why, but I've heard that RAIDZ should be avoided, in favor of stripped mirrors. Most people care about random IO (also once your filesystem has been populated and in use for a while, true linear IO really ceases to be due to fragmentation.) Striped arrays lose random IO performance as drive count goes up; an array of mirrored pairs gains random IO performance. This is less of an issue with tiered storage and cache devices, especially given you almost have to work to find an SSD less than 256GB these days. You can only upgrade a zdev by upgrading all its drives; it's a lot nicer cash-flow-wise to gradually upgrade a mirrored pair here and there, or upgrade exactly how many pairs you need to for the space you need. With RAID-Z you have a drive fail and pray a second doesn't fail during the resilver. With RAID-Z2 you can have any two drives fail. With mirrors you can lose 50% of your drives (provided that they're the right drives.)

magicalhippo 4y ago
> also once your filesystem has been populated and in use for a while, true linear IO really ceases to be due to fragmentation Enough concurrent clients doing sequential IO also looks like random IO to a storage server.
Nursie 4y ago
> You can only upgrade a zdev by upgrading all its drives This is no longer the case, or at least, should no longer be the case soon. The ability to add drives to a zpool has been announced, and will trickle through to stable before too long.

agilob 4y ago

>- ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. FreeBSD migrated from own ZFS to OpenZFS so you have single ZFS implementation in BSD and Linux https://openzfs.github.io/openzfs-docs/Getting%20Started/FreeBSD.html https://openzfs.github.io/openzfs-docs/Getting%20Started/Fre...

AviationAtom 4y ago

I also forgot to mention, your guidance on ZFS memory requirements is outdated. From what I have heard recent releases have drastically reduced the size of ARC cache necessary. One person reported it working phenomenally on a newer Raspberry Pi.

GekkePrutser 4y ago
True but COW is even harder on an SD than ext4 is so I would really not use it on a pi unless it's not using SD storage :)
- AviationAtom 4y ago
  I do think many folks using such applications are booting from disk. IIRC, Raspberry Pi 4 supports disk booting natively.
- agapon 4y ago
  Been using ZFS on eMMC with things like Orange Pi-s and Rock64-s for a few years, so far works good for me.

barrkel 4y ago

Raidz (or preferably raidz2) is good for archival / media streaming / local backup and has good sequential read/write performance, while striped mirrors - raid10 - are better for random access read and write and are a little bit more redundant (i.e. reliable), but costs more in drives for the same usuable space. Raidz needs to read all of every drive to rebuild after a drive replacement while a striped mirror only needs to read one. However if you're regularly scrubbing zfs then you read it all regularly anyway. Raidz effectively has a single spindle for random or concurrent I/O since a whole stripe needs to be read or written at a time. Raidz also had a certain amount of wastage owing to how stripes round out (it depends on how many disks are in the array), but you still get a lot more space than striped mirrors. For a home user on a budget raidz2 usually makes more sense IMO, unless you need more concurrent & random I/O, in which case you should probably build and benchmark different configurations. I've been using zfs for over 10 years, starting with Nexenta, a defunct oddity with Solaris kernel and Ubuntu userland. These days I use Zfs on Linux. I've never lost data since I started.

eatbitseveryday 4y ago
> Raidz needs to read all of every drive to rebuild after a drive replacement while a striped mirror only needs to read one. However if you're regularly scrubbing zfs then you read it all regularly anyway. Not quite. Each vdev rebuild uses disks within it. A pool with multiple vdev each being raidz does not need to read all disks to rebuild a single raidz vdev. Your statement compares one vdev vs many vdevs. It just happens to be the case folks assume one large vdev with raidz and multiple vdev with mirroring. If you have a 3- or 4- way mirror wouldn’t ZFS read from all disks in the vdev to rebuild any added disks to the mirror (there can be more than one)?
- barrkel 4y ago
  Sure, you can build striped raidz and 3+ way mirrors and other more exotic variants but the two most typical corners of the configuration space are plain raidz2+ for all the drives vs striped mirrors. You lose usable space not putting all your drives into a single parity array and maximizing usable space is usually why you go with parity. Mixing multiple arrays only makes sense from this perspective if you have a mix of drive sizes.

gjulianm 4y ago

> - TrueNAS is more targeted towards enterprise applications. If you want good utility as a home user then give Proxmox or the like a look. Then you can make it into more than just a NAS (if you're open to it). I have questions about this. I'm thinking of building my own NAS server, and I don't know which OS to use. On the one hand it looks like people recommend TrueNAS a lot, which is nice now that they have a Linux version, but I'm not really sure what does it offer over a raw Debian apart from web/configuration and some extra tools? I have quite some experience in running Debian systems and managing RAIDs (not with ZFS but doesn't seem too much of a jump) and I worry that TrueNAS, while nice at the beginning, might end up being limiting if I start to tweak too much (I plan on using that NAS for more things than just storage).

AviationAtom 4y ago
If you want it to be strictly a NAS then TrueNAS should suffice. If you want to do anything more then I'd consider Proxmox or Ubuntu.
aborsy 4y ago
What you miss with raw Debian compared to TrueNAS is compatibility. TrueNAS makes sure that all pieces are compatible with one another so that when you update each piece or OS, the storage doesn’t break. The whole package is tested throughly before release. Also, TrueNAS makes setup painless: users, permissions, shares, vdevs, ZFS tuning, nice dashboard etc. With Debian, you get a lot of config files and ansible playbooks that become hard to manage. Ideally you won’t run other stuff on a NAS, outside Docker.
- AviationAtom 4y ago
  There's been a movement in the industry to bring storage back onto servers. They use the fancy buzz term of "hyper convergence" now though. I will definitely argue that TrueNAS gives stability and ease of management. Some of that can be found with Proxmox too though. I think it just really depends on which medium you prefer. Perhaps trying both is the best option?

cm2187 4y ago

Is there any reason not to use zfs for a NAS? I am thinking of a NAS hosting multimedia files, surely the ability to roll back must comes at a cost in term of disk space for large files.

simcop2387 4y ago
Because it's a copy on write file system The snapshots don't take up any extra space until you do any writing. The common parts between snapshots are shared between them in storage so you only pay the cost for any changes (rounded to some block size).
- cm2187 4y ago
  But if I remux a large video, won't I have two versions of that video taking space on the disk? If I do that often, that will eat the capacity very quickly.