5 ms·

I build a zfs based NAS last year[1], after 10 months of usage, I have nothing but positive things to say. One thing I didn't understand conceptually at first i

I build a zfs based NAS last year[1], after 10 months of usage, I have nothing but positive things to say. One thing I didn't understand conceptually at first is that ZFS is standalone and not dependent on the host OS. This was not clear to me and it is not explained anywhere. You can wipe out the entire OS, move the drives to a new system, mount them on a new OS and run `zpool import` to import them. All the information (raid config, snapshots, etc) about ZFS is stored on the pool itself and not on the host OS, especially if you have an alias setup for invidivual drives in `/etc/zfs/vdev_id.conf`. There is no need for FreeNAS or any of that stuff, I don't use all those features. Just run like 3 commands to create a zpool, install samba service and that's all I need from a NAS. Cron job to run a scrub every month. If ZFS file system made a sound, it'd make a satisfying and reassuring "CLUNK!" sound. [1] https://neil.computer/notes/zfs-raidz2/ https://neil.computer/notes/zfs-raidz2/

aborsy 4y ago
TrueNAS ensures that different versions of different software fit together. So you update your OS, and your setup doesn’t break. The common tasks are made easy: snapshot scheduling, snapshot replication, Samba share with a few clicks, backup to cloud, users management, one click install of applications, SMART tests, etc. Config management can be a headache.
- neilpanchal 4y ago
  I use zfs-auto-snapshot for snapshots, it works great. I think the only thing I would want is early warning for a disk failure. I'll probably automate that. TrueNAS feels extremely heavy with too many features I don't need. There is a web server, database, etc which feels overkill for a simple home NAS IMO and having to manage the management system.
  - mustache_kimono 4y ago
    > I think the only thing I would want is early warning for a disk failure. https://manpages.ubuntu.com/manpages/jammy/man8/zed.8.html https://manpages.ubuntu.com/manpages/jammy/man8/zed.8.html
  - mustache_kimono 4y ago
    > I think the only thing I would want is early warning for a disk failure. The only other thing, you may be forgetting, is an app you never knew you needed.[0] ;) [0, an interactive, file-level Time Machine-like tool for ZFS]: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
    - neilpanchal 4y ago
      Hnggg.... this thing looks cool. A handful of .rs files in the /src folder. Sold.
      - mustache_kimono 4y ago
        A man/person of taste!
- vladvasiliu 4y ago
  As GP said, all these features are nice, but if you don't need them, they're just overhead. I'm in the same boat, all I have on my NAS is Samba, zrepl (ZFS snapshots and backups) and node exporter (monitoring agent for Prometheus - handles SMART, etc). It's running Arch, and I've never had my setup "break" in more than 10 years of using this distro (though this particular NAS is not that old). I don't care for a DB, web server and god forbid random applications doing who knows what on my NAS. But if you want your NAS to do everything, it should be great. To each their own, I guess. Also, TrueNAS is FreeBSD which may or may not work on dinkier hardware. I'm specifically thinking of random watchdog problems on some older Realtek Gigabit NICs, where you had to use a specific patched driver which, at one point, didn't support the latest version.
  - neilpanchal 4y ago
    Yea, TrueNAS comes with a jails manager which has nothing to do with NAS. It's incredibly feature rich: https://static.ixsystems.co/uploads/2020/04/image-4.png https://static.ixsystems.co/uploads/2020/04/image-4.png Plus the setup now gets intricately coupled with the host OS. That's a big no for me. I want to treat the host OS as disposable, ephemeral and decoupled for peace of mind. The host's only job is to serve files over samba protocol and run scrub cronjobs. That's it. It gets 2 vCPUs and 24GB of ram. Stays put.
    - vladvasiliu 4y ago
      I guess there could be an argument for the NAS part to be one of the jails. But then, the whole thing should be marketed more like "Proxmox with batteries included".
      - cassianoleal 4y ago
        > the whole thing should be marketed more like "Proxmox with batteries included". It's not though. Even if you make all the "NAS software" part of the jails (or the half-baked Kubernetes in TrueNAS SCALE), it's not even close to Proxmox when it comes to managing VMs, containers/jails, firewall, networking, etc.
bombcar 4y ago
My NAS is a zfs-root Gentoo system. I went with it because I couldn’t beat freenas and friends into doing what I wanted.
asveikau 4y ago
Is this where we're giving testimonials? I've been a ZFS user for maybe 6 years. I had data loss this year. It was quite interesting. Writes would fail on multiple disks at the same time. That's why it was data loss. A normal disk failure wouldn't look like that, failures should be spread out and redundancy would help. It turned out to be a bad power supply. I was able to predictably get it to corrupt writes with the old PSU. Then I replaced the PSU. No longer failed after that. I wouldn't have guessed to suspect the PSU. It was a frustrating experience but in the end, ZFS did help me detect it. On ext4 or ffs I wouldn't have even been aware it happened, let alone could confirm a fix.
- aborsy 4y ago
  Copy-on-write file systems such as ZFS should be resilient to power outages. How does power fluctuations or cutoff cause data loss in ZFS? I frequently force shutdown my laptop and PCs with ext4 when they hang, and never lost data.
  - mustache_kimono 4y ago
    > and never lost data. How do you know? I've seen a transient error due to a misconfiguration of ALPM that XFS/ext4/etc probably would have never caught, but ZFS did.
    - aborsy 4y ago
      Because a good part of it is checksumed with restic.
      - mustache_kimono 4y ago
        I don't mean to be critical of you, restic, or your backup strategies, but it doesn't seem like you know. ZFS is the I really need to know filesystem. I think it's pretty great and I use it where I can. But if it's not for you, it's not for you. And FYI, an power supply failure does not manifest the same as a power outage.
  - 4y ago
bubblethink 4y ago
>One thing I didn't understand conceptually at first is that ZFS is standalone and not dependent on the host OS. This is true for every mature/production file system. You can do the same with mdadm, ext4, xfs, btrfs, etc. The only constraint would be with versions in that it would be a one way street. Can't necessarily go from something new to something old, but other way round is fine.
- neilpanchal 4y ago
  Right, what was not clear to me was that raid configuration and snapshots are also part of the file system. Usually, that's done through hardware cards or software raid where the configuration sits not on the file system (?) (Intel VROC, vSAN, etc), at least that was my wrong conceptual model. I used to make multiple copies of the USB stick of FreeNAS 8 back in the day because I didn't want the USB drive to fail and not knowing how to recover the zpool. Messing around with the file system directly cleared up everything.
  - viraptor 4y ago
    That's more common for Linux storage. Whether you're using zfs, btrfs, or lvm, all the configuration required to read it is stored in the header somewhere rather than in a detached configuration.
    - vetinari 4y ago
      It has to be stored out of the band, otherwise you would have chicken-and-egg problem: what's the shape of the array, when the info is in a file stored inside the array?
      - viraptor 4y ago
        It's not stored in a file inside the array of course, but there are two types of out of band: next to the data, or completely disconnected. For example you only need to mount a single btrfs partition - in the header it already contains the information about other copies and will mount the whole raid setup as necessary. It doesn't matter if you move the drives somewhere else to a new system - mount one of them and it works. On the other hand, if you move drives from a hardware raid and put in new drives, some (all?) controllers will read the raid config from memory and offer to build the same raid on the new drives. That's completely-out-of-band. Depending on the controller, even changing the order the disks are plugged in can give you weird results.
yjftsjthsd-h 4y ago
Oh, it's better than that! If you're crazy enough, you can even multiboot multiple OSs on the same pool:) You do need to ensure that the pool features are sufficiently compatible between ZFS drivers (ex. you can't create a fully-featured pool with OpenZFS 2.1.6 on Linux and import it on OpenIndiana, last I checked), but it does work, empirically;)
- neilpanchal 4y ago
  A friend of mine suggested this and I didn't like the idea of putting the OS on there especially that it is a virtual machine. There are a couple of things that made this setup nice though: PCIe bifurcation (4x4x4x4) lanes straight to the CPU and PCIe passthrough in ESXi. From what I read that ZFS needs direct access to the underlying hardware. So, I just treat the OS as totally disposable and not worth backing up. I can spin up a VM in less than 2 mins if something were to happen to it.
  - chasil 4y ago
    SmartOS is literally disposable - it runs off a flash drive. Have you considered it for your use case?
phpisthebest 4y ago
>no need for FreeNAS or any of that stuf I would encourage anyone that does not have an system admin background to use FreeNAS(truenas now) or something Their are default security and maintenance things (like periodic scrubs of the pool) that these nas operating systems set up by default
- barrkel 4y ago
  ZoL on Ubuntu adds a cronjob for zfs scrub; shipped in xenial (16.04).
deleted 4y ago
[deleted]
jiveturkey 4y ago
> standalone and not dependent on the host OS. ?? The OS needs to implement zfs, like any other filesystem/host combination. Any filesystem can be "moved" to another host simply by attaching the drives, assuming you have hardware level compat and filesystem level compat. OK the exception is when you have host-based hardware level encryption (ie, key in TPM or other security chip). In that case, zfs and otherwise, you can't just move the drives.

aborsy 4y ago

TrueNAS ensures that different versions of different software fit together. So you update your OS, and your setup doesn’t break. The common tasks are made easy: snapshot scheduling, snapshot replication, Samba share with a few clicks, backup to cloud, users management, one click install of applications, SMART tests, etc. Config management can be a headache.

neilpanchal 4y ago
I use zfs-auto-snapshot for snapshots, it works great. I think the only thing I would want is early warning for a disk failure. I'll probably automate that. TrueNAS feels extremely heavy with too many features I don't need. There is a web server, database, etc which feels overkill for a simple home NAS IMO and having to manage the management system.
- mustache_kimono 4y ago
  > I think the only thing I would want is early warning for a disk failure. https://manpages.ubuntu.com/manpages/jammy/man8/zed.8.html https://manpages.ubuntu.com/manpages/jammy/man8/zed.8.html
- mustache_kimono 4y ago
  > I think the only thing I would want is early warning for a disk failure. The only other thing, you may be forgetting, is an app you never knew you needed.[0] ;) [0, an interactive, file-level Time Machine-like tool for ZFS]: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
  - neilpanchal 4y ago
    Hnggg.... this thing looks cool. A handful of .rs files in the /src folder. Sold.
    - mustache_kimono 4y ago
      A man/person of taste!
vladvasiliu 4y ago
As GP said, all these features are nice, but if you don't need them, they're just overhead. I'm in the same boat, all I have on my NAS is Samba, zrepl (ZFS snapshots and backups) and node exporter (monitoring agent for Prometheus - handles SMART, etc). It's running Arch, and I've never had my setup "break" in more than 10 years of using this distro (though this particular NAS is not that old). I don't care for a DB, web server and god forbid random applications doing who knows what on my NAS. But if you want your NAS to do everything, it should be great. To each their own, I guess. Also, TrueNAS is FreeBSD which may or may not work on dinkier hardware. I'm specifically thinking of random watchdog problems on some older Realtek Gigabit NICs, where you had to use a specific patched driver which, at one point, didn't support the latest version.
- neilpanchal 4y ago
  Yea, TrueNAS comes with a jails manager which has nothing to do with NAS. It's incredibly feature rich: https://static.ixsystems.co/uploads/2020/04/image-4.png https://static.ixsystems.co/uploads/2020/04/image-4.png Plus the setup now gets intricately coupled with the host OS. That's a big no for me. I want to treat the host OS as disposable, ephemeral and decoupled for peace of mind. The host's only job is to serve files over samba protocol and run scrub cronjobs. That's it. It gets 2 vCPUs and 24GB of ram. Stays put.
  - vladvasiliu 4y ago
    I guess there could be an argument for the NAS part to be one of the jails. But then, the whole thing should be marketed more like "Proxmox with batteries included".
    - cassianoleal 4y ago
      > the whole thing should be marketed more like "Proxmox with batteries included". It's not though. Even if you make all the "NAS software" part of the jails (or the half-baked Kubernetes in TrueNAS SCALE), it's not even close to Proxmox when it comes to managing VMs, containers/jails, firewall, networking, etc.

bombcar 4y ago

My NAS is a zfs-root Gentoo system. I went with it because I couldn’t beat freenas and friends into doing what I wanted.

asveikau 4y ago

Is this where we're giving testimonials? I've been a ZFS user for maybe 6 years. I had data loss this year. It was quite interesting. Writes would fail on multiple disks at the same time. That's why it was data loss. A normal disk failure wouldn't look like that, failures should be spread out and redundancy would help. It turned out to be a bad power supply. I was able to predictably get it to corrupt writes with the old PSU. Then I replaced the PSU. No longer failed after that. I wouldn't have guessed to suspect the PSU. It was a frustrating experience but in the end, ZFS did help me detect it. On ext4 or ffs I wouldn't have even been aware it happened, let alone could confirm a fix.

aborsy 4y ago
Copy-on-write file systems such as ZFS should be resilient to power outages. How does power fluctuations or cutoff cause data loss in ZFS? I frequently force shutdown my laptop and PCs with ext4 when they hang, and never lost data.
- mustache_kimono 4y ago
  > and never lost data. How do you know? I've seen a transient error due to a misconfiguration of ALPM that XFS/ext4/etc probably would have never caught, but ZFS did.
  - aborsy 4y ago
    Because a good part of it is checksumed with restic.
    - mustache_kimono 4y ago
      I don't mean to be critical of you, restic, or your backup strategies, but it doesn't seem like you know. ZFS is the I really need to know filesystem. I think it's pretty great and I use it where I can. But if it's not for you, it's not for you. And FYI, an power supply failure does not manifest the same as a power outage.
- 4y ago

bubblethink 4y ago

>One thing I didn't understand conceptually at first is that ZFS is standalone and not dependent on the host OS. This is true for every mature/production file system. You can do the same with mdadm, ext4, xfs, btrfs, etc. The only constraint would be with versions in that it would be a one way street. Can't necessarily go from something new to something old, but other way round is fine.

neilpanchal 4y ago
Right, what was not clear to me was that raid configuration and snapshots are also part of the file system. Usually, that's done through hardware cards or software raid where the configuration sits not on the file system (?) (Intel VROC, vSAN, etc), at least that was my wrong conceptual model. I used to make multiple copies of the USB stick of FreeNAS 8 back in the day because I didn't want the USB drive to fail and not knowing how to recover the zpool. Messing around with the file system directly cleared up everything.
- viraptor 4y ago
  That's more common for Linux storage. Whether you're using zfs, btrfs, or lvm, all the configuration required to read it is stored in the header somewhere rather than in a detached configuration.
  - vetinari 4y ago
    It has to be stored out of the band, otherwise you would have chicken-and-egg problem: what's the shape of the array, when the info is in a file stored inside the array?
    - viraptor 4y ago
      It's not stored in a file inside the array of course, but there are two types of out of band: next to the data, or completely disconnected. For example you only need to mount a single btrfs partition - in the header it already contains the information about other copies and will mount the whole raid setup as necessary. It doesn't matter if you move the drives somewhere else to a new system - mount one of them and it works. On the other hand, if you move drives from a hardware raid and put in new drives, some (all?) controllers will read the raid config from memory and offer to build the same raid on the new drives. That's completely-out-of-band. Depending on the controller, even changing the order the disks are plugged in can give you weird results.

yjftsjthsd-h 4y ago

Oh, it's better than that! If you're crazy enough, you can even multiboot multiple OSs on the same pool:) You do need to ensure that the pool features are sufficiently compatible between ZFS drivers (ex. you can't create a fully-featured pool with OpenZFS 2.1.6 on Linux and import it on OpenIndiana, last I checked), but it does work, empirically;)

neilpanchal 4y ago
A friend of mine suggested this and I didn't like the idea of putting the OS on there especially that it is a virtual machine. There are a couple of things that made this setup nice though: PCIe bifurcation (4x4x4x4) lanes straight to the CPU and PCIe passthrough in ESXi. From what I read that ZFS needs direct access to the underlying hardware. So, I just treat the OS as totally disposable and not worth backing up. I can spin up a VM in less than 2 mins if something were to happen to it.
- chasil 4y ago
  SmartOS is literally disposable - it runs off a flash drive. Have you considered it for your use case?

phpisthebest 4y ago

>no need for FreeNAS or any of that stuf I would encourage anyone that does not have an system admin background to use FreeNAS(truenas now) or something Their are default security and maintenance things (like periodic scrubs of the pool) that these nas operating systems set up by default

barrkel 4y ago
ZoL on Ubuntu adds a cronjob for zfs scrub; shipped in xenial (16.04).

deleted 4y ago

[deleted]

jiveturkey 4y ago

> standalone and not dependent on the host OS. ?? The OS needs to implement zfs, like any other filesystem/host combination. Any filesystem can be "moved" to another host simply by attaching the drives, assuming you have hardware level compat and filesystem level compat. OK the exception is when you have host-based hardware level encryption (ie, key in TPM or other security chip). In that case, zfs and otherwise, you can't just move the drives.