5 ms·

Well you're citing netapp, zfs, freeebsd and debian. Those are all top notch. In all honesty I can't understand why would anyone use anything other than zfs no

by znpy 6y ago

Well you're citing netapp, zfs, freeebsd and debian. Those are all top notch. In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data.

vaylian 6y ago
Maybe my understanding of ZFS is wrong, but what about storage requirements? As far as I understand ZFS, you take snapshots of all versions of all files and that can quickly lead to a lot of space usage. But maybe there is also an automatic cleanup tool that deletes old snapshots after some time?
- magicalhippo 6y ago
  You take snapshots when you want to take snapshots, and a snapshot allows you to view the data as it was at the time the snapshot was taken. Not entirely unlike a git commit. The snapshot refers to the storage blocks/records on disk, not files as such, an important distinction since ZFS can expose block storage (zvol) as well as a "regular" file system. Since ZFS is copy-on-write, the only storage you pay for with a snapshot are the blocks that have changed since the snapshot was taken (plus a little overhead). Thus for data that does not change much, a snapshot is almost "free". The blocks are reference counted. Once a snapshot is deleted, ZFS decreases the reference count of the blocks referenced by the snapshot. Any block with a refcount of zero is considered free and thus that space is reclaimed. This happens when the block has changed since the snapshot was taken and there were no other snapshots referencing that block. ZFS itself has no automatic deletion of old snapshots AFAIK, but there are tools built around ZFS that allow for periodic snapshotting and cleanup.
  - rakoo 6y ago
    Side-topic: the way you described how blocks and snapshots work is _exactly_ how git works, with references and blobs: as long as a blob can be reached by a reference (branch or tag), it will not be garbage-collected. Turns out a DAG, as is used in both ZFS and git, is a good data structure in many use cases.
- gruturo 6y ago
  Snapshots are something you control - they don't happen by themselves. Furthermore, you only "pay" their storage cost for data which actually changes. On a 2TB volume, last snapshotted 1 week ago, if only 5GB have changed since then, that's the only storage overhead (ignoring for simplicity the folder structure itself). Also, if a single data block changes 20 times since its last snapshot, you still only "pay" storage costs for its current version + the one sitting in the last snapshot.
- chungy 6y ago
  It's unfair you've been downvoted. This is an excellent question and the responses were likewise excellent. We need to encourage more of this :)
- fomine3 6y ago
  sanoid is popular tool for homelab user to scheduling snapshot creation/deletion.
traceroute66 6y ago
@znpy Why people aren't just using ZFS (or BTRFS) ? Well... Maybe because XFS is well proven in certain critical environments, especially ones which might involve frequent manipulations of large quantities of small files (think mailserver or database). Indeed, some developers will tell you in the docs that they only support XFS or ext4 and that you're on your own if you choose something else. Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations. Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation). Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc. Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might). Maybe because not everyone feels the need to "keep up with the Jonses". [1] https://www.youtube.com/watch?v=8YUC-r1aXAc https://www.youtube.com/watch?v=8YUC-r1aXAc
- lousken 6y ago
  if i remember correctly LVM snapshots were deemed to be lowering performance of a system if you had lots of them - is that still a thing?
  - post-factum 6y ago
    Thin LVM snapshots solve that.
- necheffa 6y ago
  Do people really create zpools within a VM for more than playing around? The best practice is to point a VM at a zvol the way you might use LVM on bare metal to back storage for VMs.
  - ianai 6y ago
    Right. The storage pool for VMs should be zfs (if you’re going to use zfs as an assumption) while the VMs should be something else unless you’re doing something specific to use zfs in the VM. Zfs is engineered to almost turn a system into a file system server in exclusion so use it for its strengths appropriately.
squarefoot 6y ago
> In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data. In datacenters many probably do, however most home/soho NAS machines are severely limited wrt memory and CPU power, which could be a limit. My self assembled NAS uses a Atom board to keep power requirements low since it stays always on, and I can't complain about its performance, however that CPU doesn't support more than 4GB RAM which is considered the minimum to properly use ZFS (NAS4free). Many cheap commercial NAS boxes used in small offices are even more limited. I wonder if there are any technical reasons preventing ZFS to operate (or be adapted to) in low memory environments.
- Wowfunhappy 6y ago
  I have used ZFS with 1 GB of RAM. It worked fine, it just wasn’t able to cache stuff to memory.
- amarshall 6y ago
  > 4GB RAM which is considered the minimum to properly use ZFS This is mostly a myth with origins in the high-memory requirements of ZFS deduplication (which few should use, anyway). Of course, more memory allows for more caching, but that’s true of any filesystem on a “modern” OS.
  - djsumdog 6y ago
    Yea I only have 4GB of ram on my little HP MicroServer I use as a NAS with ZFS. The only issues I seem to have is that ZFS encryption seems to use more CPU than luks encryption, but it's still not too bad (the AMD Turion(tm) II Neo on it doesn't have AES instructions). I might upgrade it to something with a better processor because I do want to run de-duplication on my 14TB drive at some point.
- linsomniac 6y ago
  I wonder if it would be useful to have a Optane for the memory-hungry portions of ZFS, particularly the dedup tables, especially to be able to run it on very small, efficient machines. My work backup server has "only" 32GB of RAM which is nowhere near enough to do dedup, even though there's a ton of stuff that could be deduplicated. Looks like it needs at least 64GB RAM to dedup, and that machine is maxed out. But at home I'm in a similar situation to the parent. I'd like to set up a NAS, but I use it mostly for backups long term storage and don't want a big machine sitting idle all the time. A tiny Atom with Optane for cache and DDT might be ideal, in my thinking. But for most of the storage use Backblaze is probably the right answer.
curt15 6y ago
Google still uses ext4 for its servers, while Facebook uses mostly btrfs nowadays.
- nix23 6y ago
  Google has on top of ext4 his GoogleFS, in theory you could even go 'without' a underlying FS, a bit like BlueStore from Ceph.
  - dannypgh 6y ago
    This does not sound right to me. If you're talking about GFS, that's been gone for a decade, and was not a filesystem as far as the kernel was concerned.
    - nix23 6y ago
      That's why i brought up the Ceph example, but counts for databases etc too, actually the early databases had all their own fs/data-structure on the platter.
retro64 6y ago
In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data. At a personal level, I opted out of it based on simplicity and familiarity. If I have a data drive where I need to move and mount in another box, I do not want to mess with the complexities of supporting ZFS to get at that data or the risk that the only OS I have available can't read it. Again, this is a non-commercial environment, but I consider my data important as well. I've decided to go the route of keeping multiple backups of my data spread across multiple drives. I've already experienced bitrot several times over the years, but for me, this approach is more practical than relying on ZFS.
- radiowave 6y ago
  I don't mean to be criticizing your choices, but I'll note here that if you're willing to avoid enabling the latest and greatest ZFS features, bringing up a machine that can mount your disks can be as simple as booting from Ubuntu install media, and having internet available so you can run "apt install zfsutils-linux". (And somebody's probably going to correct my ignorance and point out that beyond some release 'x', the ZFS stuff is already built-in.)
  - retro64 6y ago
    Is it that easy now? I actually was in this situation about 3 years ago and found I could not simply mount the ZFS drive - and I cannot recall exactly why. I think it had something to do with the pool…configuration(?), and I went down a rabbit hole trying to sort it out. Fortunately in this case the tradeoff in time was more valuable than the data (it was a system drive on a small SSD) so I had the option to ditch my efforts with little to lose. But, I did spend a handful of hours on it with no traction.
    - bpfrh 6y ago
      I was able to transfer a zfs pool from a linux server to a freebsd server with different hardware simply by moving the physical disks to the new server. For me ZFS has been rock solid, even with power cuts during running vms(linux and windows) and a scrub. I even managed to steal away enough ram from a server so that ZFS produced a stacktrace in dmesg, no data corruption, even on running vms.

by znpy 6y ago

Well you're citing netapp, zfs, freeebsd and debian. Those are all top notch. In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data.

vaylian 6y ago

Maybe my understanding of ZFS is wrong, but what about storage requirements? As far as I understand ZFS, you take snapshots of all versions of all files and that can quickly lead to a lot of space usage. But maybe there is also an automatic cleanup tool that deletes old snapshots after some time?

magicalhippo 6y ago
You take snapshots when you want to take snapshots, and a snapshot allows you to view the data as it was at the time the snapshot was taken. Not entirely unlike a git commit. The snapshot refers to the storage blocks/records on disk, not files as such, an important distinction since ZFS can expose block storage (zvol) as well as a "regular" file system. Since ZFS is copy-on-write, the only storage you pay for with a snapshot are the blocks that have changed since the snapshot was taken (plus a little overhead). Thus for data that does not change much, a snapshot is almost "free". The blocks are reference counted. Once a snapshot is deleted, ZFS decreases the reference count of the blocks referenced by the snapshot. Any block with a refcount of zero is considered free and thus that space is reclaimed. This happens when the block has changed since the snapshot was taken and there were no other snapshots referencing that block. ZFS itself has no automatic deletion of old snapshots AFAIK, but there are tools built around ZFS that allow for periodic snapshotting and cleanup.
- rakoo 6y ago
  Side-topic: the way you described how blocks and snapshots work is _exactly_ how git works, with references and blobs: as long as a blob can be reached by a reference (branch or tag), it will not be garbage-collected. Turns out a DAG, as is used in both ZFS and git, is a good data structure in many use cases.
gruturo 6y ago
Snapshots are something you control - they don't happen by themselves. Furthermore, you only "pay" their storage cost for data which actually changes. On a 2TB volume, last snapshotted 1 week ago, if only 5GB have changed since then, that's the only storage overhead (ignoring for simplicity the folder structure itself). Also, if a single data block changes 20 times since its last snapshot, you still only "pay" storage costs for its current version + the one sitting in the last snapshot.
chungy 6y ago
It's unfair you've been downvoted. This is an excellent question and the responses were likewise excellent. We need to encourage more of this :)
fomine3 6y ago
sanoid is popular tool for homelab user to scheduling snapshot creation/deletion.

traceroute66 6y ago

@znpy Why people aren't just using ZFS (or BTRFS) ? Well... Maybe because XFS is well proven in certain critical environments, especially ones which might involve frequent manipulations of large quantities of small files (think mailserver or database). Indeed, some developers will tell you in the docs that they only support XFS or ext4 and that you're on your own if you choose something else. Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations. Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation). Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc. Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might). Maybe because not everyone feels the need to "keep up with the Jonses". [1] https://www.youtube.com/watch?v=8YUC-r1aXAc https://www.youtube.com/watch?v=8YUC-r1aXAc

lousken 6y ago
if i remember correctly LVM snapshots were deemed to be lowering performance of a system if you had lots of them - is that still a thing?
- post-factum 6y ago
  Thin LVM snapshots solve that.
necheffa 6y ago
Do people really create zpools within a VM for more than playing around? The best practice is to point a VM at a zvol the way you might use LVM on bare metal to back storage for VMs.
- ianai 6y ago
  Right. The storage pool for VMs should be zfs (if you’re going to use zfs as an assumption) while the VMs should be something else unless you’re doing something specific to use zfs in the VM. Zfs is engineered to almost turn a system into a file system server in exclusion so use it for its strengths appropriately.

squarefoot 6y ago

> In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data. In datacenters many probably do, however most home/soho NAS machines are severely limited wrt memory and CPU power, which could be a limit. My self assembled NAS uses a Atom board to keep power requirements low since it stays always on, and I can't complain about its performance, however that CPU doesn't support more than 4GB RAM which is considered the minimum to properly use ZFS (NAS4free). Many cheap commercial NAS boxes used in small offices are even more limited. I wonder if there are any technical reasons preventing ZFS to operate (or be adapted to) in low memory environments.

Wowfunhappy 6y ago
I have used ZFS with 1 GB of RAM. It worked fine, it just wasn’t able to cache stuff to memory.
amarshall 6y ago
> 4GB RAM which is considered the minimum to properly use ZFS This is mostly a myth with origins in the high-memory requirements of ZFS deduplication (which few should use, anyway). Of course, more memory allows for more caching, but that’s true of any filesystem on a “modern” OS.
- djsumdog 6y ago
  Yea I only have 4GB of ram on my little HP MicroServer I use as a NAS with ZFS. The only issues I seem to have is that ZFS encryption seems to use more CPU than luks encryption, but it's still not too bad (the AMD Turion(tm) II Neo on it doesn't have AES instructions). I might upgrade it to something with a better processor because I do want to run de-duplication on my 14TB drive at some point.
linsomniac 6y ago
I wonder if it would be useful to have a Optane for the memory-hungry portions of ZFS, particularly the dedup tables, especially to be able to run it on very small, efficient machines. My work backup server has "only" 32GB of RAM which is nowhere near enough to do dedup, even though there's a ton of stuff that could be deduplicated. Looks like it needs at least 64GB RAM to dedup, and that machine is maxed out. But at home I'm in a similar situation to the parent. I'd like to set up a NAS, but I use it mostly for backups long term storage and don't want a big machine sitting idle all the time. A tiny Atom with Optane for cache and DDT might be ideal, in my thinking. But for most of the storage use Backblaze is probably the right answer.

curt15 6y ago

Google still uses ext4 for its servers, while Facebook uses mostly btrfs nowadays.

nix23 6y ago
Google has on top of ext4 his GoogleFS, in theory you could even go 'without' a underlying FS, a bit like BlueStore from Ceph.
- dannypgh 6y ago
  This does not sound right to me. If you're talking about GFS, that's been gone for a decade, and was not a filesystem as far as the kernel was concerned.
  - nix23 6y ago
    That's why i brought up the Ceph example, but counts for databases etc too, actually the early databases had all their own fs/data-structure on the platter.

retro64 6y ago

In all honesty I can't understand why would anyone use anything other than zfs nowadays for important data. At a personal level, I opted out of it based on simplicity and familiarity. If I have a data drive where I need to move and mount in another box, I do not want to mess with the complexities of supporting ZFS to get at that data or the risk that the only OS I have available can't read it. Again, this is a non-commercial environment, but I consider my data important as well. I've decided to go the route of keeping multiple backups of my data spread across multiple drives. I've already experienced bitrot several times over the years, but for me, this approach is more practical than relying on ZFS.

radiowave 6y ago
I don't mean to be criticizing your choices, but I'll note here that if you're willing to avoid enabling the latest and greatest ZFS features, bringing up a machine that can mount your disks can be as simple as booting from Ubuntu install media, and having internet available so you can run "apt install zfsutils-linux". (And somebody's probably going to correct my ignorance and point out that beyond some release 'x', the ZFS stuff is already built-in.)
- retro64 6y ago
  Is it that easy now? I actually was in this situation about 3 years ago and found I could not simply mount the ZFS drive - and I cannot recall exactly why. I think it had something to do with the pool…configuration(?), and I went down a rabbit hole trying to sort it out. Fortunately in this case the tradeoff in time was more valuable than the data (it was a system drive on a small SSD) so I had the option to ditch my efforts with little to lose. But, I did spend a handful of hours on it with no traction.
  - bpfrh 6y ago
    I was able to transfer a zfs pool from a linux server to a freebsd server with different hardware simply by moving the physical disks to the new server. For me ZFS has been rock solid, even with power cuts during running vms(linux and windows) and a scrub. I even managed to steal away enough ram from a server so that ZFS produced a stacktrace in dmesg, no data corruption, even on running vms.