4 ms·

I've been working with virtual machines in production and development for well over a decade now and a lot of what works via ZFS in this article works very simi

I've been working with virtual machines in production and development for well over a decade now and a lot of what works via ZFS in this article works very similarly to how it would work we used AWS or VMware style snapshots. In fact, ZFS snapshots are only so helpful if you make complicated changes across multiple ZFS volumes, for example, that require some more transactional style rollbacks. In such scenarios, it may be easier to just perform an instance-level rollback. One problem that didn't really occur to me before I became more ops-side was that administrators would want to heavily restrict / remove users' (read: developers and even other sysadmins) abilities to create snapshots in the first place. Why would you remove self-service temporary backups and avoid a lot of backup restoration requests? I didn't realize that other engineers could be careless and keep dozens or even hundreds of snapshots over time that gobble up expensive storage resources (SANs are not cheap regardless of manufacturer) and slow down I/O transactions over time. That was why so many of my customers deploying stuff like VMware Lab Manager and vCloud Director demanded the ability to remove access to snapshot features. As a result of the typical usage where user abuse of a very powerful feature threw things for a loop, the typical organizational structure and siloization of these environments means that nowadays SAN-side LUN snapshots are used more often than from the VM layer (the same administrators that manage VMware environments typically have rights to the SANs). Using ZFS like this is a developer-side reaction to me, but duplication of trying to solve the same problem when technical solutions have existed and are viable is exasperating.

drvdevd 10y ago
Actually, perhaps this doesn't address all your points, but I share the sentiment about VM snapshots "feeling" quite similar, in fact I experience it every day as I run ZFS as a root filesystem within VMware. The real gain for me is that all of these tools are in-band with my working OS. If I were provided a VM(s) on AWS or some SAN by my employer, that could in theory give us a mutual benefit. Ops could avoid giving me access via IAM or whatever and expanding beyond some set limit, but allow me to perform almost identictal "ops like" tasks using ZFS (which I also find to be fairly user friendly). Incidentally, I'm also a fan of nested virtualization and using git on top of ZFS... I believe all of these tools complement each other nicely as opposed to being mere repetitions.
tw04 10y ago
VMware snapshots are NOT like ZFS snapshots. VMWare snapshots are extremely heavy an will have massive performance impact if left around for an extended period of time for any I/O sensitive VM. ZFS snapshots are essentially 0-overhead from a performance perspective, and function in a very different manner.
- devonkim 10y ago
  I'm aware of how both work (I had a VCP410 cert and happily run ZFS with many snapshots on rotation at home), but IMO the uses are similar enough for the purposes explored in the article despite the significant differences in practice and felt that a traditional alternative should be mentioned. Additionally, VMware-based snapshots can be performed SAN-side and integrated with VAAI that can expect similar performance overhead as a ZFS snapshot and still support a lot of options with trade-offs and features. The compulsory indirections that happen in the traditional VMware snapshot implementation with the on-disk block map appears designed for storage compatibility (much like the design of VMFS in itself, including with that whole file-based mutex system) primarily rather than performance not to mention the support for memory persistence / quiescing with the guest OS, which ZFS snapshots do not support. In contrast, ZFS has all the metadata to quickly examine block time or other rich metadata to accelerate read and write indirection of blocks optimally. It's not clear if a SAN can accelerate ZFS using OSes in the way that they can accelerate VMware storage but I'm curious of the potential. On the other hand, recursive snapshots in ZFS are really slick and can achieve 90%+ of what people typically want from VMware-based snapshots when it comes to any non-Windows OS if there's a little room for not needing to care about indirect changes required in rollbacks.
  - tw04 10y ago
    ...just no. VAAI snapshot integration with VMWare is only for NAS platforms, not SAN - VMFS doesn't even come into the picture. Not to mention it's intended for View for rapid cloning, and not as a backup feature.

drvdevd 10y ago

Actually, perhaps this doesn't address all your points, but I share the sentiment about VM snapshots "feeling" quite similar, in fact I experience it every day as I run ZFS as a root filesystem within VMware. The real gain for me is that all of these tools are in-band with my working OS. If I were provided a VM(s) on AWS or some SAN by my employer, that could in theory give us a mutual benefit. Ops could avoid giving me access via IAM or whatever and expanding beyond some set limit, but allow me to perform almost identictal "ops like" tasks using ZFS (which I also find to be fairly user friendly). Incidentally, I'm also a fan of nested virtualization and using git on top of ZFS... I believe all of these tools complement each other nicely as opposed to being mere repetitions.

tw04 10y ago

VMware snapshots are NOT like ZFS snapshots. VMWare snapshots are extremely heavy an will have massive performance impact if left around for an extended period of time for any I/O sensitive VM. ZFS snapshots are essentially 0-overhead from a performance perspective, and function in a very different manner.

devonkim 10y ago
I'm aware of how both work (I had a VCP410 cert and happily run ZFS with many snapshots on rotation at home), but IMO the uses are similar enough for the purposes explored in the article despite the significant differences in practice and felt that a traditional alternative should be mentioned. Additionally, VMware-based snapshots can be performed SAN-side and integrated with VAAI that can expect similar performance overhead as a ZFS snapshot and still support a lot of options with trade-offs and features. The compulsory indirections that happen in the traditional VMware snapshot implementation with the on-disk block map appears designed for storage compatibility (much like the design of VMFS in itself, including with that whole file-based mutex system) primarily rather than performance not to mention the support for memory persistence / quiescing with the guest OS, which ZFS snapshots do not support. In contrast, ZFS has all the metadata to quickly examine block time or other rich metadata to accelerate read and write indirection of blocks optimally. It's not clear if a SAN can accelerate ZFS using OSes in the way that they can accelerate VMware storage but I'm curious of the potential. On the other hand, recursive snapshots in ZFS are really slick and can achieve 90%+ of what people typically want from VMware-based snapshots when it comes to any non-Windows OS if there's a little room for not needing to care about indirect changes required in rollbacks.
- tw04 10y ago
  ...just no. VAAI snapshot integration with VMWare is only for NAS platforms, not SAN - VMFS doesn't even come into the picture. Not to mention it's intended for View for rapid cloning, and not as a backup feature.