5 ms·

I'd recommend not using ubuntu, but instead gp with either Proxmox or bog standard debian. Proxmox supports using the zfs pool as storage for vms and using zfs

I'd recommend not using ubuntu, but instead gp with either Proxmox or bog standard debian. Proxmox supports using the zfs pool as storage for vms and using zfs as root disk also. Debian won't help you do the install with zfs on root but it can be done. If you don't go with zfs on root then it's as simple as apt insall zfs-dkms on debian.

voytec 3y ago
Recently tried Proxmox, spent few hours on trying to boot any VM placed on ZFS pool but with no luck. Same VMs converted from raw to vmdk (iirc) worked on 2 other, identical NVMe disks with software ext4 raid.
- simcop2387 3y ago
  Very odd, I'd love to know more since I've got that setup going on my router/edge box right now. ZFS mirror on two NVMe disks as the storage for root fs and VMs.
  - voytec 3y ago
    It was a recent Proxmox version (newest ~6 weeks ago). It's possible that I used some experimental new kernel version and haven't tried on completely basic/default setup. The disks were Samsung's MZVL21T0HCLR-00B00. The pair with ext4 sw raid booted all VMs (Linux, FreeBSD) with no issues but second pair with ZFS pool wouldn't. It may have been due to the newer kernel version but I asked myself a few questions about what's missing in FreeBSD's bhyve regarding my needs and took this route instead of trying default kernel. # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT VMs 952G 828G 124G - - 36% 87% 1.00x ONLINE - zroot 944G 1.21G 943G - - 0% 0% 1.00x ONLINE -
donmcronald 3y ago
I tried Proxmox. I mainly need a storage server that can run a few VMs. Proxmox looks pretty good for a VM heavy setup, but it adds a lot of complexity that I just don't need. I like the release cadence on Ubuntu and I had such a bad experience with ZFS DKMS (many) years ago that I'm probably a bit bias towards Ubuntu because I simply don't have to think about it. I won't use ZFS on root unless it simplifies dealing with a failed boot disk. I still prefer CSM over UEFI because UEFI makes it more complicated to recover from failure :-(
dark-star 3y ago
> If you don't go with zfs on root then it's as simple as apt insall zfs-dkms on debian That's basically how it works in almost all Linux distros ;-)
squarefoot 3y ago
I never used Proxmox, so after having read great comments about it tried to install it on a couple small mini PCs I had around just to test some functionalities, but every time it failed. If memory serves (was like 1-2 months ago) on both machines the installer complained it couldn't find the partition to install. Both machines would install and run other Linux distros flawlessly. No idea how to solve that problem.

voytec 3y ago

Recently tried Proxmox, spent few hours on trying to boot any VM placed on ZFS pool but with no luck. Same VMs converted from raw to vmdk (iirc) worked on 2 other, identical NVMe disks with software ext4 raid.

simcop2387 3y ago
Very odd, I'd love to know more since I've got that setup going on my router/edge box right now. ZFS mirror on two NVMe disks as the storage for root fs and VMs.
- voytec 3y ago
  It was a recent Proxmox version (newest ~6 weeks ago). It's possible that I used some experimental new kernel version and haven't tried on completely basic/default setup. The disks were Samsung's MZVL21T0HCLR-00B00. The pair with ext4 sw raid booted all VMs (Linux, FreeBSD) with no issues but second pair with ZFS pool wouldn't. It may have been due to the newer kernel version but I asked myself a few questions about what's missing in FreeBSD's bhyve regarding my needs and took this route instead of trying default kernel. # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT VMs 952G 828G 124G - - 36% 87% 1.00x ONLINE - zroot 944G 1.21G 943G - - 0% 0% 1.00x ONLINE -

donmcronald 3y ago

I tried Proxmox. I mainly need a storage server that can run a few VMs. Proxmox looks pretty good for a VM heavy setup, but it adds a lot of complexity that I just don't need. I like the release cadence on Ubuntu and I had such a bad experience with ZFS DKMS (many) years ago that I'm probably a bit bias towards Ubuntu because I simply don't have to think about it. I won't use ZFS on root unless it simplifies dealing with a failed boot disk. I still prefer CSM over UEFI because UEFI makes it more complicated to recover from failure :-(

dark-star 3y ago

> If you don't go with zfs on root then it's as simple as apt insall zfs-dkms on debian That's basically how it works in almost all Linux distros ;-)

squarefoot 3y ago

I never used Proxmox, so after having read great comments about it tried to install it on a couple small mini PCs I had around just to test some functionalities, but every time it failed. If memory serves (was like 1-2 months ago) on both machines the installer complained it couldn't find the partition to install. Both machines would install and run other Linux distros flawlessly. No idea how to solve that problem.