4 ms·

I like Promox a lot, but I wish it had an equivalent to VMware's VMFS. The last time I tried, there wasn't a way to use shared storage (i.e., iscsi block device

I like Promox a lot, but I wish it had an equivalent to VMware's VMFS. The last time I tried, there wasn't a way to use shared storage (i.e., iscsi block devices) across multiple nodes and have a failover of VMs that use that storage. And by failover I mean moving a VM to another host and booting it there, not even keeping the VM running.

SlavikCA 1y ago
Proxmox has built-in support for CEPH, which is promoted as VMFS equivalent. I don't have much experience with them, so can't tell if it's really on the same level.
- thyristan 1y ago
  Proxmox with Ceph can do failover when a node fails. You can configure a VM as high-availability to automatically make it boot on a leftover node after a crash: https://pve.proxmox.com/wiki/High_Availability https://pve.proxmox.com/wiki/High_Availability . When you add ProxLB, you can also automatically load-balance those VMs. One advantage Ceph has over VMware is that you don't need specially approved hardware to run it. Just use any old disks/SSDs/controllers. No special extra expensive vSAN hardware. But I cannot give you a full comparison, because I don't know all of VMware that well.
woleium 1y ago
Yes, you can do this with ceph on commodity hardware (or even your compute nodes, if you are brave), or if you have a bit of cash, something like a netapp to do NFS/iSCSI/NVME-oF Use any of these with the built in HA manager in proxmox
- redundantly 1y ago
  As far as I understand it, Ceph allows you to create distributed storage by using the hardware across your hosts. Can it be used to format a single shared block device that is accessed by multiple hosts like VMFS does? My understanding is this isn't possible.
  - nyrikki 1y ago
    Ceph RBD can technically support multi-writer access through exclusive locking, but it won't be the same as multi-writer. You can set up a radosgw outside of the proxmox and use objects. But ceph is fundamentally a distributed object store, shared LUNs with block level multi-writer is fundamentally a tightly coupled solution. If you have a legacy need that has OCFS or a quorum drive, the underlying tools proxmox is an abstraction can sometimes be used as these types of systems tend to be pets. But if you were just using multi-writer because it was there, there are alternatives that are typically more robust under the shared nothing model like Ceph uses. But it is all tradeoffs and horses for courses.
    - throw0101d 1y ago
      > But ceph is fundamentally a distributed object store, shared LUNs with block level multi-writer is fundamentally a tightly coupled solution. There is Ceph and then there is CephFS, "a POSIX-compliant file system built on top of Ceph’s distributed object store, RADOS": * https://docs.ceph.com/en/latest/cephfs/ https://docs.ceph.com/en/latest/cephfs/ * https://pve.proxmox.com/wiki/Storage:_CephFS https://pve.proxmox.com/wiki/Storage:_CephFS
  - guerby 1y ago
    From what I read about VMFS yes ceph allows you to have a shared block devices (RBD = rados block devices) across a cluster, and proxmox VE HA makes sure only one instance of the VM is active on the cluster to avoid having multiple writers to the same disk image.
aaronius 1y ago
That should have been possible for a while. Get the block storage to the node (FC or configure iSCSI), configure multipathing in most situations, and then configure LVM (thick) on top and mark it as shared. One nice thing this release brings is the option to finally also have snapshots for such a shared storage.
- redundantly 1y ago
  I tried that, but had two problems: When migrating a VM from one host to another it would require cloning the LVM volume, rather than just importing the group on the other node and starting the VM up. I have existing VMware gusts that I'd like to migrate over in bulk. This would be easy enough to do by converting the VMDK files, but using LVM means creating an LVM group for each VM and importing the contents of the VMDK into the LV.
  - aaronius 1y ago
    Hmm, staying with iSCSI. You should create one large LUN that is available on each node. Then it is important to mark the LVM as "shared". This way, PVE knows that all nodes access the same LVM, so copying the disk images is not necessary on a live migration. With such a setup, PVE will create LVs on the same VG for each disk image. So no handling of multiple VGs or LUNs is necessary. The multipathing PVE wiki page lines out the whole process: https://pve.proxmox.com/wiki/Multipath https://pve.proxmox.com/wiki/Multipath
pdntspa 1y ago
That, and configuring mount points for read-write access on the host is incredibly confusing and needlessly painful
keeperofdakeys 1y ago
Unfortunately clustered storage is just a hard problem, and there is a lack of good implementations. OCFS2 and GFS2 exist, but IIRC there are challenges for using them for VM storage, especially for snapshots. Proxmox 9 added a new feature to use multiple QCOW2 files as a volume chain, which may improve this, but for now that's only used for LVM. (Making Proxmox 9 much more viable on a shared iSCSI/FC LUN). If your requirements are flexible Proxmox does have one nice alternative though - local ZFS + scheduled replication. This feature performs ZFS snapshots + ZFS send every few minutes, giving you snapshots on your other nodes. This snapshot can be used for manual HA, auto HA, and even for fast live migration. Not great for databases, but a decent alternative for homelab and small business.

SlavikCA 1y ago

Proxmox has built-in support for CEPH, which is promoted as VMFS equivalent. I don't have much experience with them, so can't tell if it's really on the same level.

thyristan 1y ago
Proxmox with Ceph can do failover when a node fails. You can configure a VM as high-availability to automatically make it boot on a leftover node after a crash: https://pve.proxmox.com/wiki/High_Availability https://pve.proxmox.com/wiki/High_Availability . When you add ProxLB, you can also automatically load-balance those VMs. One advantage Ceph has over VMware is that you don't need specially approved hardware to run it. Just use any old disks/SSDs/controllers. No special extra expensive vSAN hardware. But I cannot give you a full comparison, because I don't know all of VMware that well.

woleium 1y ago

Yes, you can do this with ceph on commodity hardware (or even your compute nodes, if you are brave), or if you have a bit of cash, something like a netapp to do NFS/iSCSI/NVME-oF Use any of these with the built in HA manager in proxmox

redundantly 1y ago
As far as I understand it, Ceph allows you to create distributed storage by using the hardware across your hosts. Can it be used to format a single shared block device that is accessed by multiple hosts like VMFS does? My understanding is this isn't possible.
- nyrikki 1y ago
  Ceph RBD can technically support multi-writer access through exclusive locking, but it won't be the same as multi-writer. You can set up a radosgw outside of the proxmox and use objects. But ceph is fundamentally a distributed object store, shared LUNs with block level multi-writer is fundamentally a tightly coupled solution. If you have a legacy need that has OCFS or a quorum drive, the underlying tools proxmox is an abstraction can sometimes be used as these types of systems tend to be pets. But if you were just using multi-writer because it was there, there are alternatives that are typically more robust under the shared nothing model like Ceph uses. But it is all tradeoffs and horses for courses.
  - throw0101d 1y ago
    > But ceph is fundamentally a distributed object store, shared LUNs with block level multi-writer is fundamentally a tightly coupled solution. There is Ceph and then there is CephFS, "a POSIX-compliant file system built on top of Ceph’s distributed object store, RADOS": * https://docs.ceph.com/en/latest/cephfs/ https://docs.ceph.com/en/latest/cephfs/ * https://pve.proxmox.com/wiki/Storage:_CephFS https://pve.proxmox.com/wiki/Storage:_CephFS
- guerby 1y ago
  From what I read about VMFS yes ceph allows you to have a shared block devices (RBD = rados block devices) across a cluster, and proxmox VE HA makes sure only one instance of the VM is active on the cluster to avoid having multiple writers to the same disk image.

aaronius 1y ago

That should have been possible for a while. Get the block storage to the node (FC or configure iSCSI), configure multipathing in most situations, and then configure LVM (thick) on top and mark it as shared. One nice thing this release brings is the option to finally also have snapshots for such a shared storage.

redundantly 1y ago
I tried that, but had two problems: When migrating a VM from one host to another it would require cloning the LVM volume, rather than just importing the group on the other node and starting the VM up. I have existing VMware gusts that I'd like to migrate over in bulk. This would be easy enough to do by converting the VMDK files, but using LVM means creating an LVM group for each VM and importing the contents of the VMDK into the LV.
- aaronius 1y ago
  Hmm, staying with iSCSI. You should create one large LUN that is available on each node. Then it is important to mark the LVM as "shared". This way, PVE knows that all nodes access the same LVM, so copying the disk images is not necessary on a live migration. With such a setup, PVE will create LVs on the same VG for each disk image. So no handling of multiple VGs or LUNs is necessary. The multipathing PVE wiki page lines out the whole process: https://pve.proxmox.com/wiki/Multipath https://pve.proxmox.com/wiki/Multipath

pdntspa 1y ago

That, and configuring mount points for read-write access on the host is incredibly confusing and needlessly painful

keeperofdakeys 1y ago

Unfortunately clustered storage is just a hard problem, and there is a lack of good implementations. OCFS2 and GFS2 exist, but IIRC there are challenges for using them for VM storage, especially for snapshots. Proxmox 9 added a new feature to use multiple QCOW2 files as a volume chain, which may improve this, but for now that's only used for LVM. (Making Proxmox 9 much more viable on a shared iSCSI/FC LUN). If your requirements are flexible Proxmox does have one nice alternative though - local ZFS + scheduled replication. This feature performs ZFS snapshots + ZFS send every few minutes, giving you snapshots on your other nodes. This snapshot can be used for manual HA, auto HA, and even for fast live migration. Not great for databases, but a decent alternative for homelab and small business.