3 ms·

With your clustered Ceph Promox, does Ceph there provide the storage to your VM's running in Promox? And if so, how resilient are the running VMs to the (tempo

With your clustered Ceph Promox, does Ceph there provide the storage to your VM's running in Promox? And if so, how resilient are the running VMs to the (temporary) loss of a Promox node? Asking just because I'm trying out some clustered Ceph Promox stuff currently, and with 3 nodes (so far) the VMs aren't coping well the loss of a node. They seem to either freeze, fault or hard reset (after a while) when it occurs. I'm going to try with 5 nodes next and see how it goes. Am new to both Ceph and Proxmox, so it could also be something to do with my current newbie-ness. :)

sgarland 3y ago
At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ceph on the NVMe drives on its nodes, and then exposes the pool over RBD to VMs when you add a disk. In that regard, it’s very much like EBS from the perspective of the VM. Nothing stops me from shifting to Ceph for existing workloads, but I have it in my head that I need to shift the K8s hosting to Talos first. I have used it successfully to run MySQL and Postgres instances FWIW. Never had any issues with either of those, even with abrupt VM shutdowns. A VM, if set up as HA, can be live-migrated to a different node to perform maintenance with zero downtime. If the node just goes away, of course, you can’t do much about that. I confess I haven’t used Ceph heavily while dropping a node, so unfortunately I can’t give you a good answer on that behavior, other than ensuring that you have adequate bandwidth for both Ceph and Proxmox. I know they both say 10G networks are preferable. FWIW I have a completely separate switch (UniFi Flex Mini) for Proxmox’s corosync; in theory the main switch would be fine, but for an extra $30 I had more peace of mind.
- justinclift 3y ago
  Thanks, that's all good info. Yeah, the live migration bit is a useful looking feature. I'm just investigating ways to reduce single points of failure. :) Live migration is working fine with storage off-the-cluster. eg local iSCSI server sharing storage to the nodes. But then the iSCSI server is the failure point. Almost done setting up the 5 node cluster, so will test that shortly. Then probably need to read a bunch more docs. ;D
  - justinclift 3y ago
    The 5 node cluster with Ceph on all the nodes is working "ok" (resilience wise) to loosing a node. The VMs keep on going. Unfortunately, it's not a complete win though. Doing a live migration of a VM between nodes actually hung (locking the VM completely), and needed the VM to be restarted then the migration re-run (which then worked). The logs for the failed migration were completely useless. :( So, a 5 node setup seems like it might be a better way forward, but things like migration hangs need eliminating first. More experimentation and learning coming up. :)
    - sgarland 3y ago
      Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.
      - justinclift 3y ago
        Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
        justinclift 3y ago
        And another migration has just gotten stuck. This time of the Debian 12 installer running. Nothing is looking obviously weird in the output log: 2023-12-24 10:05:33 starting migration of VM 101 to node 'proxmox4' (10.1.1.101) 2023-12-24 10:05:33 starting VM 101 on remote node 'proxmox4' 2023-12-24 10:05:36 start remote tunnel 2023-12-24 10:05:38 ssh tunnel ver 1 2023-12-24 10:05:38 starting online/live migration on unix:/run/qemu-server/101.migrate 2023-12-24 10:05:38 set migration capabilities 2023-12-24 10:05:38 migration downtime limit: 100 ms 2023-12-24 10:05:38 migration cachesize: 256.0 MiB 2023-12-24 10:05:38 set migration parameters 2023-12-24 10:05:38 start migrate command to unix:/run/qemu-server/101.migrate 2023-12-24 10:05:39 migration active, transferred 58.6 MiB of 2.0 GiB VM-state, 101.7 MiB/s 2023-12-24 10:05:40 migration active, transferred 131.4 MiB of 2.0 GiB VM-state, 92.7 MiB/s 2023-12-24 10:05:41 migration active, transferred 233.6 MiB of 2.0 GiB VM-state, 96.8 MiB/s 2023-12-24 10:05:42 migration active, transferred 341.5 MiB of 2.0 GiB VM-state, 108.9 MiB/s 2023-12-24 10:05:43 migration active, transferred 457.1 MiB of 2.0 GiB VM-state, 107.5 MiB/s 2023-12-24 10:05:44 migration active, transferred 557.6 MiB of 2.0 GiB VM-state, 108.2 MiB/s 2023-12-24 10:05:45 migration active, transferred 661.4 MiB of 2.0 GiB VM-state, 113.3 MiB/s 2023-12-24 10:05:46 migration active, transferred 773.7 MiB of 2.0 GiB VM-state, 57.4 MiB/s 2023-12-24 10:05:47 migration active, transferred 851.0 MiB of 2.0 GiB VM-state, 127.4 MiB/s 2023-12-24 10:05:48 migration active, transferred 973.5 MiB of 2.0 GiB VM-state, 141.0 MiB/s 2023-12-24 10:05:49 migration active, transferred 1.1 GiB of 2.0 GiB VM-state, 119.3 MiB/s 2023-12-24 10:05:50 migration active, transferred 1.2 GiB of 2.0 GiB VM-state, 113.5 MiB/s 2023-12-24 10:05:51 migration active, transferred 1.3 GiB of 2.0 GiB VM-state, 142.3 MiB/s 2023-12-24 10:05:52 migration active, transferred 1.4 GiB of 2.0 GiB VM-state, 122.8 MiB/s 2023-12-24 10:05:53 migration active, transferred 1.5 GiB of 2.0 GiB VM-state, 174.4 MiB/s 2023-12-24 10:05:54 migration active, transferred 1.7 GiB of 2.0 GiB VM-state, 141.7 MiB/s 2023-12-24 10:05:55 migration active, transferred 1.8 GiB of 2.0 GiB VM-state, 102.6 MiB/s 2023-12-24 10:05:56 migration active, transferred 1.9 GiB of 2.0 GiB VM-state, 121.8 MiB/s 2023-12-24 10:05:57 migration active, transferred 2.1 GiB of 2.0 GiB VM-state, 179.7 MiB/s It got to that point and stopped. It's been 10 minutes. The VM itself is completely frozen meanwhile. That sucks. :(

sgarland 3y ago

At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ceph on the NVMe drives on its nodes, and then exposes the pool over RBD to VMs when you add a disk. In that regard, it’s very much like EBS from the perspective of the VM. Nothing stops me from shifting to Ceph for existing workloads, but I have it in my head that I need to shift the K8s hosting to Talos first. I have used it successfully to run MySQL and Postgres instances FWIW. Never had any issues with either of those, even with abrupt VM shutdowns. A VM, if set up as HA, can be live-migrated to a different node to perform maintenance with zero downtime. If the node just goes away, of course, you can’t do much about that. I confess I haven’t used Ceph heavily while dropping a node, so unfortunately I can’t give you a good answer on that behavior, other than ensuring that you have adequate bandwidth for both Ceph and Proxmox. I know they both say 10G networks are preferable. FWIW I have a completely separate switch (UniFi Flex Mini) for Proxmox’s corosync; in theory the main switch would be fine, but for an extra $30 I had more peace of mind.

justinclift 3y ago
Thanks, that's all good info. Yeah, the live migration bit is a useful looking feature. I'm just investigating ways to reduce single points of failure. :) Live migration is working fine with storage off-the-cluster. eg local iSCSI server sharing storage to the nodes. But then the iSCSI server is the failure point. Almost done setting up the 5 node cluster, so will test that shortly. Then probably need to read a bunch more docs. ;D
- justinclift 3y ago
  The 5 node cluster with Ceph on all the nodes is working "ok" (resilience wise) to loosing a node. The VMs keep on going. Unfortunately, it's not a complete win though. Doing a live migration of a VM between nodes actually hung (locking the VM completely), and needed the VM to be restarted then the migration re-run (which then worked). The logs for the failed migration were completely useless. :( So, a 5 node setup seems like it might be a better way forward, but things like migration hangs need eliminating first. More experimentation and learning coming up. :)
  - sgarland 3y ago
    Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.
    - justinclift 3y ago
      Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
      - justinclift 3y ago
        And another migration has just gotten stuck. This time of the Debian 12 installer running. Nothing is looking obviously weird in the output log: 2023-12-24 10:05:33 starting migration of VM 101 to node 'proxmox4' (10.1.1.101) 2023-12-24 10:05:33 starting VM 101 on remote node 'proxmox4' 2023-12-24 10:05:36 start remote tunnel 2023-12-24 10:05:38 ssh tunnel ver 1 2023-12-24 10:05:38 starting online/live migration on unix:/run/qemu-server/101.migrate 2023-12-24 10:05:38 set migration capabilities 2023-12-24 10:05:38 migration downtime limit: 100 ms 2023-12-24 10:05:38 migration cachesize: 256.0 MiB 2023-12-24 10:05:38 set migration parameters 2023-12-24 10:05:38 start migrate command to unix:/run/qemu-server/101.migrate 2023-12-24 10:05:39 migration active, transferred 58.6 MiB of 2.0 GiB VM-state, 101.7 MiB/s 2023-12-24 10:05:40 migration active, transferred 131.4 MiB of 2.0 GiB VM-state, 92.7 MiB/s 2023-12-24 10:05:41 migration active, transferred 233.6 MiB of 2.0 GiB VM-state, 96.8 MiB/s 2023-12-24 10:05:42 migration active, transferred 341.5 MiB of 2.0 GiB VM-state, 108.9 MiB/s 2023-12-24 10:05:43 migration active, transferred 457.1 MiB of 2.0 GiB VM-state, 107.5 MiB/s 2023-12-24 10:05:44 migration active, transferred 557.6 MiB of 2.0 GiB VM-state, 108.2 MiB/s 2023-12-24 10:05:45 migration active, transferred 661.4 MiB of 2.0 GiB VM-state, 113.3 MiB/s 2023-12-24 10:05:46 migration active, transferred 773.7 MiB of 2.0 GiB VM-state, 57.4 MiB/s 2023-12-24 10:05:47 migration active, transferred 851.0 MiB of 2.0 GiB VM-state, 127.4 MiB/s 2023-12-24 10:05:48 migration active, transferred 973.5 MiB of 2.0 GiB VM-state, 141.0 MiB/s 2023-12-24 10:05:49 migration active, transferred 1.1 GiB of 2.0 GiB VM-state, 119.3 MiB/s 2023-12-24 10:05:50 migration active, transferred 1.2 GiB of 2.0 GiB VM-state, 113.5 MiB/s 2023-12-24 10:05:51 migration active, transferred 1.3 GiB of 2.0 GiB VM-state, 142.3 MiB/s 2023-12-24 10:05:52 migration active, transferred 1.4 GiB of 2.0 GiB VM-state, 122.8 MiB/s 2023-12-24 10:05:53 migration active, transferred 1.5 GiB of 2.0 GiB VM-state, 174.4 MiB/s 2023-12-24 10:05:54 migration active, transferred 1.7 GiB of 2.0 GiB VM-state, 141.7 MiB/s 2023-12-24 10:05:55 migration active, transferred 1.8 GiB of 2.0 GiB VM-state, 102.6 MiB/s 2023-12-24 10:05:56 migration active, transferred 1.9 GiB of 2.0 GiB VM-state, 121.8 MiB/s 2023-12-24 10:05:57 migration active, transferred 2.1 GiB of 2.0 GiB VM-state, 179.7 MiB/s It got to that point and stopped. It's been 10 minutes. The VM itself is completely frozen meanwhile. That sucks. :(