5 ms·
At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ce
by sgarland 3y ago
At the moment, Ceph is a tangential storage solution only used for testing (the bulk of my pods use Longhorn), but yes, that’s what it does – Proxmox sets up Ceph on the NVMe drives on its nodes, and then exposes the pool over RBD to VMs when you add a disk. In that regard, it’s very much like EBS from the perspective of the VM.
Nothing stops me from shifting to Ceph for existing workloads, but I have it in my head that I need to shift the K8s hosting to Talos first. I have used it successfully to run MySQL and Postgres instances FWIW. Never had any issues with either of those, even with abrupt VM shutdowns.
A VM, if set up as HA, can be live-migrated to a different node to perform maintenance with zero downtime. If the node just goes away, of course, you can’t do much about that.
I confess I haven’t used Ceph heavily while dropping a node, so unfortunately I can’t give you a good answer on that behavior, other than ensuring that you have adequate bandwidth for both Ceph and Proxmox. I know they both say 10G networks are preferable. FWIW I have a completely separate switch (UniFi Flex Mini) for Proxmox’s corosync; in theory the main switch would be fine, but for an extra $30 I had more peace of mind.
- justinclift 3y agoThanks, that's all good info. Yeah, the live migration bit is a useful looking feature. I'm just investigating ways to reduce single points of failure. :) Live migration is working fine with storage off-the-cluster. eg local iSCSI server sharing storage to the nodes. But then the iSCSI server is the failure point. Almost done setting up the 5 node cluster, so will test that shortly. Then probably need to read a bunch more docs. ;D
- justinclift 3y agoThe 5 node cluster with Ceph on all the nodes is working "ok" (resilience wise) to loosing a node. The VMs keep on going. Unfortunately, it's not a complete win though. Doing a live migration of a VM between nodes actually hung (locking the VM completely), and needed the VM to be restarted then the migration re-run (which then worked). The logs for the failed migration were completely useless. :( So, a 5 node setup seems like it might be a better way forward, but things like migration hangs need eliminating first. More experimentation and learning coming up. :)
- sgarland 3y agoDo all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.
- justinclift 3y agoYeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
- justinclift 3y agoAnd another migration has just gotten stuck. This time of the Debian 12 installer running. Nothing is looking obviously weird in the output log: 2023-12-24 10:05:33 starting migration of VM 101 to node 'proxmox4' (10.1.1.101) 2023-12-24 10:05:33 starting VM 101 on remote node 'proxmox4' 2023-12-24 10:05:36 start remote tunnel 2023-12-24 10:05:38 ssh tunnel ver 1 2023-12-24 10:05:38 starting online/live migration on unix:/run/qemu-server/101.migrate 2023-12-24 10:05:38 set migration capabilities 2023-12-24 10:05:38 migration downtime limit: 100 ms 2023-12-24 10:05:38 migration cachesize: 256.0 MiB 2023-12-24 10:05:38 set migration parameters 2023-12-24 10:05:38 start migrate command to unix:/run/qemu-server/101.migrate 2023-12-24 10:05:39 migration active, transferred 58.6 MiB of 2.0 GiB VM-state, 101.7 MiB/s 2023-12-24 10:05:40 migration active, transferred 131.4 MiB of 2.0 GiB VM-state, 92.7 MiB/s 2023-12-24 10:05:41 migration active, transferred 233.6 MiB of 2.0 GiB VM-state, 96.8 MiB/s 2023-12-24 10:05:42 migration active, transferred 341.5 MiB of 2.0 GiB VM-state, 108.9 MiB/s 2023-12-24 10:05:43 migration active, transferred 457.1 MiB of 2.0 GiB VM-state, 107.5 MiB/s 2023-12-24 10:05:44 migration active, transferred 557.6 MiB of 2.0 GiB VM-state, 108.2 MiB/s 2023-12-24 10:05:45 migration active, transferred 661.4 MiB of 2.0 GiB VM-state, 113.3 MiB/s 2023-12-24 10:05:46 migration active, transferred 773.7 MiB of 2.0 GiB VM-state, 57.4 MiB/s 2023-12-24 10:05:47 migration active, transferred 851.0 MiB of 2.0 GiB VM-state, 127.4 MiB/s 2023-12-24 10:05:48 migration active, transferred 973.5 MiB of 2.0 GiB VM-state, 141.0 MiB/s 2023-12-24 10:05:49 migration active, transferred 1.1 GiB of 2.0 GiB VM-state, 119.3 MiB/s 2023-12-24 10:05:50 migration active, transferred 1.2 GiB of 2.0 GiB VM-state, 113.5 MiB/s 2023-12-24 10:05:51 migration active, transferred 1.3 GiB of 2.0 GiB VM-state, 142.3 MiB/s 2023-12-24 10:05:52 migration active, transferred 1.4 GiB of 2.0 GiB VM-state, 122.8 MiB/s 2023-12-24 10:05:53 migration active, transferred 1.5 GiB of 2.0 GiB VM-state, 174.4 MiB/s 2023-12-24 10:05:54 migration active, transferred 1.7 GiB of 2.0 GiB VM-state, 141.7 MiB/s 2023-12-24 10:05:55 migration active, transferred 1.8 GiB of 2.0 GiB VM-state, 102.6 MiB/s 2023-12-24 10:05:56 migration active, transferred 1.9 GiB of 2.0 GiB VM-state, 121.8 MiB/s 2023-12-24 10:05:57 migration active, transferred 2.1 GiB of 2.0 GiB VM-state, 179.7 MiB/s It got to that point and stopped. It's been 10 minutes. The VM itself is completely frozen meanwhile. That sucks. :(