3 ms·

Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a fr

Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.

justinclift 3y ago
Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)
- justinclift 3y ago
  And another migration has just gotten stuck. This time of the Debian 12 installer running. Nothing is looking obviously weird in the output log: 2023-12-24 10:05:33 starting migration of VM 101 to node 'proxmox4' (10.1.1.101) 2023-12-24 10:05:33 starting VM 101 on remote node 'proxmox4' 2023-12-24 10:05:36 start remote tunnel 2023-12-24 10:05:38 ssh tunnel ver 1 2023-12-24 10:05:38 starting online/live migration on unix:/run/qemu-server/101.migrate 2023-12-24 10:05:38 set migration capabilities 2023-12-24 10:05:38 migration downtime limit: 100 ms 2023-12-24 10:05:38 migration cachesize: 256.0 MiB 2023-12-24 10:05:38 set migration parameters 2023-12-24 10:05:38 start migrate command to unix:/run/qemu-server/101.migrate 2023-12-24 10:05:39 migration active, transferred 58.6 MiB of 2.0 GiB VM-state, 101.7 MiB/s 2023-12-24 10:05:40 migration active, transferred 131.4 MiB of 2.0 GiB VM-state, 92.7 MiB/s 2023-12-24 10:05:41 migration active, transferred 233.6 MiB of 2.0 GiB VM-state, 96.8 MiB/s 2023-12-24 10:05:42 migration active, transferred 341.5 MiB of 2.0 GiB VM-state, 108.9 MiB/s 2023-12-24 10:05:43 migration active, transferred 457.1 MiB of 2.0 GiB VM-state, 107.5 MiB/s 2023-12-24 10:05:44 migration active, transferred 557.6 MiB of 2.0 GiB VM-state, 108.2 MiB/s 2023-12-24 10:05:45 migration active, transferred 661.4 MiB of 2.0 GiB VM-state, 113.3 MiB/s 2023-12-24 10:05:46 migration active, transferred 773.7 MiB of 2.0 GiB VM-state, 57.4 MiB/s 2023-12-24 10:05:47 migration active, transferred 851.0 MiB of 2.0 GiB VM-state, 127.4 MiB/s 2023-12-24 10:05:48 migration active, transferred 973.5 MiB of 2.0 GiB VM-state, 141.0 MiB/s 2023-12-24 10:05:49 migration active, transferred 1.1 GiB of 2.0 GiB VM-state, 119.3 MiB/s 2023-12-24 10:05:50 migration active, transferred 1.2 GiB of 2.0 GiB VM-state, 113.5 MiB/s 2023-12-24 10:05:51 migration active, transferred 1.3 GiB of 2.0 GiB VM-state, 142.3 MiB/s 2023-12-24 10:05:52 migration active, transferred 1.4 GiB of 2.0 GiB VM-state, 122.8 MiB/s 2023-12-24 10:05:53 migration active, transferred 1.5 GiB of 2.0 GiB VM-state, 174.4 MiB/s 2023-12-24 10:05:54 migration active, transferred 1.7 GiB of 2.0 GiB VM-state, 141.7 MiB/s 2023-12-24 10:05:55 migration active, transferred 1.8 GiB of 2.0 GiB VM-state, 102.6 MiB/s 2023-12-24 10:05:56 migration active, transferred 1.9 GiB of 2.0 GiB VM-state, 121.8 MiB/s 2023-12-24 10:05:57 migration active, transferred 2.1 GiB of 2.0 GiB VM-state, 179.7 MiB/s It got to that point and stopped. It's been 10 minutes. The VM itself is completely frozen meanwhile. That sucks. :(
  - justinclift 3y ago
    Looks like I'll need to put some time aside to properly debug this in the new year and figure out what's going wrong. I'd like to think it's just some kind of dumb setting I should have changed (being a newbie with Proxmox), but so far this "feels" more like a bug of some kind. Will hopefully be able to get that sorted. :)

Do all of your nodes have identical CPUs (or if not, a generic CPU selected to be virtualized)? That’s the main thing off the top of my head that can cause a freeze.

justinclift 3y ago

Yeah, the whole setup (for now) is just running virtualised in my desktop (Ryzen 5950x, 64GB ram). So each node has 6x cores and 8GB ram, with 2 network interfaces (1 public, 1 for ceph cluster traffic). After I have more experience I'll look to replicate it in hardware, but starting with a virtualised setup lets me test different configurations easily. :)

justinclift 3y ago
And another migration has just gotten stuck. This time of the Debian 12 installer running. Nothing is looking obviously weird in the output log: 2023-12-24 10:05:33 starting migration of VM 101 to node 'proxmox4' (10.1.1.101) 2023-12-24 10:05:33 starting VM 101 on remote node 'proxmox4' 2023-12-24 10:05:36 start remote tunnel 2023-12-24 10:05:38 ssh tunnel ver 1 2023-12-24 10:05:38 starting online/live migration on unix:/run/qemu-server/101.migrate 2023-12-24 10:05:38 set migration capabilities 2023-12-24 10:05:38 migration downtime limit: 100 ms 2023-12-24 10:05:38 migration cachesize: 256.0 MiB 2023-12-24 10:05:38 set migration parameters 2023-12-24 10:05:38 start migrate command to unix:/run/qemu-server/101.migrate 2023-12-24 10:05:39 migration active, transferred 58.6 MiB of 2.0 GiB VM-state, 101.7 MiB/s 2023-12-24 10:05:40 migration active, transferred 131.4 MiB of 2.0 GiB VM-state, 92.7 MiB/s 2023-12-24 10:05:41 migration active, transferred 233.6 MiB of 2.0 GiB VM-state, 96.8 MiB/s 2023-12-24 10:05:42 migration active, transferred 341.5 MiB of 2.0 GiB VM-state, 108.9 MiB/s 2023-12-24 10:05:43 migration active, transferred 457.1 MiB of 2.0 GiB VM-state, 107.5 MiB/s 2023-12-24 10:05:44 migration active, transferred 557.6 MiB of 2.0 GiB VM-state, 108.2 MiB/s 2023-12-24 10:05:45 migration active, transferred 661.4 MiB of 2.0 GiB VM-state, 113.3 MiB/s 2023-12-24 10:05:46 migration active, transferred 773.7 MiB of 2.0 GiB VM-state, 57.4 MiB/s 2023-12-24 10:05:47 migration active, transferred 851.0 MiB of 2.0 GiB VM-state, 127.4 MiB/s 2023-12-24 10:05:48 migration active, transferred 973.5 MiB of 2.0 GiB VM-state, 141.0 MiB/s 2023-12-24 10:05:49 migration active, transferred 1.1 GiB of 2.0 GiB VM-state, 119.3 MiB/s 2023-12-24 10:05:50 migration active, transferred 1.2 GiB of 2.0 GiB VM-state, 113.5 MiB/s 2023-12-24 10:05:51 migration active, transferred 1.3 GiB of 2.0 GiB VM-state, 142.3 MiB/s 2023-12-24 10:05:52 migration active, transferred 1.4 GiB of 2.0 GiB VM-state, 122.8 MiB/s 2023-12-24 10:05:53 migration active, transferred 1.5 GiB of 2.0 GiB VM-state, 174.4 MiB/s 2023-12-24 10:05:54 migration active, transferred 1.7 GiB of 2.0 GiB VM-state, 141.7 MiB/s 2023-12-24 10:05:55 migration active, transferred 1.8 GiB of 2.0 GiB VM-state, 102.6 MiB/s 2023-12-24 10:05:56 migration active, transferred 1.9 GiB of 2.0 GiB VM-state, 121.8 MiB/s 2023-12-24 10:05:57 migration active, transferred 2.1 GiB of 2.0 GiB VM-state, 179.7 MiB/s It got to that point and stopped. It's been 10 minutes. The VM itself is completely frozen meanwhile. That sucks. :(
- justinclift 3y ago
  Looks like I'll need to put some time aside to properly debug this in the new year and figure out what's going wrong. I'd like to think it's just some kind of dumb setting I should have changed (being a newbie with Proxmox), but so far this "feels" more like a bug of some kind. Will hopefully be able to get that sorted. :)