4 ms·

We're spinning up a medium sized Proxmox clusters (~50 nodes in total) to replace our aging Xen clusters. I saw Ceph is available on the Proxmox platform, but

We're spinning up a medium sized Proxmox clusters (~50 nodes in total) to replace our aging Xen clusters. I saw Ceph is available on the Proxmox platform, but was hesitant to make all the VM storage backed by Ceph (throwing all the eggs into a single basket). What were some of the other hurdles you faced in your Ceph deployment?

tasqa 5y ago
I found proxmox not to be very user friendly growing to such cluster sizes. Proxmox itself has been very stable and supports pretty much anything but the GUI is not that great if you have many nodes and VMS, and the API can be lacking. However, using ceph as a backing store for VM images is pretty easy in proxmox. I have not used the cephFS stuff. I used it in a separate cluster both physically and standalone (not using proxmox integration). So RBD is easy, S3 is somewhat more complicated as you need to run multiple gateways, but still very doable. The FS stuff also needs extra daemons, but I have not yet tested it.
tinco 5y ago
We've been playing around with migrating our bare metals to proxmox as well. Though one main argument, being able to reboot/manage crashed GPU accelerated nodes, was invalidated by proxmox (KVM?) itself crashing whenever the GPU would crash, so it didn't solve our core problem. This is of course also due to that we're not using industrial components, but it is what it is. I found Ceph's error messages very hard to debug. Just google around a bit for the manuals of how to deal with buggy or fully defective drives. There's a lot of SSH'ing in, running vague commands looking up id's of drives and matching them to linux device mount points and reading vague error logs. To me as a high level operator it feels it should be simple. If a drive supplies a block of data, and that data fails its checksum, it's gone. The drive already does its very best internally to cope with physical issues, if the drive couldn't come up with valid data, it's toast or as close to toast as anyone should be comfortable with. So it's simple, fail a checksum, out of the cluster, send me an e-mail, I don't get why Ceph has to be so much more complicated than that.

tasqa 5y ago

I found proxmox not to be very user friendly growing to such cluster sizes. Proxmox itself has been very stable and supports pretty much anything but the GUI is not that great if you have many nodes and VMS, and the API can be lacking. However, using ceph as a backing store for VM images is pretty easy in proxmox. I have not used the cephFS stuff. I used it in a separate cluster both physically and standalone (not using proxmox integration). So RBD is easy, S3 is somewhat more complicated as you need to run multiple gateways, but still very doable. The FS stuff also needs extra daemons, but I have not yet tested it.

tinco 5y ago

We've been playing around with migrating our bare metals to proxmox as well. Though one main argument, being able to reboot/manage crashed GPU accelerated nodes, was invalidated by proxmox (KVM?) itself crashing whenever the GPU would crash, so it didn't solve our core problem. This is of course also due to that we're not using industrial components, but it is what it is. I found Ceph's error messages very hard to debug. Just google around a bit for the manuals of how to deal with buggy or fully defective drives. There's a lot of SSH'ing in, running vague commands looking up id's of drives and matching them to linux device mount points and reading vague error logs. To me as a high level operator it feels it should be simple. If a drive supplies a block of data, and that data fails its checksum, it's gone. The drive already does its very best internally to cope with physical issues, if the drive couldn't come up with valid data, it's toast or as close to toast as anyone should be comfortable with. So it's simple, fail a checksum, out of the cluster, send me an e-mail, I don't get why Ceph has to be so much more complicated than that.