4 ms·

Care to write something about your experiences with Ceph? I was recently looking at something that could be a use case for it, but it is a bit too close to the

by xorcist 12y ago

Care to write something about your experiences with Ceph? I was recently looking at something that could be a use case for it, but it is a bit too close to the holy grail of storage for me to feel comfortable. I've toyed with similar systems in the part and they've always been very iffy with their POSIX semantics.

andyidsinga 12y ago
We're not doing anything explicitly posix with ceph - not using the ceph filesystem. We're using ceph for block storage under virtual machines that use KVM/QEMU with libvirt etc. The block devices exposed to the VMs end up using having an ext4 filesystem on them but that has nothing to do with ceph. So far seems fantastic from a VM perspective -- I can take down storage nodes, add new nodes etc, all while the VMs running up top stay alive and well. I once lost 1/3 of my cluster (on a 3 node cluster with 30 disks) ..and everything stayed running until I fixed the broken node. Ceph also supports Object storage and an S3 interface -- I haven't experimented much with that yet, but I hear its good.
- xorcist 12y ago
  Thanks, that sounds like a very interesting use case. It sounds almost perfect for KVM storage. How do you expose the file system to the KVM nodes? Just mount it? What's performance like so far?
  - andyidsinga 12y ago
    qemu has support for ceph's "rbd" ( rados block device). you create an image of a particular size using the ceph tools, then when you create a vm xml definition you specify and rbd disk. ( see here http://libvirt.org/formatdomain.html#elementsDisks http://libvirt.org/formatdomain.html#elementsDisks ) finally, when the vm is booted you can use that block device like any other machine. for first install it will be empty ( need to attach a cdrom image to get going ) then youncan start making snapshots for "thin provisioning" using copy on write clones ...very sweet stuff :) re performance ...i think its good, but i have heard its not as good as expensive proprietary systems. the product we're building is around storage analytics on open source. ..so eventually well know more :)

by xorcist 12y ago

andyidsinga 12y ago

We're not doing anything explicitly posix with ceph - not using the ceph filesystem. We're using ceph for block storage under virtual machines that use KVM/QEMU with libvirt etc. The block devices exposed to the VMs end up using having an ext4 filesystem on them but that has nothing to do with ceph. So far seems fantastic from a VM perspective -- I can take down storage nodes, add new nodes etc, all while the VMs running up top stay alive and well. I once lost 1/3 of my cluster (on a 3 node cluster with 30 disks) ..and everything stayed running until I fixed the broken node. Ceph also supports Object storage and an S3 interface -- I haven't experimented much with that yet, but I hear its good.

xorcist 12y ago
Thanks, that sounds like a very interesting use case. It sounds almost perfect for KVM storage. How do you expose the file system to the KVM nodes? Just mount it? What's performance like so far?
- andyidsinga 12y ago
  qemu has support for ceph's "rbd" ( rados block device). you create an image of a particular size using the ceph tools, then when you create a vm xml definition you specify and rbd disk. ( see here http://libvirt.org/formatdomain.html#elementsDisks http://libvirt.org/formatdomain.html#elementsDisks ) finally, when the vm is booted you can use that block device like any other machine. for first install it will be empty ( need to attach a cdrom image to get going ) then youncan start making snapshots for "thin provisioning" using copy on write clones ...very sweet stuff :) re performance ...i think its good, but i have heard its not as good as expensive proprietary systems. the product we're building is around storage analytics on open source. ..so eventually well know more :)