5 ms·

> A Backblaze Storage Pod is a Building Block > But the intelligence of where to store data and how to encrypt it, deduplicate it, and index it is all at a hig

> A Backblaze Storage Pod is a Building Block > But the intelligence of where to store data and how to encrypt it, deduplicate it, and index it is all at a higher level (outside the scope of this blog post). I'm curious about their software that works outside the nodes too. I've been working on storage clusters over this past 9 months using the Ceph ( http://ceph.com/ http://ceph.com/ ) open source storage software. Its pretty amazing -- and I suspect it could be deployed to a set of backblaze pods too. It seems to be that for production environment where you wanted to maintain availability you would to build at least 3 of those pods for any deployment - enabling replication across pods/storage nodes.

xorcist 12y ago
Care to write something about your experiences with Ceph? I was recently looking at something that could be a use case for it, but it is a bit too close to the holy grail of storage for me to feel comfortable. I've toyed with similar systems in the part and they've always been very iffy with their POSIX semantics.
- andyidsinga 12y ago
  We're not doing anything explicitly posix with ceph - not using the ceph filesystem. We're using ceph for block storage under virtual machines that use KVM/QEMU with libvirt etc. The block devices exposed to the VMs end up using having an ext4 filesystem on them but that has nothing to do with ceph. So far seems fantastic from a VM perspective -- I can take down storage nodes, add new nodes etc, all while the VMs running up top stay alive and well. I once lost 1/3 of my cluster (on a 3 node cluster with 30 disks) ..and everything stayed running until I fixed the broken node. Ceph also supports Object storage and an S3 interface -- I haven't experimented much with that yet, but I hear its good.
  - xorcist 12y ago
    Thanks, that sounds like a very interesting use case. It sounds almost perfect for KVM storage. How do you expose the file system to the KVM nodes? Just mount it? What's performance like so far?
    - andyidsinga 12y ago
      qemu has support for ceph's "rbd" ( rados block device). you create an image of a particular size using the ceph tools, then when you create a vm xml definition you specify and rbd disk. ( see here http://libvirt.org/formatdomain.html#elementsDisks http://libvirt.org/formatdomain.html#elementsDisks ) finally, when the vm is booted you can use that block device like any other machine. for first install it will be empty ( need to attach a cdrom image to get going ) then youncan start making snapshots for "thin provisioning" using copy on write clones ...very sweet stuff :) re performance ...i think its good, but i have heard its not as good as expensive proprietary systems. the product we're building is around storage analytics on open source. ..so eventually well know more :)

xorcist 12y ago

Care to write something about your experiences with Ceph? I was recently looking at something that could be a use case for it, but it is a bit too close to the holy grail of storage for me to feel comfortable. I've toyed with similar systems in the part and they've always been very iffy with their POSIX semantics.

andyidsinga 12y ago
We're not doing anything explicitly posix with ceph - not using the ceph filesystem. We're using ceph for block storage under virtual machines that use KVM/QEMU with libvirt etc. The block devices exposed to the VMs end up using having an ext4 filesystem on them but that has nothing to do with ceph. So far seems fantastic from a VM perspective -- I can take down storage nodes, add new nodes etc, all while the VMs running up top stay alive and well. I once lost 1/3 of my cluster (on a 3 node cluster with 30 disks) ..and everything stayed running until I fixed the broken node. Ceph also supports Object storage and an S3 interface -- I haven't experimented much with that yet, but I hear its good.
- xorcist 12y ago
  Thanks, that sounds like a very interesting use case. It sounds almost perfect for KVM storage. How do you expose the file system to the KVM nodes? Just mount it? What's performance like so far?
  - andyidsinga 12y ago
    qemu has support for ceph's "rbd" ( rados block device). you create an image of a particular size using the ceph tools, then when you create a vm xml definition you specify and rbd disk. ( see here http://libvirt.org/formatdomain.html#elementsDisks http://libvirt.org/formatdomain.html#elementsDisks ) finally, when the vm is booted you can use that block device like any other machine. for first install it will be empty ( need to attach a cdrom image to get going ) then youncan start making snapshots for "thin provisioning" using copy on write clones ...very sweet stuff :) re performance ...i think its good, but i have heard its not as good as expensive proprietary systems. the product we're building is around storage analytics on open source. ..so eventually well know more :)