We've been using Ceph and the S3-compatible radosgw at a small scale while building http://www.pogoapp.com http://www.pogoapp.com and I have no real complaints.

We've been using Ceph and the S3-compatible radosgw at a small scale while building http://www.pogoapp.com http://www.pogoapp.com and I have no real complaints. It's a very new technology and even things like running a cluster backup aren't yet possible, but Dreamhost has been providing its customers a free beta for a while now, the program is under heavy development, has Dreamhost and Shuttleworth providing funding, and the devs on IRC are quite helpful and responsive. And it's being integrated into OpenStack It looks like Ceph is still waiting on stable btrfs to truly shine (http://ceph.com/docs/master/rec/filesystem/ http://ceph.com/docs/master/rec/filesystem/), but overall the technology looks very promising and works well today, with a small number of groups running large multi-rack storage clusters (e.g. I believe it runs DreamHost's entire backup system)