Ask HN: What are your experiences with Ceph?

I've been playing with Ceph/librados/CephFS lately and I'm evaluating it for various projects at work (petabytes of large VM images, petabytes of super small files, fun, right?). What are your experiences with it? Tell me everything: the good, the bad and the ugly.