4 ms·

Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?

by Tepix 6y ago

hutrdvnj 6y ago
Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.
- luto 6y ago
  can confirm this on two 24 node / 100TB / 132 OSD clusters and ~2 years.
  - elongatedMusku 6y ago
    Getting my feet wet with 3 nodes, 3 osds. Wish me luck.
- effie 6y ago
  Curious about Ceph, what feature are you using it for? Would it make sense to serve big number of websites, stored on 20 servers, using single Ceph filesystem with some SSD cache? Or am I better off having just 20 independent servers?
  - hutrdvnj 6y ago
    > Curious about Ceph, what feature are you using it for? Mainly CephFS. It allows you similar to NFS to have a big shared filesystem over network. If you have clients that require access to a triple digit terabyte or petabyte sized filessystem, then you might want to consider ceph. > Or am I better off having just 20 independent servers? If you have 20 independent websites, then I would recommend to provide them their own storage. That is because if your ceph instance has a downtime, then all your 20 websites would be down simultaneously.
sekh60 6y ago
I run a home ceph cluster as a hobby. It works really well. I have hit a few snags, but the mailing list is super friendly and helpful and the devs are quick to respond to bug reports.
antongribok 6y ago
I (with 2-3 other people) run over 100 PB on Ceph clusters in production, supporting some critical functions of a Fortune 50 enterprise. We use Ceph for block and object workloads (no CephFS). Most of our clusters are still on Luminous (v12) and Filestore (XFS), and only our newer clusters that are being built now are on Nautilus (v14) Bluestore. We plan on migrating to Bluestore this year (and likely next year). I'm on-call basically all the time, but we'll have maybe 1 issue per year where we have to act immediately. Most failures that would happen on a Saturday can wait until Monday to be acted on. The smallest cluster that we'll build is 6 nodes. Our largest cluster now is 120 nodes with 1920 OSDs. We might build a larger cluster this year. Back in Ceph Hammer days, I had a 6-node cluster lose 3 nodes, one node at a time, over a course of a couple of days with zero downtime. We deploy with Ansible, and have our own, very paranoid and opinionated playbook for doing a rolling cluster reboot or restart, that checks multiple things before moving on to the next node.

Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?

by Tepix 6y ago

Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?

Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.

luto 6y ago
can confirm this on two 24 node / 100TB / 132 OSD clusters and ~2 years.
- elongatedMusku 6y ago
  Getting my feet wet with 3 nodes, 3 osds. Wish me luck.
effie 6y ago
Curious about Ceph, what feature are you using it for? Would it make sense to serve big number of websites, stored on 20 servers, using single Ceph filesystem with some SSD cache? Or am I better off having just 20 independent servers?
- hutrdvnj 6y ago
  > Curious about Ceph, what feature are you using it for? Mainly CephFS. It allows you similar to NFS to have a big shared filesystem over network. If you have clients that require access to a triple digit terabyte or petabyte sized filessystem, then you might want to consider ceph. > Or am I better off having just 20 independent servers? If you have 20 independent websites, then I would recommend to provide them their own storage. That is because if your ceph instance has a downtime, then all your 20 websites would be down simultaneously.

sekh60 6y ago

I run a home ceph cluster as a hobby. It works really well. I have hit a few snags, but the mailing list is super friendly and helpful and the devs are quick to respond to bug reports.

antongribok 6y ago

I (with 2-3 other people) run over 100 PB on Ceph clusters in production, supporting some critical functions of a Fortune 50 enterprise. We use Ceph for block and object workloads (no CephFS). Most of our clusters are still on Luminous (v12) and Filestore (XFS), and only our newer clusters that are being built now are on Nautilus (v14) Bluestore. We plan on migrating to Bluestore this year (and likely next year). I'm on-call basically all the time, but we'll have maybe 1 issue per year where we have to act immediately. Most failures that would happen on a Saturday can wait until Monday to be acted on. The smallest cluster that we'll build is 6 nodes. Our largest cluster now is 120 nodes with 1920 OSDs. We might build a larger cluster this year. Back in Ceph Hammer days, I had a 6-node cluster lose 3 nodes, one node at a time, over a course of a couple of days with zero downtime. We deploy with Ansible, and have our own, very paranoid and opinionated playbook for doing a rolling cluster reboot or restart, that checks multiple things before moving on to the next node.