3 ms·

Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time,

by puzzle 9y ago

Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild. Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops. Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf https://static.googleusercontent.com/media/research.google.c...

tene 9y ago
That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.
antongribok 9y ago
Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.
- puzzle 9y ago
  That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.

by puzzle 9y ago

tene 9y ago

That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.

antongribok 9y ago

Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.

puzzle 9y ago
That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.