4 ms·

This is absolutely not true. The normal flags you're looking for are: noout, noscrub, nodeep-scrub. Do your maintenance. Unset those flags. The noout is wha

This is absolutely not true. The normal flags you're looking for are: noout, noscrub, nodeep-scrub. Do your maintenance. Unset those flags. The noout is what tells Ceph to essentially not shuffle data when OSDs go down. This is also referenced in the article.

puzzle 9y ago
Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild. Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops. Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf https://static.googleusercontent.com/media/research.google.c...
- tene 9y ago
  That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.
- antongribok 9y ago
  Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.
  - puzzle 9y ago
    That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.
rconti 9y ago
Thank you. I was (obviously) very far from a Ceph expert, and I never had to implement it, in part because we could never get a satisfactory approach to this problem. Nobody I met with had an answer, and I couldn't find one after doing quite a bit of research. However, as usual with these kinds of things, simple document reading and keyword searches won't always find you the answer -- you need to understand the technology in depth to even understand which questions to ask. In an ideal world, we would have had multiple people to dedicate to building the kind of cluster they wanted to build. In the real world, after a dozen people left, I'm afraid they may well have ended up in a situation not much better than the University in question.

puzzle 9y ago

Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild. Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops. Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf https://static.googleusercontent.com/media/research.google.c...

tene 9y ago
That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.
antongribok 9y ago
Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.
- puzzle 9y ago
  That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.

rconti 9y ago

Thank you. I was (obviously) very far from a Ceph expert, and I never had to implement it, in part because we could never get a satisfactory approach to this problem. Nobody I met with had an answer, and I couldn't find one after doing quite a bit of research. However, as usual with these kinds of things, simple document reading and keyword searches won't always find you the answer -- you need to understand the technology in depth to even understand which questions to ask. In an ideal world, we would have had multiple people to dedicate to building the kind of cluster they wanted to build. In the real world, after a dozen people left, I'm afraid they may well have ended up in a situation not much better than the University in question.