4 ms·

I distinctly remember being dialed into a meeting by phone where someone was explaining the project requirements of what would be our first Ceph deployment. Ove

by rconti 9y ago

I distinctly remember being dialed into a meeting by phone where someone was explaining the project requirements of what would be our first Ceph deployment. Over and over it was explained there was simply NO provision for restarting nodes without the remaining nodes trying to rebuild for the configured n+x redundancy. I completely disagreed and said it made no sense at all, until I read up all I could on the matter. No provision for patching. Period. If a node goes down, the cluster tries to rebuild. I'm not sure what reality this design comes from, but it's not one I care to inhabit.

antongribok 9y ago
This is absolutely not true. The normal flags you're looking for are: noout, noscrub, nodeep-scrub. Do your maintenance. Unset those flags. The noout is what tells Ceph to essentially not shuffle data when OSDs go down. This is also referenced in the article.
- puzzle 9y ago
  Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild. Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops. Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf https://static.googleusercontent.com/media/research.google.c...
  - tene 9y ago
    That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.
  - antongribok 9y ago
    Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.
    - puzzle 9y ago
      That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.
- rconti 9y ago
  Thank you. I was (obviously) very far from a Ceph expert, and I never had to implement it, in part because we could never get a satisfactory approach to this problem. Nobody I met with had an answer, and I couldn't find one after doing quite a bit of research. However, as usual with these kinds of things, simple document reading and keyword searches won't always find you the answer -- you need to understand the technology in depth to even understand which questions to ask. In an ideal world, we would have had multiple people to dedicate to building the kind of cluster they wanted to build. In the real world, after a dozen people left, I'm afraid they may well have ended up in a situation not much better than the University in question.
tene 9y ago
I'm curious if there's a use case or edge scenario I'm not aware of that noout does not cover. I can't say I've got exhaustive testing for this, but when performing maintenance on a ceph cluster I ran at a previous employer, "ceph osd set noout" before maintenance did exactly what it is documented to do and prevented rebalancing while OSDs were stopped. Despite my poor memory, I'm pretty confident about that, because at least one upgrade I performed did end up requiring some data migration (I believe it was migrating from straw to straw2 bucket type), and we only made that change after all nodes were successfully upgraded. http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing http://docs.ceph.com/docs/master/rados/troubleshooting/troub... My memory is generally quite poor, but I vaguely recall this feature being present for as long as I've been familiar with Ceph. I obviously have no way of knowing what happened in that meeting you were in, and maybe the other people who were proposing using Ceph were not very familiar with performing maintenance on it, or maybe there was other constraint or use case they had in mind, but I'm quite confident in saying that the normal case for Ceph maintenance involves only a very marginal amount of data movement (to bring the temporarily-down OSD back up-to-date with changes that occurred while it was down).
- kjetijor 9y ago
  I've used ceph since firefly - and noout existed back then as well. Assuming that you run with 3 failure domains, and only maintain one failure domain at a time. Noout mostly gets the job done. What it doesn't do for you is save you from an actual failure in a different failure domain during maintenance. EC pools & k+(m>=2) or replication > 3 - would cover this as well. We've had mostly great success with noout + maintain failure domain at a time, wait for recovery, proceed to next failure domain, repeat until done. To the point where we've been comfortable leaving a lot of the babysitting & work to machines.

by rconti 9y ago

antongribok 9y ago

This is absolutely not true. The normal flags you're looking for are: noout, noscrub, nodeep-scrub. Do your maintenance. Unset those flags. The noout is what tells Ceph to essentially not shuffle data when OSDs go down. This is also referenced in the article.

puzzle 9y ago
Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild. Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops. Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf https://static.googleusercontent.com/media/research.google.c...
- tene 9y ago
  That's also configurable. The default setting for "mon osd down out interval" is 600 seconds. See http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ http://docs.ceph.com/docs/master/rados/configuration/mon-osd... for a lot more configuration that can be done here.
- antongribok 9y ago
  Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it). The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess. EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.
  - puzzle 9y ago
    That's cool. Thanks! I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.
rconti 9y ago
Thank you. I was (obviously) very far from a Ceph expert, and I never had to implement it, in part because we could never get a satisfactory approach to this problem. Nobody I met with had an answer, and I couldn't find one after doing quite a bit of research. However, as usual with these kinds of things, simple document reading and keyword searches won't always find you the answer -- you need to understand the technology in depth to even understand which questions to ask. In an ideal world, we would have had multiple people to dedicate to building the kind of cluster they wanted to build. In the real world, after a dozen people left, I'm afraid they may well have ended up in a situation not much better than the University in question.

tene 9y ago

I'm curious if there's a use case or edge scenario I'm not aware of that noout does not cover. I can't say I've got exhaustive testing for this, but when performing maintenance on a ceph cluster I ran at a previous employer, "ceph osd set noout" before maintenance did exactly what it is documented to do and prevented rebalancing while OSDs were stopped. Despite my poor memory, I'm pretty confident about that, because at least one upgrade I performed did end up requiring some data migration (I believe it was migrating from straw to straw2 bucket type), and we only made that change after all nodes were successfully upgraded. http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing http://docs.ceph.com/docs/master/rados/troubleshooting/troub... My memory is generally quite poor, but I vaguely recall this feature being present for as long as I've been familiar with Ceph. I obviously have no way of knowing what happened in that meeting you were in, and maybe the other people who were proposing using Ceph were not very familiar with performing maintenance on it, or maybe there was other constraint or use case they had in mind, but I'm quite confident in saying that the normal case for Ceph maintenance involves only a very marginal amount of data movement (to bring the temporarily-down OSD back up-to-date with changes that occurred while it was down).

kjetijor 9y ago
I've used ceph since firefly - and noout existed back then as well. Assuming that you run with 3 failure domains, and only maintain one failure domain at a time. Noout mostly gets the job done. What it doesn't do for you is save you from an actual failure in a different failure domain during maintenance. EC pools & k+(m>=2) or replication > 3 - would cover this as well. We've had mostly great success with noout + maintain failure domain at a time, wait for recovery, proceed to next failure domain, repeat until done. To the point where we've been comfortable leaving a lot of the babysitting & work to machines.