4 ms·

data loss due to a broken replica-repair strategy. That's the thing: Redis has no read repair strategy. Redis has a "be an exact copy of your master" strategy

by seiji 12y ago

data loss due to a broken replica-repair strategy. That's the thing: Redis has no read repair strategy. Redis has a "be an exact copy of your master" strategy. It's not a secret. It's the exact design. The complaints are like yelling at Linus when you rm -rf / your entire machine. Sure, it sucks, but it's a repercussion if your own actions, not a fault in the system. If you don't want to rm -rf / your machine, go use an OS designed for babies (cough ubuntu cough). (Plus, there are already Redis improvements (designed within a day or two of the original problem being reported) to provide workarounds to users who _do_ want to run that exact use case. The answer to problems is solutions—not complaining and blaming endlessly.) "the 99% percentile is bad" in a more negative way than he intended? Text. It's only text. You can't read the intonation and people want to read absolutes. People want to read anger. Always anger. Always confrontation. It's possible the author of the text didn't mean to insult your mother. Breathe. It'll be okay. The entire goal of the Internet is to get ALL THE ATTENTION YOURSELF. If anybody hates you online, it's because you got attention and they didn't. Nobody hates insignificant people. So, often times people with lower profiles/attention in conversations will try to increase their attention profile by arguing/hating the high-attention people. If people hate you, you've already won.

notacoward 12y ago
When you say "be an exact copy of your master" is the strategy, you're kind of missing the problem here - who should be the master? In this case, Redis made a really dumb choice of which node should be master, and brought everything in sync by replicating emptiness instead of replicating data. There are very few use cases where that would be preferable. People have pointed out well known and fairly simple solutions to the selection problem, which are leading to the fixes and workarounds you've mentioned (though without the courtesy of acknowledging where the ideas came from). Your analogy to "rm -rf /" is invalid, because that's well defined, well documented, and well known behavior. That's not true of Redis's autonomous and non-deterministic response to a failure (not to a user action). No spec or doc precluded choosing a different master and preserving data instead of discarding it. In the absence of such explicit guidance, preserving data should always be the default. How can it be user error when the user did nothing? Redis did the wrong thing because something was missed in its implementation, not because of any rational or deliberate choice.
- seiji 12y ago
  you're kind of missing the problem here - who should be the master? The choice of master is a static configuration set by the user. Redis itself has no failover or promotion ability. There's an additional thing called Sentinel that can failover and promote individual Redis instances, but it is designed to recover complete instance failures (without immediately restarting), so a quick restart means no failover happens [an improvement to the "quick restart" scenario is showing up soon]. Redis made a really dumb choice of which node should be master, (see previously; master is static, defined by the user) pointed out well known and fairly simple solutions to the selection problem (redis doesn't select things) Also, this issue showed up last week. Last week. People are making it sound like this issue has been ignored for years. Nobody ran into this (and reported it) until recently. This use case is already being adapted into SOP Redis capabilities soon. Try running into a big problem with any other DB and getting both attention and a concrete fix within two weeks. For free. The entire progress of the project has paused to address these immediate user issues. because that's well defined, well documented, and well known behavior. The Redis behavior is: always be a copy of a statically configured master. When the master has an empty dataset, all the replicas replicate an empty dataset. Pretty simple. :) No spec or doc precluded choosing a different master and preserving data instead of discarding it. Yup, specs and documentation did exactly that. Redis has no failover capability on its own. preserving data should always be the default. Ooops, you just re-invented the Mac trashcan. How can it be user error when the user did nothing? The user disabled persistence, enabled replication, restarted the process with zero data, then the replication recovered and stayed in sync with the newly zero-data master. something was missed in its implementation, not because of any rational or deliberate choice. nopers. more a lack of thinking it through from the user's point of view. an exact copy of nothing ends up being nothing.
  - notacoward 12y ago
    "People are making it sound like this issue has been ignored for years." Hasn't it been? How does leaving that latent in the system for years make things better? I rather think it reflects on an inability to reason about failure modes (including user failure modes), and deal with them proactively instead of after data was lost.

by seiji 12y ago

notacoward 12y ago

When you say "be an exact copy of your master" is the strategy, you're kind of missing the problem here - who should be the master? In this case, Redis made a really dumb choice of which node should be master, and brought everything in sync by replicating emptiness instead of replicating data. There are very few use cases where that would be preferable. People have pointed out well known and fairly simple solutions to the selection problem, which are leading to the fixes and workarounds you've mentioned (though without the courtesy of acknowledging where the ideas came from). Your analogy to "rm -rf /" is invalid, because that's well defined, well documented, and well known behavior. That's not true of Redis's autonomous and non-deterministic response to a failure (not to a user action). No spec or doc precluded choosing a different master and preserving data instead of discarding it. In the absence of such explicit guidance, preserving data should always be the default. How can it be user error when the user did nothing? Redis did the wrong thing because something was missed in its implementation, not because of any rational or deliberate choice.

seiji 12y ago
you're kind of missing the problem here - who should be the master? The choice of master is a static configuration set by the user. Redis itself has no failover or promotion ability. There's an additional thing called Sentinel that can failover and promote individual Redis instances, but it is designed to recover complete instance failures (without immediately restarting), so a quick restart means no failover happens [an improvement to the "quick restart" scenario is showing up soon]. Redis made a really dumb choice of which node should be master, (see previously; master is static, defined by the user) pointed out well known and fairly simple solutions to the selection problem (redis doesn't select things) Also, this issue showed up last week. Last week. People are making it sound like this issue has been ignored for years. Nobody ran into this (and reported it) until recently. This use case is already being adapted into SOP Redis capabilities soon. Try running into a big problem with any other DB and getting both attention and a concrete fix within two weeks. For free. The entire progress of the project has paused to address these immediate user issues. because that's well defined, well documented, and well known behavior. The Redis behavior is: always be a copy of a statically configured master. When the master has an empty dataset, all the replicas replicate an empty dataset. Pretty simple. :) No spec or doc precluded choosing a different master and preserving data instead of discarding it. Yup, specs and documentation did exactly that. Redis has no failover capability on its own. preserving data should always be the default. Ooops, you just re-invented the Mac trashcan. How can it be user error when the user did nothing? The user disabled persistence, enabled replication, restarted the process with zero data, then the replication recovered and stayed in sync with the newly zero-data master. something was missed in its implementation, not because of any rational or deliberate choice. nopers. more a lack of thinking it through from the user's point of view. an exact copy of nothing ends up being nothing.
- notacoward 12y ago
  "People are making it sound like this issue has been ignored for years." Hasn't it been? How does leaving that latent in the system for years make things better? I rather think it reflects on an inability to reason about failure modes (including user failure modes), and deal with them proactively instead of after data was lost.