4 ms·

Did you read the post? "the 99% percentile is bad" is something I never wrote, it was part of a larger sentence where it was obvious it meant if the 99th percen

by antirez 12y ago

Did you read the post? "the 99% percentile is bad" is something I never wrote, it was part of a larger sentence where it was obvious it meant if the 99th percentile figure is too big. About pointing to relevant papers: things like pointing to timestamped replication paper, which is a CP system for replicated state machines, in response to the fact that Redis does not support slaves when the master disk persistence is turned off, is actually just another instance of why it is not possible to have decent tech conversations on Twitter. The problem experienced at Stripe is a result of Redis replication documented behavior, regardless of using Sentinel or not. Redis replication is a very simple system where replicas will try to exactly mimic the master, and if the link breaks, will try to connect with it again and again forever. There is no builtin HA, nor failover or alike. So before considering failover (and yet IMHO pointing to timestamped replication is not very informative even in this context, since in distributed systems the details matter, and you can't just give a random reference to a completely different system which happens to have just superficially similar issues to fix), there were different useful observations to do. Like: Hey @antirez, what about supporting master-slaves setups where the master can be configured without persistence at all, and yet when it reboots, it will not be considered viable for reconnections? Which is the fundamental problem: stopping the basic Redis replication behavior from working as it works, with slaves that always want to replicate the current master, which is not ok if we want a system supporing the master restarted without persistence (which wipes the dataset on restart). However the whole problem with that is that there are a lot of people like you that will regard linking to a paper as a great way to help, and as a very smart thing, while there are other that are instead trying to work to really make stuff better. Before commenting you should make the effort to understand exactly the problem domain and its subtleness: exact details or what you say is not relevant in a discussion which is all about details.

notacoward 12y ago
See my response to seiji. Exact mimicry might be a defined behavior, but where is it mandated who should mimic whom? Where had you said, before this, that the node with no data must be the one whose (empty) view should prevail? As for your false dichotomy between reading the literature and getting stuff done, or your "make the effort to understand" ad hominem - go to hell. I've been doing this longer than you, I've been doing it better than you, and I've been writing about it as well. You make the effort to understand the problem before you shoot yourself in the foot yet again.
- antirez 12y ago
  "but where is it mandated who should mimic whom" In Redis if you don't use any HA system like Sentinel, the map is fixed, it is an old-style replication system where there is the master IP address written in the configuration file. Since the system is not supposed to lose the data on restarts, this is fine, but as soon as you want to support a different mode of operation with persistence-less masters, this must be modified, being Redis used with HA or not. Now if you want to put Sentinel in the mix, the problem with this setup is not that the returning master gets promoted with a broken data set, but that if the restart is fast enough, the failure detection of Sentinel is not triggered at all, so the configuration remains the same. Just what is still, and was previous of the reboot, the current master of the system, restarted with a wiped data set. In distributed systems this is the process not acting as specified, since Redis processes must reload their dataset on restart, otherwise they break everything, per design. Now if it is a good idea to change this design: in the future yes, but so far we had not diskless replication, so for the replica to synchronize to write on disk was, anyway, needed, so why turn off persistence, and why to support it if the disk is needed anyway? See? Arguments instead of random blabling and we can construct a reality or a model we both agree about. Then we can debate about what we think is right or not.
  - notacoward 12y ago
    "it is an old-style replication system where there is the master IP address written in the configuration file." "Old style" doesn't explain it. Even if the original master has unconditional priority when it returns, that does not preclude it gathering whatever data might still exist from the others. I've been working on replication systems since '92, and I can't recall seeing any that would make such a poor choice. Can you point to any, or is this really a "new style" idea? "we can construct a reality or a model we both agree about." I will concede that the behavior might be compliant with how the system was specified. I'm not 100% convinced yet, but at least - now - you've made a credible case for that. "Then we can debate about what we think is right or not." Not. The data-preserving mechanisms (e.g. view/epoch IDs) are so easy to implement in this case that leaving them out is unjustifiable. You even seem to be coming around to that view yourself when you say "in the future yes" but apparently you can't bring yourself to admit that it was always the right choice.

by antirez 12y ago

notacoward 12y ago

See my response to seiji. Exact mimicry might be a defined behavior, but where is it mandated who should mimic whom? Where had you said, before this, that the node with no data must be the one whose (empty) view should prevail? As for your false dichotomy between reading the literature and getting stuff done, or your "make the effort to understand" ad hominem - go to hell. I've been doing this longer than you, I've been doing it better than you, and I've been writing about it as well. You make the effort to understand the problem before you shoot yourself in the foot yet again.

antirez 12y ago
"but where is it mandated who should mimic whom" In Redis if you don't use any HA system like Sentinel, the map is fixed, it is an old-style replication system where there is the master IP address written in the configuration file. Since the system is not supposed to lose the data on restarts, this is fine, but as soon as you want to support a different mode of operation with persistence-less masters, this must be modified, being Redis used with HA or not. Now if you want to put Sentinel in the mix, the problem with this setup is not that the returning master gets promoted with a broken data set, but that if the restart is fast enough, the failure detection of Sentinel is not triggered at all, so the configuration remains the same. Just what is still, and was previous of the reboot, the current master of the system, restarted with a wiped data set. In distributed systems this is the process not acting as specified, since Redis processes must reload their dataset on restart, otherwise they break everything, per design. Now if it is a good idea to change this design: in the future yes, but so far we had not diskless replication, so for the replica to synchronize to write on disk was, anyway, needed, so why turn off persistence, and why to support it if the disk is needed anyway? See? Arguments instead of random blabling and we can construct a reality or a model we both agree about. Then we can debate about what we think is right or not.
- notacoward 12y ago
  "it is an old-style replication system where there is the master IP address written in the configuration file." "Old style" doesn't explain it. Even if the original master has unconditional priority when it returns, that does not preclude it gathering whatever data might still exist from the others. I've been working on replication systems since '92, and I can't recall seeing any that would make such a poor choice. Can you point to any, or is this really a "new style" idea? "we can construct a reality or a model we both agree about." I will concede that the behavior might be compliant with how the system was specified. I'm not 100% convinced yet, but at least - now - you've made a credible case for that. "Then we can debate about what we think is right or not." Not. The data-preserving mechanisms (e.g. view/epoch IDs) are so easy to implement in this case that leaving them out is unjustifiable. You even seem to be coming around to that view yourself when you say "in the future yes" but apparently you can't bring yourself to admit that it was always the right choice.