4 ms·

This is not what happened, a replica with persistence turned off restarted with an empty data set, and the slaves replicated that empty data set. Basically Redi

by antirez 12y ago

This is not what happened, a replica with persistence turned off restarted with an empty data set, and the slaves replicated that empty data set. Basically Redis replication currently must be used with some form of on-disk persistence turned on. However after the introduction of diskless replication (http://antirez.com/news/81 http://antirez.com/news/81) we now have a good reason to support replication with persistence turned down in a proper way, so there is already a feature work in progress in order to support the Stripe-alike use case: https://github.com/antirez/redis/issues/2087 https://github.com/antirez/redis/issues/2087

dkarapetyan 12y ago
Yes, it's obvious something went wrong with their promotion logic but in a complicated enough system this is going to happen and if anyone is using a volatile store as anything other than a cache then this is going to bite them in the ass sooner or later. I'm happy to see there will be features added to support the use cases that Stripe and others have in mind.
- grandalf 12y ago
  Redis is not volatile when on-disk persistence is turned on. It's non-volatile and non-atomic.
  - dkarapetyan 12y ago
    Well in this instance the master did not have on-disk persistence turned on because of latency issues but the slaves did have it turned on. What would you call that kind of setup? Volatile or not?
    - grandalf 12y ago
      The issue was not latency of the replica but the logic used when restarting. It's just a configuration bug. In a distributed system, some nodes can act like a cache and others like a persistent store.
    - seiji 12y ago
      Quick semantic cleanup: Redis has two types of persistence: snapshot persistence (RDB) and journal persistence (AOF). It sounds like they tried only snapshot ("copy the world") persistence, found it to increase latency, then turned it off. It probably would have been okay with journal persistence. Their setup would have been fine _if_ the master never came back up. If the master didn't restart, then something in their infrastructure would have probably started using the replicas for data (and/or they would have promoted one of the replicas to be the new master, then the other replica would replicate from the new master). But—having persistence on the replicas did allow them to copy the dataset off to backup storage. Then they were able to restore the old data when it was needed. So, we can prove their setup was persistent since they lost all their data and recovered it. :)
      - dkarapetyan 12y ago
        Thanks for the clarifications.
abalone 12y ago
If the promotion logic is wrong how would persistence have helped? Say the primary disk fails. If it's still considered primary when it's brought back up with a fresh disk, wouldn't you get the same empty-replication problem? (I know nothing about redis, just wondering.)
- seiji 12y ago
  In the case described, there is no promotion logic. The replicas will try to reconnect to their original master forever unless something else (like Sentinel) redirects them in an actual failover/promotion setup. So, the master had data, it died, it restart with no data, then the replicas immediately reconnect. If the master had persistence enabled, it would have reloaded the old dataset on startup and the replicas would have re-downloaded everything—since they are replicas of the master, they will always prefer the master data over their own, even if the master is empty. If you were in a strange case where the disk failed and you replaced it with an empty disk (is that what you mean by "fresh disk?") then it's the same as starting an empty dataset. Not entirely relevant since the server would be intentionally started empty after a maintenance action instead of just restarting the already-populated process that restarts as empty because there's no saved dataset to load on startup. The "all replicas resync an empty dataset" is a logical consequence of the configuration they enabled, but one without obvious repercussions without either directly experiencing it or a longer multi-chain thought experiment. (but, fixes for such things are already on the way—soon!)
  - deleted 12y ago
    [deleted]
  - antirez 12y ago
    Just to add some more info: Funny enough what triggers this problem when you have master persistence turned down is, the lack of failover, if the reboot happens fast enough, in case you are using Sentinel, for it to failover to a replica. So no failure was sensed at all, just the master magically wiped its data set. So from the point of view of distributed systems, if you want to analyze the sum of Redis replicated nodes + Sentinel, the problem is that the system is not designed to cope with nodes losing state on restarts. However it is possible to improve it, and I'm doing it, but before diskless replication it was IMHO pretty useless to have support for persistence-less operations in conjunction with replication, since the slaves to synchronize required anyway the master to save an RDB file on disk.

by antirez 12y ago

dkarapetyan 12y ago

Yes, it's obvious something went wrong with their promotion logic but in a complicated enough system this is going to happen and if anyone is using a volatile store as anything other than a cache then this is going to bite them in the ass sooner or later. I'm happy to see there will be features added to support the use cases that Stripe and others have in mind.

grandalf 12y ago
Redis is not volatile when on-disk persistence is turned on. It's non-volatile and non-atomic.
- dkarapetyan 12y ago
  Well in this instance the master did not have on-disk persistence turned on because of latency issues but the slaves did have it turned on. What would you call that kind of setup? Volatile or not?
  - grandalf 12y ago
    The issue was not latency of the replica but the logic used when restarting. It's just a configuration bug. In a distributed system, some nodes can act like a cache and others like a persistent store.
  - seiji 12y ago
    Quick semantic cleanup: Redis has two types of persistence: snapshot persistence (RDB) and journal persistence (AOF). It sounds like they tried only snapshot ("copy the world") persistence, found it to increase latency, then turned it off. It probably would have been okay with journal persistence. Their setup would have been fine _if_ the master never came back up. If the master didn't restart, then something in their infrastructure would have probably started using the replicas for data (and/or they would have promoted one of the replicas to be the new master, then the other replica would replicate from the new master). But—having persistence on the replicas did allow them to copy the dataset off to backup storage. Then they were able to restore the old data when it was needed. So, we can prove their setup was persistent since they lost all their data and recovered it. :)
    - dkarapetyan 12y ago
      Thanks for the clarifications.

abalone 12y ago

If the promotion logic is wrong how would persistence have helped? Say the primary disk fails. If it's still considered primary when it's brought back up with a fresh disk, wouldn't you get the same empty-replication problem? (I know nothing about redis, just wondering.)

seiji 12y ago
In the case described, there is no promotion logic. The replicas will try to reconnect to their original master forever unless something else (like Sentinel) redirects them in an actual failover/promotion setup. So, the master had data, it died, it restart with no data, then the replicas immediately reconnect. If the master had persistence enabled, it would have reloaded the old dataset on startup and the replicas would have re-downloaded everything—since they are replicas of the master, they will always prefer the master data over their own, even if the master is empty. If you were in a strange case where the disk failed and you replaced it with an empty disk (is that what you mean by "fresh disk?") then it's the same as starting an empty dataset. Not entirely relevant since the server would be intentionally started empty after a maintenance action instead of just restarting the already-populated process that restarts as empty because there's no saved dataset to load on startup. The "all replicas resync an empty dataset" is a logical consequence of the configuration they enabled, but one without obvious repercussions without either directly experiencing it or a longer multi-chain thought experiment. (but, fixes for such things are already on the way—soon!)
- deleted 12y ago
  [deleted]
- antirez 12y ago
  Just to add some more info: Funny enough what triggers this problem when you have master persistence turned down is, the lack of failover, if the reboot happens fast enough, in case you are using Sentinel, for it to failover to a replica. So no failure was sensed at all, just the master magically wiped its data set. So from the point of view of distributed systems, if you want to analyze the sum of Redis replicated nodes + Sentinel, the problem is that the system is not designed to cope with nodes losing state on restarts. However it is possible to improve it, and I'm doing it, but before diskless replication it was IMHO pretty useless to have support for persistence-less operations in conjunction with replication, since the slaves to synchronize required anyway the master to save an RDB file on disk.