4 ms·

Yes, it's obvious something went wrong with their promotion logic but in a complicated enough system this is going to happen and if anyone is using a volatile s

Yes, it's obvious something went wrong with their promotion logic but in a complicated enough system this is going to happen and if anyone is using a volatile store as anything other than a cache then this is going to bite them in the ass sooner or later. I'm happy to see there will be features added to support the use cases that Stripe and others have in mind.

grandalf 12y ago
Redis is not volatile when on-disk persistence is turned on. It's non-volatile and non-atomic.
- dkarapetyan 12y ago
  Well in this instance the master did not have on-disk persistence turned on because of latency issues but the slaves did have it turned on. What would you call that kind of setup? Volatile or not?
  - grandalf 12y ago
    The issue was not latency of the replica but the logic used when restarting. It's just a configuration bug. In a distributed system, some nodes can act like a cache and others like a persistent store.
  - seiji 12y ago
    Quick semantic cleanup: Redis has two types of persistence: snapshot persistence (RDB) and journal persistence (AOF). It sounds like they tried only snapshot ("copy the world") persistence, found it to increase latency, then turned it off. It probably would have been okay with journal persistence. Their setup would have been fine _if_ the master never came back up. If the master didn't restart, then something in their infrastructure would have probably started using the replicas for data (and/or they would have promoted one of the replicas to be the new master, then the other replica would replicate from the new master). But—having persistence on the replicas did allow them to copy the dataset off to backup storage. Then they were able to restore the old data when it was needed. So, we can prove their setup was persistent since they lost all their data and recovered it. :)
    - dkarapetyan 12y ago
      Thanks for the clarifications.

grandalf 12y ago

Redis is not volatile when on-disk persistence is turned on. It's non-volatile and non-atomic.

dkarapetyan 12y ago
Well in this instance the master did not have on-disk persistence turned on because of latency issues but the slaves did have it turned on. What would you call that kind of setup? Volatile or not?
- grandalf 12y ago
  The issue was not latency of the replica but the logic used when restarting. It's just a configuration bug. In a distributed system, some nodes can act like a cache and others like a persistent store.
- seiji 12y ago
  Quick semantic cleanup: Redis has two types of persistence: snapshot persistence (RDB) and journal persistence (AOF). It sounds like they tried only snapshot ("copy the world") persistence, found it to increase latency, then turned it off. It probably would have been okay with journal persistence. Their setup would have been fine _if_ the master never came back up. If the master didn't restart, then something in their infrastructure would have probably started using the replicas for data (and/or they would have promoted one of the replicas to be the new master, then the other replica would replicate from the new master). But—having persistence on the replicas did allow them to copy the dataset off to backup storage. Then they were able to restore the old data when it was needed. So, we can prove their setup was persistent since they lost all their data and recovered it. :)
  - dkarapetyan 12y ago
    Thanks for the clarifications.