5 ms·

It supports backup and restore today but RocksDB doesn't have a replication story at the moment. Side note: Resque will lose jobs if it crashes, it doesn't use

It supports backup and restore today but RocksDB doesn't have a replication story at the moment. Side note: Resque will lose jobs if it crashes, it doesn't use RPOPLPUSH. I hope you'll take another look at Faktory in a few months, maybe we'll have addressed your issues.

koolba 9y ago
Using RPOPLPUSH there's no guarantee you won't lose the job if the Redis server crashes prior to publishing it. Even with persistence enabled (AOF) there's still a small window of time between receiving the message and the next AOF fsync. The sender would think the message is queued as it got an OK response back but the message will be lost. There are three common solutions to this: First is to pretend it doesn't exist (quite common!). Second is to understand that losing messages is a possibility and only use it for message types that the loss of a message wouldn't be critical (which covers quite a bit). The third solution, which actually addresses the problem, separates the persistence of message details to transactional store from the event notification of a new message. A "sweeper" type task can thing check the persisted message list for messages that haven't been processed and re-publish them. (Disclaimer: I'm a huge fan of Redis and think it's the bee's knees of data structure servers.)
- elvinyung 9y ago
  You mean RDB? I thought AOF can be configured to fsync at every write.
  - koolba 9y ago
    The default is to fsync once per second so the failure window is small but it's not zero. It's possible to fsync on every write[1] but it may be too slow as the single threaded nature of Redis means you've serialized every write operation. Plus that'd apply to all usage of that Redis server, not just the queue. [1]: https://redis.io/topics/persistence#how-durable-is-the-append-only-file https://redis.io/topics/persistence#how-durable-is-the-appen...
- je42 9y ago
  you can run a redis cluster.
jitl 9y ago
Fair, and I will :-) Thanks for your response. Are you married to RocksDB/instance storage, or are you considering adapters or something to a pluggable persistence layer?
zellyn 9y ago
Do you have plans for multi-node persistence / geographic redundancy? We'd love something like this, but right now it would probably be easier to roll something custom on top of either Kafka or CockroachDB…

koolba 9y ago

Using RPOPLPUSH there's no guarantee you won't lose the job if the Redis server crashes prior to publishing it. Even with persistence enabled (AOF) there's still a small window of time between receiving the message and the next AOF fsync. The sender would think the message is queued as it got an OK response back but the message will be lost. There are three common solutions to this: First is to pretend it doesn't exist (quite common!). Second is to understand that losing messages is a possibility and only use it for message types that the loss of a message wouldn't be critical (which covers quite a bit). The third solution, which actually addresses the problem, separates the persistence of message details to transactional store from the event notification of a new message. A "sweeper" type task can thing check the persisted message list for messages that haven't been processed and re-publish them. (Disclaimer: I'm a huge fan of Redis and think it's the bee's knees of data structure servers.)

elvinyung 9y ago
You mean RDB? I thought AOF can be configured to fsync at every write.
- koolba 9y ago
  The default is to fsync once per second so the failure window is small but it's not zero. It's possible to fsync on every write[1] but it may be too slow as the single threaded nature of Redis means you've serialized every write operation. Plus that'd apply to all usage of that Redis server, not just the queue. [1]: https://redis.io/topics/persistence#how-durable-is-the-append-only-file https://redis.io/topics/persistence#how-durable-is-the-appen...
je42 9y ago
you can run a redis cluster.

jitl 9y ago

Fair, and I will :-) Thanks for your response. Are you married to RocksDB/instance storage, or are you considering adapters or something to a pluggable persistence layer?

zellyn 9y ago

Do you have plans for multi-node persistence / geographic redundancy? We'd love something like this, but right now it would probably be easier to roll something custom on top of either Kafka or CockroachDB…