3 ms·
Using RPOPLPUSH there's no guarantee you won't lose the job if the Redis server crashes prior to publishing it. Even with persistence enabled (AOF) there's stil
by koolba 9y ago
Using RPOPLPUSH there's no guarantee you won't lose the job if the Redis server crashes prior to publishing it. Even with persistence enabled (AOF) there's still a small window of time between receiving the message and the next AOF fsync. The sender would think the message is queued as it got an OK response back but the message will be lost.
There are three common solutions to this:
First is to pretend it doesn't exist (quite common!).
Second is to understand that losing messages is a possibility and only use it for message types that the loss of a message wouldn't be critical (which covers quite a bit).
The third solution, which actually addresses the problem, separates the persistence of message details to transactional store from the event notification of a new message. A "sweeper" type task can thing check the persisted message list for messages that haven't been processed and re-publish them.
(Disclaimer: I'm a huge fan of Redis and think it's the bee's knees of data structure servers.)
- elvinyung 9y agoYou mean RDB? I thought AOF can be configured to fsync at every write.
- koolba 9y agoThe default is to fsync once per second so the failure window is small but it's not zero. It's possible to fsync on every write[1] but it may be too slow as the single threaded nature of Redis means you've serialized every write operation. Plus that'd apply to all usage of that Redis server, not just the queue. [1]: https://redis.io/topics/persistence#how-durable-is-the-append-only-file https://redis.io/topics/persistence#how-durable-is-the-appen...
- je42 9y agoyou can run a redis cluster.