6 ms·
Redis scripts do not expire keys atomically
- andydunstall 5y agoHey, I'm the author of this blog post. Ask me anything!
- ElectronShak 5y agoWould Redis Keyspace Notifications [1] help in this case? Like a script that runs when a specific key expires, such that all other related keys can be deleted too. [1] https://redis.io/topics/notifications https://redis.io/topics/notifications
- andydunstall 5y agothanks for pointing that out - will take a look :)
- avinassh 5y agothese notifications aren't guaranteed to fire at the expire time, wouldn't that be same as current problem?
- c0l0 5y agoI don't want to sound snarky, but since you mentioned that Redis does not document if/that "multiple key expiry is guaranteed to occur at the same time if keys have the same EXPIREAT setting", I am tempted to ask - why did you not check the implementation you're using? That's the unique selling point of using FOSS - you don't have rely on documentation or observational guesswork - you can wade in and really see for yourself! Redis' source code is especially readable in my opinion (thanks again, antirez, for creating this little load-bearing gem of a data structure server), so I would very much encourage you to try, even if C is not your particular forte :)
- staticassertion 5y agoNot who you're asking but... why have documentation at all? So that we don't have to read source code.
- edf13 5y agoYou can’t cover every usage possibility in docs… having the code available allows you to check expected behaviour
- rileymat2 5y agoYou also can’t tell what is supported behavior and what is implementation coincidence from the code…
- c0l0 5y agoBecause documentation is what is needs to be: A simplified and necessarily incomplete description of program behavior. It's a trade-off, since a complete and exhaustive description of all intricacies of potential program behavior would be at least as complex as the definition (= its source code) of said program. If you happen to have a question that ends up on the wrong side of that trade-off between documentation's completeness and accessibility, you will have to descend into the depths that lie beneath. I believe the redis documentation actually strikes a rather OK balance in that regard.
- staticassertion 5y agoGood docs generally make guarantees and pitfalls clear.
- scraptor 5y agoOf course having documentation is better but that's not a reason not to read the source code if the documentation isn't enough.
- voidfunc 5y agoBecause good docs are like a good search engine: high hit rate for what you want most of the time. The edge cases and quirky impl detail questions are what src is for.
- deleted 5y ago[deleted]
- blutack 5y agoRedis doesn't guarantee that expiry happens exactly at expiry time ([1] gives up to a 1 millisecond delay), so if your system assumes that all related keys are removed at exactly the same time it is still broken (although the race window is now tighter). Maybe you could store the related keys in a hash or something? I'm also guilty of trying to use features in redis in ways they weren't really designed for - it's such a fantastic bit of technology that a lot of the time you can still get away with it but it's not redis's fault when it doesn't work! 1: https://redis.io/commands/expire https://redis.io/commands/expire
- VWWHFSfQ 5y agoRight. Relying on atomic and absolutely reliable expiration is the problem anyway. If it's really that important then it needs to run in a transaction that manually sweeps the keys.
- andydunstall 5y agoyep - part of our fix was ordering the expires such that the invariants are maintained even if the expires arn't exactly at the same time (we are assuming expires are at least ordered)
- macspoofing 5y ago>we are assuming expires are at least ordered Why are you assuming that? And if true, why do you think that assumption will hold in the future????? Redis could always tweak or change the behavior.
- deleted 5y ago[deleted]
- WJW 5y agoI'm pretty sure they aren't ordered. Redis active key expiry is a random process: https://redis.io/commands/expire#how-redis-expires-keys https://redis.io/commands/expire#how-redis-expires-keys. TL;DR: Redis will randomly sample the keyspace and delete any expired keys it finds in the sample. If a large enough percentage of the sample was expired, it will repeat the process. Since the sample is random, it cannot be ordered. There is also a "passive" key expiry where any read to a key will check the expiry time and delete the key if the current time is larger than the expiry time.
- kristoff_it 5y agoI think the solutions to this problem are: - If the keys are strings, just use a single hashmap and you're done. - If the keys have different types, then you need a Redis module that implements something like `MPEXPIREAT time key1 key2 ...` and then you need to either be able to handle the case where you successfully retrieve key1 but not key2 (because they expired half-way through) or you use a lua script to execute the multi-key access atomically. That said, without the lua script the whole point of having the keys expire at the same time makes less sense, IMO.
- alaties 5y agoIt should be noted that an EXPIREAT triggers a deletion if the timestamp being set is in the past and could lead to an increase in memory and disk pressure during script execution. Worth checking if the behavior there differs in any meaningful way since y'all made the change. EXPIRE usually just marks a key and relies on either subsequent operations to the same key or garbage collection to actually excise the entry and do the subsequent bookkeeping, I believe. It's been a while since I had to deal with this kind of problem though, so please tell me if things are different now.
- gorjusborg 5y agoIt seems the article's implication is that they are relying on redis TTL and expiry to guarantee all keys exists or no keys exist, at all times, for a given key set. I never got the impression that Redis gave guarantees on when expired keys were deleted, just that they would be eventually. If the issue is that you want a consistent all or nothing, why not just use a single key to make that determination?
- andydunstall 5y agowe reordered our keys such that they can expire at different times - also redis checks a keys TTL before returning - and wont return if its expired. i think it will 'eventually' delete if its not accessed, but we only cared about keys being returned, not when its actually deleted
- gorjusborg 5y agoThanks for clarifying. I wasn't aware of the fact that redis removes expired keys even if they aren't deleted. I've always thought of expiry as more of a resource management annotation. Interesting way to use it.
- macspoofing 5y ago>The integrity of a set of related keys requires that either all keys exist, or none exist. Using a Key-Value DB to store relational data eh - that never leads to problems. Their solution isn't any better because the API does not guarantee a particular order or atomicity of key expiration - so great that it works for you now[1], but next version may change the behavior. [1] Does it work though? Are you really really sure?
- andydunstall 5y ago>The integrity of a set of related keys requires that either all keys exist, or none exist Sorry it wasent that clear that part of our fix was such that it no longer matters if all keys exist or none - we reordered the expires such that the invariants still hold even if all the keys dont still exist My understanding of expiry is - its not guaranteed when keys are expired if they are not accessed - but if a looked up keys TTL is hit it will not be returned - which is all we cared about
- matthewaveryusa 5y agoI ran into a similar issue and used a hashset of all the related keys with ttls on the individual keys. the hashset would expire making the keys unreachable, and the TTls basically garbages collects the individual keys. anything that required to transact with the set would do so atomically by being wrapped in a lua script.
- jpswain 5y agoIs there a reason that the DEL command wouldn’t work? Is there some semantic difference with expire?