4 ms·
A lot of (especially web) applications suffer from race conditions. Luckily for most of them the effects are either temporary and/or not severe. It does not on
by eis 15y ago
A lot of (especially web) applications suffer from race conditions. Luckily for most of them the effects are either temporary and/or not severe.
It does not only apply to Redis but basically all data stores unless something like transactions in RDBMSs are used and even then you can get into trouble at times.
The last example still has a problem when an exception is thrown and the 'redis.del "setting_up_auth_token"' fails for whatever reason.
Furthermore it has a race condition if a process wants to read the content of 'web_service_auth_token' because it sees 'setting_up_auth_token' has been set but the other process who is setting it up has not reached the redis.set line for it.
The example itself is not very good as I'd rather do a little bit of extra work (would not hurt here) than being vulnerable to failures.
Assuming the work really needs to be done strictly once - either because it's a lot or the logic requires it - you would probably have to do something like the following pseudo code:
while (true) {
watch("setup_key")
v = get("setup_key")
if v == 2 {
break
}
if v == 1 {
sleep(1)
continue
}
multi()
setex("setup_key", 10, 1) // 10s timeout
if !exec() {
continue // someone else got the lock in the meantime, try again
}
// we now have a temporary exclusive "lock"
... do the real work ...
set("setup_key", 2)
}
What does this do? Well, setup_key acts like an exclusive lock which has 3 states: non-existing, 1 and 2.
non-existing means we are the first, lets do the work.
1 means the lock is being held, sleep and check again.
2 means the work has been finished.
The watch(),multi(),exec() work as a CAS (really, whytf is this not a single command?) to atomically acquire the lock. If the lock is held by someone, we just sleep 1 second and try again.
I use a timeout on the key in case the client holding the lock dies to prevent a deadlock.
But this is STILL not perfect. What if your setup work takes longer than the timeout specified for the temporary key? I haven't spent enough time thinking about it right now to come up with a solution if there even is any for redis, so I'll leave it as an exercise to the reader :)
To be honest: with slightly different commands/semantics, Redis could be MUCH easier to write correct applications for. Especially the "transaction" support is mediocre imho.
Conclusion: Many web applications are inherintly and severely concurrent.
Concurrency is HARD to get right. There are many patterns like locks, atomic operations, transactions etc. - all with completely different semantics. Redis is singlethreaded for a reason :)
- atamyrat 15y agoI've used counters in Redis for mutex control. If incr returns 1, it means we have succeeded. Otherwise it decrements it and tries again after 1 second until it suceeds. Here is the code: https://gist.github.com/1062584 https://gist.github.com/1062584
- statictype 15y agoIn our setup (using Sql Server, alas, not redis but same concept applies) the worker process sets the lock and the time of acquiring it. Other workers look for pending jobs that either don't have the lock set or the lock has expired where the expiry time is 5min + lock time. Since each 'job' is a small amount of work, 5 minutes is fairly generous. If a worker goes down because of a system crash or network failure, then that job wouldn't get processed again for 5 minutes. That's (for us) an acceptable overhead. There's a chance the job gets processed twice, if it gets locked, work gets done, but releasing the lock fails. That's a trade-off we're willing to make as we can't really think of any better solution. I believe, since redis has built-in support for expiring records, this maybe even easier. To process job 332, you would try to check the existence of 'lock:332' and then set it if it didn't exist, and give it an expiry time. Would be interested in hearing how other people solve these types of issues.
- santosh_kumar 15y agoThanks for the detailed analysis. A quick thing to consider though, if redis.del fails, it can only mean redis is busted. This means, that none of the examples in any context that use redis in anyway have any bearing. Hence, guarding against that is a little excessive imo.