3 ms·
> Memory duplication has a relatively simple solution, namely using external cache such as redis. But the thundering herds problem remains. At time t=0, each pr
by zzzeek 13y ago
> Memory duplication has a relatively simple solution, namely using external cache such as redis. But the thundering herds problem remains. At time t=0, each process receives a request for f(new input). Each process looks in the cache, finds it empty, and begins computing f(new input). As a result every single process is blocked.
this is incorrect. The processes coordinate on a lock held in redis itself.
This solution is available right now using the Redis backend in dogpile.cache (of which I am the author): https://dogpilecache.readthedocs.org/en/latest/usage.html https://dogpilecache.readthedocs.org/en/latest/usage.html
- yummyfajitas 13y agoI was unaware that redis had that feature, I'll update the blog post to reference it. Thanks.
- njbooher 13y agoIt's also not too hard to DIY: http://www.dr-josiah.com/2012/01/creating-lock-with-redis.html http://www.dr-josiah.com/2012/01/creating-lock-with-redis.ht... https://github.com/njbooher/boglab_tools/blob/dece35f13a8fcb9cb5b7eefdee2d6f9916350918/entrez_cache.py#L105 https://github.com/njbooher/boglab_tools/blob/dece35f13a8fcb... When multiple processes ask this for the same file one of them downloads it and the others wait for it to finish.