3 ms·
I've had a lot of sucess using a key in redis with a TTL value instead of a local PID file. Although adding redis to the picture adds a large new point of fail
by wc- 11y ago
I've had a lot of sucess using a key in redis with a TTL value instead of a local PID file. Although adding redis to the picture adds a large new point of failure, I can then have a cronjob set up on multiple instances and still ensure it only runs once across all of them.
I'm sure there is a simpler way of doing this, how have other people solved redundantly ensuring a single cronjob runs?
- kiallmacinnes 11y agoThe common and "traditional" way of doing distributed locking is with a coordination service like ZooKeeper. ZooKeeper style services have an advantage of no TTLs - the moment a process dies, the lock is released, and the next in line waiting on the lock is immediately notified. Redis/Memcache with a TTL serves this purpose for the most part, but if you require as close to a 100% guarantee that 1 and only 1 process holds the lock at any given time, these will eventually fail you. Think network partitions, tasks outlasting the TTL, replication lag/eventual consistency etc. ZooKeeper and similar use concensus protocols like ZAB, Paxos or Raft to provide guarantees even in the face of failure.
- ccannon 11y agoThis is not meant for distributed systems.
- kiallmacinnes 11y agoThe parent comment was certainly talking about distributed systems!