4 ms·
> I'm not even going to touch his argument that using system clocks in a distributed locking algorithm is safe... Please do
by dvirsky 11y ago
> I'm not even going to touch his argument that using system clocks in a distributed locking algorithm is safe...
Please do
- carllerche 11y agoI don't need to, Martin did an excellent job touching on this in his original post, including real world references.
- dvirsky 11y agoAntirez responded to that and disagreed to this point in his new post. What do you make of his response?
- Guvante 11y agoClock drift is a real thing in distributed systems. If I have a 5 second lock that expires at 12:01:00 and my clock is off by 1 second I could write at 12:01:01 potentially after someone else has a lock, or worse, after someone else has written.
- dvirsky 11y agobut the expiration time is relative to the redis server's time, and the time measurement on the client side is done relative to the client's time. Other clients don't care about your clock and you don't care about redis' clock AFAIK. From TFM: > The client computes how much time elapsed in order to acquire the lock, by subtracting from the current time the timestamp obtained in step 1. If and only if the client was able to acquire the lock in the majority of the instances (at least 3), and the total time elapsed to acquire the lock is less than lock validity time, the lock is considered to be acquired.
- dsp1234 11y agoWhat is the effect on the client of a VM live migration "pause"? or daylight savings change? or ntp updates which change the time? Since the system clock can change relative to itself at any time, what effect does that have on the algorithm?
- dvirsky 11y agoI don't disagree, I'm just saying that time drift between nodes is not an issue with this algorithm. Also, using monotonic clock as antirez says he intends to in the post, will take care of most of these scenarios besides VM migration pauses.
- xorcist 11y agoDistributed time is tricky, but it depends on what the intended use is. Under normal operation it will only move forward, for example. > VM live migration "pause"? System clock will take a larger step than usual. It won't go backwards. > daylight savings change? None. System clock is UTC for a reason. > ntp updates which change the time? NTP only drifts the clock (under normal operation).
- dsp1234 11y agoNot that I think it effects your point, but just off the top of my head: System clock will take a larger step than usual. It won't go backwards. Assumes that checkpoint and restore to a previous time won't happen. NTP only drifts the clock (under normal operation). Unless it's a system like this. Note that this was pulled at about 1pm EST. Waiting for clock tick... ...got clock tick Time read from Hardware Clock: 2016/02/09 22:36:50 Hw clock time : 2016/02/09 22:36:50 = 1455057410 seconds since 1969 Tue 09 Feb 2016 02:36:50 PM PST -1.038948 seconds Under normal operation it will only move forward 'Under normal operation' is not really a high bar for distributed systems. After all, the network is reliable 'under normal operation' too.
- SrslyJosh 11y ago> Assumes that checkpoint and restore to a previous time won't happen. Or migration to a host whose clock is behind the original host's clock.
- Dylan16807 11y agoUse CLOCK_MONOTONIC. Or you could make your client abort when the clock goes backwards. If someone forks a VM from a running snapshot, you're screwed whether they mess with the clock or not.
- scurvy 11y agoWhat about requiring PTP sync or shutting down the node? I have no idea about the practical implications, just something I thought of. You can get a CDMA time server on ebay for a few hundred bucks that support PTP.
- Guvante 11y agoOff hand you either need multiple time servers or you might as well have a central lock (since you are bottlenecked in either case) which just punts the issue, now the various time servers need to be in sync or the issue comes up.
- carllerche 11y agoReplying here since I can't reply to your deeper comment. A client can never know with certainty that it has successfully acquired the lock because, for example, the process could freeze between reading the current time from hardware and the steady_time function returning. You can not rely on time, for safety. Only for advice.
- dvirsky 11y agoyes, that goes back to the first issue in Martin's article, and he mentioned the tokens thing for this. But time drift between the machines is not an issue, is my point.